Oct 30 17:55:30.839240 ip-10-0-138-176 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 17:55:30.839252 ip-10-0-138-176 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 17:55:30.839261 ip-10-0-138-176 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 17:55:30.839576 ip-10-0-138-176 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 17:55:42.045104 ip-10-0-138-176 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 17:55:42.045126 ip-10-0-138-176 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot e92f1dbc5cb04908a2bd5927dbebbac5 -- Oct 30 17:57:42.824510 ip-10-0-138-176 systemd[1]: Starting Kubernetes Kubelet... Oct 30 17:57:43.208018 ip-10-0-138-176 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:57:43.208018 ip-10-0-138-176 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 17:57:43.208018 ip-10-0-138-176 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:57:43.208018 ip-10-0-138-176 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:57:43.211885 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.211792 2565 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:57:43.211885 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.211881 2565 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:57:43.211885 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.211886 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:57:43.211885 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.211890 2565 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:57:43.211885 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.211894 2565 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:57:43.212221 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.211898 2565 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:57:43.212221 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.211902 2565 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:57:43.212221 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.211906 2565 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:57:43.212221 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.211910 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:57:43.212221 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.211913 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:57:43.212221 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.211917 2565 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:57:43.212221 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.211928 2565 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:57:43.212221 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.211932 2565 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:57:43.212221 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.211937 2565 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:57:43.212221 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.211940 2565 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:57:43.212221 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.211944 2565 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:57:43.212221 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.211947 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:57:43.212221 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.211953 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:57:43.212221 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.211957 2565 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:57:43.212221 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.211961 2565 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:57:43.212221 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.211966 2565 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:57:43.212221 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.211969 2565 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:57:43.212221 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.211973 2565 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:57:43.212221 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.211977 2565 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:57:43.212221 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.211981 2565 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:57:43.213030 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.211985 2565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:57:43.213030 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.211989 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:57:43.213030 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.211993 2565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:57:43.213030 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.211997 2565 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:57:43.213030 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212001 2565 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:57:43.213030 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212005 2565 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:57:43.213030 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212009 2565 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:57:43.213030 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212012 2565 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:57:43.213030 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212017 2565 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:57:43.213030 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212021 2565 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:57:43.213030 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212026 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:57:43.213030 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212030 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:57:43.213030 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212035 2565 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:57:43.213030 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212041 2565 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:57:43.213030 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212046 2565 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:57:43.213030 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212051 2565 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:57:43.213030 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212055 2565 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:57:43.213030 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212059 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:57:43.213030 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212063 2565 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:57:43.214695 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212069 2565 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:57:43.214695 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212073 2565 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:57:43.214695 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212077 2565 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:57:43.214695 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212081 2565 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:57:43.214695 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212086 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:57:43.214695 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212090 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:57:43.214695 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212094 2565 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:57:43.214695 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212098 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:57:43.214695 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212102 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:57:43.214695 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212106 2565 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:57:43.214695 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212111 2565 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:57:43.214695 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212115 2565 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:57:43.214695 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212119 2565 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:57:43.214695 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212124 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:57:43.214695 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212128 2565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:57:43.214695 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212150 2565 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:57:43.214695 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212154 2565 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:57:43.214695 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212159 2565 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:57:43.214695 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212163 2565 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:57:43.214695 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212167 2565 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:57:43.215408 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212171 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:57:43.215408 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212175 2565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:57:43.215408 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212180 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:57:43.215408 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212188 2565 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:57:43.215408 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212194 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:57:43.215408 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212199 2565 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:57:43.215408 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212204 2565 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:57:43.215408 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212208 2565 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:57:43.215408 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212215 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:57:43.215408 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212219 2565 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:57:43.215408 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212227 2565 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:57:43.215408 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212233 2565 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:57:43.215408 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212237 2565 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:57:43.215408 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212241 2565 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:57:43.215408 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212245 2565 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:57:43.215408 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212250 2565 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:57:43.215408 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212254 2565 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:57:43.215408 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212259 2565 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:57:43.215408 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212264 2565 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:57:43.215863 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212268 2565 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:57:43.215863 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212273 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:57:43.215863 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212278 2565 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:57:43.215863 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212282 2565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:57:43.215863 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212286 2565 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:57:43.215863 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212290 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:57:43.215863 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212294 2565 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:57:43.215863 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212298 2565 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:57:43.215863 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212303 2565 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:57:43.215863 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212307 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:57:43.215863 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212311 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:57:43.215863 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212316 2565 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:57:43.215863 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212320 2565 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:57:43.215863 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212324 2565 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:57:43.215863 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212328 2565 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:57:43.215863 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212332 2565 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:57:43.215863 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212337 2565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:57:43.215863 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212341 2565 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:57:43.215863 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212348 2565 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:57:43.216745 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212353 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:57:43.216745 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212357 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:57:43.216745 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212361 2565 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:57:43.216745 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.212366 2565 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:57:43.216745 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213645 2565 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:57:43.216745 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213654 2565 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:57:43.216745 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213657 2565 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:57:43.216745 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213660 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:57:43.216745 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213663 2565 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:57:43.216745 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213665 2565 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:57:43.216745 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213668 2565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:57:43.216745 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213671 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:57:43.216745 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213673 2565 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:57:43.216745 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213676 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:57:43.216745 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213678 2565 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:57:43.216745 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213681 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:57:43.216745 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213683 2565 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:57:43.216745 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213686 2565 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:57:43.216745 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213688 2565 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:57:43.217581 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213691 2565 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:57:43.217581 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213693 2565 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:57:43.217581 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213695 2565 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:57:43.217581 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213700 2565 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:57:43.217581 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213704 2565 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:57:43.217581 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213707 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:57:43.217581 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213710 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:57:43.217581 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213712 2565 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:57:43.217581 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213715 2565 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:57:43.217581 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213717 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:57:43.217581 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213720 2565 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:57:43.217581 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213723 2565 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:57:43.217581 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213725 2565 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:57:43.217581 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213729 2565 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:57:43.217581 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213732 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:57:43.217581 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213735 2565 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:57:43.217581 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213738 2565 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:57:43.217581 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213743 2565 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:57:43.217581 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213746 2565 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:57:43.218334 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213748 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:57:43.218334 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213751 2565 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:57:43.218334 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213753 2565 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:57:43.218334 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213756 2565 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:57:43.218334 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213758 2565 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:57:43.218334 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213761 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:57:43.218334 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213763 2565 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:57:43.218334 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213766 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:57:43.218334 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213769 2565 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:57:43.218334 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213771 2565 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:57:43.218334 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213774 2565 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:57:43.218334 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213776 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:57:43.218334 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213779 2565 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:57:43.218334 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213781 2565 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:57:43.218334 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213784 2565 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:57:43.218334 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213786 2565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:57:43.218334 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213789 2565 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:57:43.218334 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213791 2565 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:57:43.218334 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213794 2565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:57:43.218334 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213796 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:57:43.219065 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213798 2565 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:57:43.219065 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213801 2565 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:57:43.219065 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213803 2565 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:57:43.219065 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213806 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:57:43.219065 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213809 2565 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:57:43.219065 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213812 2565 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:57:43.219065 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213814 2565 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:57:43.219065 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213817 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:57:43.219065 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213819 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:57:43.219065 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213822 2565 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:57:43.219065 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213824 2565 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:57:43.219065 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213827 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:57:43.219065 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213829 2565 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:57:43.219065 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213832 2565 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:57:43.219065 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213834 2565 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:57:43.219065 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213838 2565 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:57:43.219065 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213840 2565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:57:43.219065 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213842 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:57:43.219065 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213845 2565 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:57:43.219065 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213849 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:57:43.219720 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213851 2565 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:57:43.219720 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213854 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:57:43.219720 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213857 2565 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:57:43.219720 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213859 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:57:43.219720 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213861 2565 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:57:43.219720 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213864 2565 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:57:43.219720 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213866 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:57:43.219720 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213869 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:57:43.219720 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213871 2565 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:57:43.219720 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213873 2565 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:57:43.219720 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213878 2565 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:57:43.219720 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213881 2565 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:57:43.219720 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213884 2565 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:57:43.219720 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213886 2565 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:57:43.219720 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213889 2565 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:57:43.219720 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213891 2565 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:57:43.219720 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213894 2565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:57:43.219720 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213896 2565 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:57:43.219720 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213899 2565 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:57:43.219720 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213901 2565 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:57:43.220225 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213904 2565 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:57:43.220225 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213906 2565 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:57:43.220225 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213909 2565 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:57:43.220225 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213911 2565 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:57:43.220225 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213914 2565 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:57:43.220225 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213917 2565 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:57:43.220225 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213920 2565 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:57:43.220225 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213922 2565 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:57:43.220225 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213925 2565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:57:43.220225 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213927 2565 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:57:43.220225 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213930 2565 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:57:43.220225 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.213934 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:57:43.220225 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214546 2565 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:57:43.220225 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214557 2565 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:57:43.220225 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214562 2565 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:57:43.220225 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214567 2565 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:57:43.220225 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214572 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:57:43.220225 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214576 2565 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:57:43.220225 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214580 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:57:43.220225 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214585 2565 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:57:43.220776 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214589 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:57:43.220776 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214594 2565 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:57:43.220776 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214601 2565 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:57:43.220776 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214605 2565 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:57:43.220776 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214609 2565 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:57:43.220776 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214613 2565 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:57:43.220776 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214618 2565 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:57:43.220776 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214622 2565 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:57:43.220776 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214626 2565 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:57:43.220776 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214630 2565 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:57:43.220776 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214634 2565 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:57:43.220776 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214638 2565 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:57:43.220776 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214643 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:57:43.220776 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214647 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:57:43.220776 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214650 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:57:43.220776 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214654 2565 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:57:43.220776 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214660 2565 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:57:43.220776 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214664 2565 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:57:43.220776 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214669 2565 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:57:43.221239 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214673 2565 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:57:43.221239 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214677 2565 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:57:43.221239 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214682 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:57:43.221239 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214686 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:57:43.221239 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214692 2565 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:57:43.221239 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214696 2565 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:57:43.221239 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214699 2565 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:57:43.221239 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214703 2565 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:57:43.221239 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214707 2565 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:57:43.221239 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214711 2565 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:57:43.221239 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214715 2565 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:57:43.221239 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214719 2565 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:57:43.221239 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214723 2565 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:57:43.221239 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214727 2565 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:57:43.221239 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214732 2565 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:57:43.221239 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214737 2565 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:57:43.221239 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214742 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:57:43.221239 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214746 2565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:57:43.221239 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214750 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:57:43.221239 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214754 2565 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:57:43.221701 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214758 2565 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:57:43.221701 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214762 2565 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:57:43.221701 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214766 2565 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:57:43.221701 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214771 2565 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:57:43.221701 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214775 2565 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:57:43.221701 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214779 2565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:57:43.221701 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214783 2565 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:57:43.221701 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214787 2565 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:57:43.221701 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214791 2565 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:57:43.221701 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214796 2565 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:57:43.221701 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214800 2565 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:57:43.221701 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214805 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:57:43.221701 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214809 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:57:43.221701 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214814 2565 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:57:43.221701 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214818 2565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:57:43.221701 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214823 2565 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:57:43.221701 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214828 2565 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:57:43.221701 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214832 2565 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:57:43.221701 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214837 2565 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:57:43.222186 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214840 2565 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:57:43.222186 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214844 2565 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:57:43.222186 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214848 2565 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:57:43.222186 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214852 2565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:57:43.222186 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214856 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:57:43.222186 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214860 2565 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:57:43.222186 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214865 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:57:43.222186 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214869 2565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:57:43.222186 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214875 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:57:43.222186 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214879 2565 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:57:43.222186 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214883 2565 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:57:43.222186 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214887 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:57:43.222186 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214891 2565 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:57:43.222186 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214895 2565 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:57:43.222186 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214901 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:57:43.222186 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214905 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:57:43.222186 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214909 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:57:43.222186 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214913 2565 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:57:43.222186 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214917 2565 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:57:43.222664 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214921 2565 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:57:43.222664 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214925 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:57:43.222664 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214929 2565 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:57:43.222664 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214933 2565 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:57:43.222664 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214937 2565 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:57:43.222664 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214945 2565 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:57:43.222664 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214951 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:57:43.222664 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214956 2565 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:57:43.222664 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214961 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:57:43.222664 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214966 2565 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:57:43.222664 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214971 2565 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:57:43.222664 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214975 2565 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:57:43.222664 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214980 2565 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:57:43.222664 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214984 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:57:43.222664 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214988 2565 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:57:43.222664 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214992 2565 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:57:43.222664 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.214996 2565 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:57:43.222664 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.215000 2565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:57:43.222664 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.215004 2565 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:57:43.223103 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.215011 2565 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:57:43.223103 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.215015 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:57:43.223103 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.215867 2565 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 17:57:43.223103 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.215891 2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 17:57:43.223103 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.215901 2565 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 17:57:43.223103 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.215908 2565 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 17:57:43.223103 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.215915 2565 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 17:57:43.223103 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.215920 2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 17:57:43.223103 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.215927 2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 17:57:43.223103 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.215934 2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 17:57:43.223103 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.215940 2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 17:57:43.223103 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.215944 2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 17:57:43.223103 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.215950 2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 17:57:43.223103 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.215955 2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 17:57:43.223103 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.215960 2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 17:57:43.223103 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.215965 2565 flags.go:64] FLAG: --cgroup-root="" Oct 30 17:57:43.223103 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.215970 2565 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 17:57:43.223103 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.215974 2565 flags.go:64] FLAG: --client-ca-file="" Oct 30 17:57:43.223103 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.215979 2565 flags.go:64] FLAG: --cloud-provider="external" Oct 30 17:57:43.223103 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.215986 2565 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 17:57:43.223103 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.215992 2565 flags.go:64] FLAG: --cluster-domain="" Oct 30 17:57:43.223103 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.215997 2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 17:57:43.223103 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216001 2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 17:57:43.223764 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216007 2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 17:57:43.223764 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216012 2565 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 17:57:43.223764 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216019 2565 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 17:57:43.223764 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216024 2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 17:57:43.223764 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216029 2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 17:57:43.223764 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216034 2565 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 17:57:43.223764 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216039 2565 flags.go:64] FLAG: --contention-profiling="false" Oct 30 17:57:43.223764 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216044 2565 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 17:57:43.223764 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216048 2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 17:57:43.223764 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216054 2565 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 17:57:43.223764 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216059 2565 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 17:57:43.223764 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216066 2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 17:57:43.223764 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216071 2565 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 17:57:43.223764 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216077 2565 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 17:57:43.223764 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216081 2565 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 17:57:43.223764 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216086 2565 flags.go:64] FLAG: --enable-server="true" Oct 30 17:57:43.223764 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216091 2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 17:57:43.223764 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216097 2565 flags.go:64] FLAG: --event-burst="100" Oct 30 17:57:43.223764 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216103 2565 flags.go:64] FLAG: --event-qps="50" Oct 30 17:57:43.223764 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216108 2565 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 17:57:43.223764 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216114 2565 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 17:57:43.223764 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216119 2565 flags.go:64] FLAG: --eviction-hard="" Oct 30 17:57:43.223764 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216126 2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 17:57:43.223764 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216151 2565 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 17:57:43.224365 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216156 2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 17:57:43.224365 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216161 2565 flags.go:64] FLAG: --eviction-soft="" Oct 30 17:57:43.224365 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216166 2565 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 17:57:43.224365 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216171 2565 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 17:57:43.224365 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216176 2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 17:57:43.224365 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216181 2565 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 17:57:43.224365 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216186 2565 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 17:57:43.224365 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216191 2565 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 17:57:43.224365 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216196 2565 flags.go:64] FLAG: --feature-gates="" Oct 30 17:57:43.224365 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216203 2565 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 17:57:43.224365 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216208 2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 17:57:43.224365 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216213 2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 17:57:43.224365 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216218 2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 17:57:43.224365 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216223 2565 flags.go:64] FLAG: --healthz-port="10248" Oct 30 17:57:43.224365 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216229 2565 flags.go:64] FLAG: --help="false" Oct 30 17:57:43.224365 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216233 2565 flags.go:64] FLAG: --hostname-override="ip-10-0-138-176.ec2.internal" Oct 30 17:57:43.224365 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216238 2565 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 17:57:43.224365 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216244 2565 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 17:57:43.224365 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216248 2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 17:57:43.224365 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216254 2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 17:57:43.224365 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216260 2565 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 17:57:43.224365 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216265 2565 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 17:57:43.224365 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216270 2565 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 17:57:43.225087 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216274 2565 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 17:57:43.225087 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216279 2565 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 17:57:43.225087 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216283 2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 17:57:43.225087 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216289 2565 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 17:57:43.225087 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216293 2565 flags.go:64] FLAG: --kube-reserved="" Oct 30 17:57:43.225087 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216298 2565 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 17:57:43.225087 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216303 2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 17:57:43.225087 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216308 2565 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 17:57:43.225087 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216313 2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 17:57:43.225087 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216317 2565 flags.go:64] FLAG: --lock-file="" Oct 30 17:57:43.225087 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216322 2565 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 17:57:43.225087 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216327 2565 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 17:57:43.225087 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216331 2565 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 17:57:43.225087 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216341 2565 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 17:57:43.225087 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216346 2565 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 17:57:43.225087 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216352 2565 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 17:57:43.225087 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216356 2565 flags.go:64] FLAG: --logging-format="text" Oct 30 17:57:43.225087 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216362 2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 17:57:43.225087 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216367 2565 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 17:57:43.225087 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216372 2565 flags.go:64] FLAG: --manifest-url="" Oct 30 17:57:43.225087 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216376 2565 flags.go:64] FLAG: --manifest-url-header="" Oct 30 17:57:43.225087 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216383 2565 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 17:57:43.225087 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216388 2565 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 17:57:43.225087 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216395 2565 flags.go:64] FLAG: --max-pods="110" Oct 30 17:57:43.225087 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216400 2565 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 17:57:43.225931 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216404 2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 17:57:43.225931 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216409 2565 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 17:57:43.225931 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216414 2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 17:57:43.225931 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216430 2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 17:57:43.225931 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216435 2565 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 17:57:43.225931 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216440 2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 17:57:43.225931 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216452 2565 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 17:57:43.225931 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216457 2565 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 17:57:43.225931 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216462 2565 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 17:57:43.225931 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216467 2565 flags.go:64] FLAG: --pod-cidr="" Oct 30 17:57:43.225931 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216472 2565 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 17:57:43.225931 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216476 2565 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 17:57:43.225931 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216481 2565 flags.go:64] FLAG: --pods-per-core="0" Oct 30 17:57:43.225931 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216486 2565 flags.go:64] FLAG: --port="10250" Oct 30 17:57:43.225931 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216491 2565 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 17:57:43.225931 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216496 2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c6984eabe761abec" Oct 30 17:57:43.225931 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216501 2565 flags.go:64] FLAG: --qos-reserved="" Oct 30 17:57:43.225931 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216506 2565 flags.go:64] FLAG: --read-only-port="10255" Oct 30 17:57:43.225931 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216511 2565 flags.go:64] FLAG: --register-node="true" Oct 30 17:57:43.225931 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216516 2565 flags.go:64] FLAG: --register-with-taints="" Oct 30 17:57:43.225931 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216522 2565 flags.go:64] FLAG: --registry-burst="10" Oct 30 17:57:43.225931 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216527 2565 flags.go:64] FLAG: --registry-qps="5" Oct 30 17:57:43.225931 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216531 2565 flags.go:64] FLAG: --reserved-cpus="" Oct 30 17:57:43.225931 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216537 2565 flags.go:64] FLAG: --reserved-memory="" Oct 30 17:57:43.225931 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216543 2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 17:57:43.225931 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216548 2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 17:57:43.226602 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216553 2565 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 17:57:43.226602 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216558 2565 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 17:57:43.226602 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216562 2565 flags.go:64] FLAG: --runonce="false" Oct 30 17:57:43.226602 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216567 2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 17:57:43.226602 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216572 2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 17:57:43.226602 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216577 2565 flags.go:64] FLAG: --seccomp-default="false" Oct 30 17:57:43.226602 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216582 2565 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 17:57:43.226602 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216586 2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 17:57:43.226602 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216591 2565 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 17:57:43.226602 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216597 2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 17:57:43.226602 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216602 2565 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 17:57:43.226602 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216606 2565 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 17:57:43.226602 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216611 2565 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 17:57:43.226602 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216616 2565 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 17:57:43.226602 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216620 2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 17:57:43.226602 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216625 2565 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 17:57:43.226602 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216630 2565 flags.go:64] FLAG: --system-cgroups="" Oct 30 17:57:43.226602 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216635 2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 17:57:43.226602 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216644 2565 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 17:57:43.226602 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216648 2565 flags.go:64] FLAG: --tls-cert-file="" Oct 30 17:57:43.226602 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216653 2565 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 17:57:43.226602 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216659 2565 flags.go:64] FLAG: --tls-min-version="" Oct 30 17:57:43.226602 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216664 2565 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 17:57:43.226602 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216668 2565 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 17:57:43.226602 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216673 2565 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 17:57:43.227233 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216678 2565 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 17:57:43.227233 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216683 2565 flags.go:64] FLAG: --v="2" Oct 30 17:57:43.227233 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216690 2565 flags.go:64] FLAG: --version="false" Oct 30 17:57:43.227233 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216696 2565 flags.go:64] FLAG: --vmodule="" Oct 30 17:57:43.227233 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216702 2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 17:57:43.227233 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.216711 2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 17:57:43.227233 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.216932 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:57:43.227233 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.216939 2565 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:57:43.227233 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.216943 2565 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:57:43.227233 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.216948 2565 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:57:43.227233 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.216954 2565 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:57:43.227233 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.216958 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:57:43.227233 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.216962 2565 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:57:43.227233 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.216966 2565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:57:43.227233 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.216970 2565 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:57:43.227233 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.216975 2565 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:57:43.227233 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.216979 2565 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:57:43.227233 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.216984 2565 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:57:43.227233 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.216988 2565 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:57:43.227233 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.216992 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:57:43.227233 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.216997 2565 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:57:43.227730 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217001 2565 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:57:43.227730 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217005 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:57:43.227730 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217009 2565 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:57:43.227730 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217014 2565 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:57:43.227730 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217018 2565 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:57:43.227730 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217022 2565 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:57:43.227730 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217026 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:57:43.227730 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217030 2565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:57:43.227730 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217034 2565 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:57:43.227730 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217039 2565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:57:43.227730 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217043 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:57:43.227730 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217047 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:57:43.227730 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217051 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:57:43.227730 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217056 2565 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:57:43.227730 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217060 2565 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:57:43.227730 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217064 2565 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:57:43.227730 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217071 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:57:43.227730 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217076 2565 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:57:43.227730 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217080 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:57:43.227730 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217085 2565 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:57:43.228216 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217089 2565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:57:43.228216 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217094 2565 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:57:43.228216 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217098 2565 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:57:43.228216 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217103 2565 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:57:43.228216 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217107 2565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:57:43.228216 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217111 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:57:43.228216 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217115 2565 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:57:43.228216 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217119 2565 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:57:43.228216 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217123 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:57:43.228216 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217144 2565 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:57:43.228216 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217149 2565 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:57:43.228216 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217153 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:57:43.228216 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217157 2565 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:57:43.228216 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217161 2565 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:57:43.228216 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217165 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:57:43.228216 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217170 2565 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:57:43.228216 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217174 2565 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:57:43.228216 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217178 2565 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:57:43.228216 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217184 2565 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:57:43.228674 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217192 2565 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:57:43.228674 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217197 2565 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:57:43.228674 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217202 2565 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:57:43.228674 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217206 2565 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:57:43.228674 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217211 2565 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:57:43.228674 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217215 2565 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:57:43.228674 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217219 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:57:43.228674 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217223 2565 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:57:43.228674 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217228 2565 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:57:43.228674 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217234 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:57:43.228674 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217238 2565 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:57:43.228674 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217244 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:57:43.228674 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217249 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:57:43.228674 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217253 2565 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:57:43.228674 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217259 2565 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:57:43.228674 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217263 2565 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:57:43.228674 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217267 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:57:43.228674 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217272 2565 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:57:43.228674 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217276 2565 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:57:43.229118 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217281 2565 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:57:43.229118 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217285 2565 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:57:43.229118 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217289 2565 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:57:43.229118 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217293 2565 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:57:43.229118 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217298 2565 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:57:43.229118 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217302 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:57:43.229118 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217306 2565 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:57:43.229118 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217310 2565 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:57:43.229118 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217314 2565 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:57:43.229118 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217319 2565 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:57:43.229118 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217323 2565 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:57:43.229118 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217327 2565 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:57:43.229118 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217332 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:57:43.229118 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217338 2565 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:57:43.229118 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217345 2565 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:57:43.229118 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217351 2565 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:57:43.229118 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217355 2565 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:57:43.229118 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217360 2565 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:57:43.229118 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217364 2565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:57:43.229118 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217368 2565 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:57:43.229608 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217373 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:57:43.229608 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217377 2565 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:57:43.229608 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217382 2565 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:57:43.229608 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217386 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:57:43.229608 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217391 2565 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:57:43.229608 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217396 2565 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:57:43.229608 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217400 2565 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:57:43.229608 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217405 2565 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:57:43.229608 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217410 2565 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:57:43.229608 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217414 2565 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:57:43.229608 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217418 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:57:43.229608 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217422 2565 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:57:43.229608 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217426 2565 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:57:43.229608 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.217471 2565 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:57:43.229608 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217537 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:57:43.229608 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217544 2565 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:57:43.229978 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217550 2565 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:57:43.229978 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217554 2565 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:57:43.229978 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217559 2565 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:57:43.229978 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217563 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:57:43.229978 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217568 2565 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:57:43.229978 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217572 2565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:57:43.229978 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217576 2565 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:57:43.229978 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217580 2565 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:57:43.229978 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217584 2565 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:57:43.229978 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217587 2565 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:57:43.229978 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217591 2565 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:57:43.229978 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217595 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:57:43.229978 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217600 2565 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:57:43.229978 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217603 2565 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:57:43.229978 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217607 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:57:43.229978 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217612 2565 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:57:43.229978 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217616 2565 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:57:43.229978 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217620 2565 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:57:43.229978 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217627 2565 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:57:43.229978 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217631 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:57:43.230473 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217635 2565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:57:43.230473 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217639 2565 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:57:43.230473 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217645 2565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:57:43.230473 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217649 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:57:43.230473 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217655 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:57:43.230473 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217659 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:57:43.230473 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217664 2565 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:57:43.230473 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217668 2565 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:57:43.230473 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217672 2565 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:57:43.230473 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217676 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:57:43.230473 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217680 2565 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:57:43.230473 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217684 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:57:43.230473 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217688 2565 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:57:43.230473 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217692 2565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:57:43.230473 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217696 2565 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:57:43.230473 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217701 2565 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:57:43.230473 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217704 2565 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:57:43.230473 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217709 2565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:57:43.230473 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217713 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:57:43.230917 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217717 2565 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:57:43.230917 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217721 2565 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:57:43.230917 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217725 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:57:43.230917 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217729 2565 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:57:43.230917 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217733 2565 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:57:43.230917 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217737 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:57:43.230917 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217741 2565 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:57:43.230917 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217746 2565 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:57:43.230917 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217750 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:57:43.230917 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217754 2565 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:57:43.230917 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217758 2565 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:57:43.230917 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217764 2565 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:57:43.230917 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217770 2565 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:57:43.230917 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217777 2565 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:57:43.230917 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217783 2565 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:57:43.230917 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217787 2565 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:57:43.230917 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217793 2565 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:57:43.230917 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217799 2565 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:57:43.230917 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217803 2565 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:57:43.231386 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217808 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:57:43.231386 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217812 2565 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:57:43.231386 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217817 2565 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:57:43.231386 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217822 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:57:43.231386 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217826 2565 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:57:43.231386 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217830 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:57:43.231386 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217834 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:57:43.231386 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217839 2565 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:57:43.231386 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217843 2565 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:57:43.231386 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217847 2565 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:57:43.231386 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217852 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:57:43.231386 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217856 2565 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:57:43.231386 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217860 2565 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:57:43.231386 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217864 2565 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:57:43.231386 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217869 2565 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:57:43.231386 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217873 2565 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:57:43.231386 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217877 2565 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:57:43.231386 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217881 2565 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:57:43.231386 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217885 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:57:43.231386 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217890 2565 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:57:43.231928 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217894 2565 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:57:43.231928 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217898 2565 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:57:43.231928 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217902 2565 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:57:43.231928 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217906 2565 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:57:43.231928 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217912 2565 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:57:43.231928 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217916 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:57:43.231928 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217923 2565 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:57:43.231928 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217929 2565 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:57:43.231928 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217934 2565 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:57:43.231928 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217939 2565 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:57:43.231928 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217946 2565 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:57:43.231928 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217950 2565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:57:43.231928 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217955 2565 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:57:43.231928 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217959 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:57:43.231928 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217963 2565 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:57:43.231928 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217968 2565 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:57:43.231928 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217972 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:57:43.231928 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217976 2565 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:57:43.231928 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217981 2565 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:57:43.231928 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217986 2565 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:57:43.232446 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217990 2565 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:57:43.232446 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217994 2565 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:57:43.232446 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.217998 2565 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:57:43.232446 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.218002 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:57:43.232446 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.218006 2565 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:57:43.232446 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.218010 2565 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:57:43.232446 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.226625 2565 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 17:57:43.232446 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.226643 2565 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 17:57:43.232446 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.226660 2565 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 17:57:43.232446 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.226665 2565 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 17:57:43.232446 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226725 2565 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:57:43.232446 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226730 2565 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:57:43.232446 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226734 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:57:43.232446 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226736 2565 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:57:43.232446 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226739 2565 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:57:43.232446 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226742 2565 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:57:43.232446 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226744 2565 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:57:43.232446 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226747 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:57:43.232446 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226750 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:57:43.232882 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226753 2565 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:57:43.232882 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226755 2565 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:57:43.232882 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226758 2565 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:57:43.232882 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226761 2565 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:57:43.232882 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226763 2565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:57:43.232882 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226766 2565 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:57:43.232882 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226769 2565 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:57:43.232882 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226771 2565 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:57:43.232882 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226774 2565 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:57:43.232882 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226776 2565 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:57:43.232882 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226779 2565 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:57:43.232882 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226781 2565 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:57:43.232882 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226784 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:57:43.232882 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226786 2565 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:57:43.232882 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226789 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:57:43.232882 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226791 2565 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:57:43.232882 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226793 2565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:57:43.232882 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226796 2565 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:57:43.232882 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226799 2565 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:57:43.232882 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226802 2565 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:57:43.233427 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226806 2565 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:57:43.233427 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226809 2565 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:57:43.233427 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226811 2565 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:57:43.233427 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226813 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:57:43.233427 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226816 2565 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:57:43.233427 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226819 2565 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:57:43.233427 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226822 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:57:43.233427 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226825 2565 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:57:43.233427 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226827 2565 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:57:43.233427 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226830 2565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:57:43.233427 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226832 2565 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:57:43.233427 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226835 2565 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:57:43.233427 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226837 2565 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:57:43.233427 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226840 2565 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:57:43.233427 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226842 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:57:43.233427 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226845 2565 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:57:43.233427 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226847 2565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:57:43.233427 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226850 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:57:43.233427 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226853 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:57:43.233427 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226855 2565 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:57:43.233885 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226857 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:57:43.233885 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226860 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:57:43.233885 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226863 2565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:57:43.233885 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226866 2565 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:57:43.233885 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226868 2565 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:57:43.233885 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226871 2565 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:57:43.233885 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226873 2565 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:57:43.233885 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226876 2565 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:57:43.233885 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226880 2565 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:57:43.233885 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226884 2565 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:57:43.233885 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226887 2565 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:57:43.233885 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226890 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:57:43.233885 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226893 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:57:43.233885 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226897 2565 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:57:43.233885 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226900 2565 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:57:43.233885 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226903 2565 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:57:43.233885 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226906 2565 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:57:43.233885 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226908 2565 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:57:43.233885 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226911 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:57:43.234385 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226914 2565 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:57:43.234385 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226916 2565 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:57:43.234385 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226919 2565 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:57:43.234385 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226921 2565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:57:43.234385 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226924 2565 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:57:43.234385 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226926 2565 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:57:43.234385 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226929 2565 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:57:43.234385 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226931 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:57:43.234385 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226934 2565 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:57:43.234385 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226936 2565 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:57:43.234385 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226939 2565 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:57:43.234385 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226946 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:57:43.234385 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226949 2565 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:57:43.234385 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226951 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:57:43.234385 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226954 2565 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:57:43.234385 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226956 2565 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:57:43.234385 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226959 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:57:43.234385 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226961 2565 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:57:43.234385 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226964 2565 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:57:43.234385 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226966 2565 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:57:43.234846 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226969 2565 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:57:43.234846 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226973 2565 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:57:43.234846 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226976 2565 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:57:43.234846 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226979 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:57:43.234846 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226982 2565 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:57:43.234846 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226985 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:57:43.234846 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226988 2565 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:57:43.234846 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226991 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:57:43.234846 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226993 2565 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:57:43.234846 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226995 2565 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:57:43.234846 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.226998 2565 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:57:43.234846 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227000 2565 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:57:43.234846 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227003 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:57:43.234846 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227006 2565 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:57:43.234846 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227008 2565 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:57:43.234846 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227010 2565 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:57:43.234846 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227013 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:57:43.234846 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227015 2565 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:57:43.235344 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.227043 2565 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:57:43.235344 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227208 2565 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:57:43.235344 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227214 2565 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:57:43.235344 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227217 2565 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:57:43.235344 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227220 2565 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:57:43.235344 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227222 2565 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:57:43.235344 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227226 2565 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:57:43.235344 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227228 2565 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:57:43.235344 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227231 2565 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:57:43.235344 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227233 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:57:43.235344 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227236 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:57:43.235344 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227239 2565 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:57:43.235344 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227242 2565 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:57:43.235344 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227245 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:57:43.235344 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227248 2565 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:57:43.235344 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227251 2565 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:57:43.235722 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227254 2565 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:57:43.235722 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227256 2565 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:57:43.235722 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227259 2565 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:57:43.235722 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227262 2565 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:57:43.235722 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227264 2565 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:57:43.235722 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227267 2565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:57:43.235722 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227270 2565 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:57:43.235722 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227272 2565 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:57:43.235722 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227274 2565 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:57:43.235722 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227277 2565 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:57:43.235722 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227279 2565 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:57:43.235722 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227282 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:57:43.235722 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227285 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:57:43.235722 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227288 2565 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:57:43.235722 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227290 2565 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:57:43.235722 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227292 2565 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:57:43.235722 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227295 2565 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:57:43.235722 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227298 2565 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:57:43.235722 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227300 2565 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:57:43.236276 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227303 2565 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:57:43.236276 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227305 2565 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:57:43.236276 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227308 2565 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:57:43.236276 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227310 2565 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:57:43.236276 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227314 2565 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:57:43.236276 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227316 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:57:43.236276 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227319 2565 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:57:43.236276 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227322 2565 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:57:43.236276 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227326 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:57:43.236276 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227329 2565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:57:43.236276 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227333 2565 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:57:43.236276 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227336 2565 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:57:43.236276 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227340 2565 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:57:43.236276 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227343 2565 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:57:43.236276 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227346 2565 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:57:43.236276 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227349 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:57:43.236276 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227352 2565 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:57:43.236276 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227355 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:57:43.236276 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227358 2565 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:57:43.236854 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227361 2565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:57:43.236854 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227364 2565 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:57:43.236854 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227367 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:57:43.236854 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227369 2565 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:57:43.236854 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227372 2565 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:57:43.236854 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227375 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:57:43.236854 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227377 2565 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:57:43.236854 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227379 2565 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:57:43.236854 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227382 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:57:43.236854 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227384 2565 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:57:43.236854 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227390 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:57:43.236854 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227392 2565 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:57:43.236854 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227395 2565 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:57:43.236854 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227397 2565 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:57:43.236854 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227399 2565 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:57:43.236854 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227402 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:57:43.236854 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227404 2565 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:57:43.236854 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227407 2565 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:57:43.236854 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227410 2565 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:57:43.236854 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227412 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:57:43.237346 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227415 2565 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:57:43.237346 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227417 2565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:57:43.237346 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227420 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:57:43.237346 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227422 2565 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:57:43.237346 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227425 2565 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:57:43.237346 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227427 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:57:43.237346 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227430 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:57:43.237346 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227432 2565 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:57:43.237346 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227435 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:57:43.237346 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227437 2565 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:57:43.237346 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227439 2565 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:57:43.237346 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227442 2565 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:57:43.237346 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227444 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:57:43.237346 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227447 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:57:43.237346 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227449 2565 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:57:43.237346 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227452 2565 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:57:43.237346 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227454 2565 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:57:43.237346 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227456 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:57:43.237346 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227459 2565 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:57:43.237346 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227461 2565 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:57:43.237872 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227464 2565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:57:43.237872 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227466 2565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:57:43.237872 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227470 2565 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:57:43.237872 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227472 2565 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:57:43.237872 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227475 2565 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:57:43.237872 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227477 2565 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:57:43.237872 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227479 2565 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:57:43.237872 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227482 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:57:43.237872 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227484 2565 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:57:43.237872 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227487 2565 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:57:43.237872 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227489 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:57:43.237872 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227493 2565 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:57:43.237872 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227495 2565 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:57:43.237872 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.227519 2565 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:57:43.237872 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227552 2565 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:57:43.237872 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227558 2565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:57:43.238308 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227561 2565 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:57:43.238308 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227564 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:57:43.238308 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227567 2565 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:57:43.238308 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227569 2565 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:57:43.238308 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227572 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:57:43.238308 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227574 2565 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:57:43.238308 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227577 2565 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:57:43.238308 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227580 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:57:43.238308 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227582 2565 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:57:43.238308 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227585 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:57:43.238308 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227587 2565 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:57:43.238308 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227589 2565 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:57:43.238308 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227592 2565 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:57:43.238308 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227595 2565 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:57:43.238308 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227597 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:57:43.238308 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227600 2565 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:57:43.238308 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227602 2565 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:57:43.238308 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227605 2565 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:57:43.238308 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227608 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:57:43.238308 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227611 2565 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:57:43.238935 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227613 2565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:57:43.238935 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227616 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:57:43.238935 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227619 2565 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:57:43.238935 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227621 2565 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:57:43.238935 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227624 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:57:43.238935 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227626 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:57:43.238935 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227629 2565 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:57:43.238935 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227631 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:57:43.238935 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227635 2565 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:57:43.238935 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227637 2565 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:57:43.238935 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227640 2565 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:57:43.238935 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227644 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:57:43.238935 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227647 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:57:43.238935 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227650 2565 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:57:43.238935 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227652 2565 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:57:43.238935 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227655 2565 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:57:43.238935 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227657 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:57:43.238935 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227659 2565 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:57:43.238935 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227662 2565 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:57:43.238935 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227664 2565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:57:43.239457 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227667 2565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:57:43.239457 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227669 2565 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:57:43.239457 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227672 2565 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:57:43.239457 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227675 2565 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:57:43.239457 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227677 2565 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:57:43.239457 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227679 2565 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:57:43.239457 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227682 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:57:43.239457 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227685 2565 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:57:43.239457 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227688 2565 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:57:43.239457 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227690 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:57:43.239457 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227694 2565 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:57:43.239457 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227696 2565 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:57:43.239457 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227699 2565 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:57:43.239457 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227701 2565 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:57:43.239457 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227704 2565 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:57:43.239457 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227707 2565 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:57:43.239457 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227710 2565 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:57:43.239457 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227712 2565 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:57:43.239457 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227714 2565 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:57:43.239457 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227717 2565 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:57:43.239930 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227719 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:57:43.239930 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227723 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:57:43.239930 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227726 2565 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:57:43.239930 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227730 2565 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:57:43.239930 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227732 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:57:43.239930 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227735 2565 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:57:43.239930 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227738 2565 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:57:43.239930 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227740 2565 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:57:43.239930 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227743 2565 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:57:43.239930 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227745 2565 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:57:43.239930 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227748 2565 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:57:43.239930 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227750 2565 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:57:43.239930 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227752 2565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:57:43.239930 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227755 2565 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:57:43.239930 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227757 2565 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:57:43.239930 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227760 2565 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:57:43.239930 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227762 2565 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:57:43.239930 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227765 2565 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:57:43.239930 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227767 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:57:43.239930 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227769 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:57:43.240563 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227772 2565 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:57:43.240563 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227774 2565 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:57:43.240563 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227778 2565 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:57:43.240563 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227780 2565 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:57:43.240563 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227783 2565 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:57:43.240563 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227785 2565 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:57:43.240563 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227787 2565 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:57:43.240563 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227790 2565 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:57:43.240563 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227792 2565 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:57:43.240563 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227794 2565 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:57:43.240563 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227797 2565 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:57:43.240563 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227799 2565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:57:43.240563 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227802 2565 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:57:43.240563 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227805 2565 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:57:43.240563 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227811 2565 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:57:43.240563 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227815 2565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:57:43.240563 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227819 2565 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:57:43.240563 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227822 2565 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:57:43.240983 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227826 2565 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:57:43.240983 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227828 2565 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:57:43.240983 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227831 2565 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:57:43.240983 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227835 2565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:57:43.240983 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227837 2565 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:57:43.240983 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:43.227839 2565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:57:43.240983 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.227899 2565 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 17:57:43.240983 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.231280 2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 17:57:43.240983 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.232216 2565 server.go:1014] "Starting client certificate rotation" Oct 30 17:57:43.240983 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.232315 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 17:57:43.240983 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.232351 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 17:57:43.240983 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.234005 2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 17:57:43.240983 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.235127 2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 17:57:43.245853 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.245837 2565 log.go:25] "Validated CRI v1 runtime API" Oct 30 17:57:43.249293 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.249281 2565 log.go:25] "Validated CRI v1 image API" Oct 30 17:57:43.250072 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.250049 2565 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 17:57:43.253763 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.253744 2565 fs.go:135] Filesystem UUIDs: map[6ed5bd6d-db36-4bb3-a60a-f4a4624e9220:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8d22543c-06b7-4209-b7be-433a475b3533:/dev/nvme0n1p4] Oct 30 17:57:43.253818 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.253763 2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 17:57:43.258796 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.258673 2565 manager.go:217] Machine: {Timestamp:2026-10-30 17:57:43.257283359 +0000 UTC m=+0.318661805 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3108418 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20b0856e4023d926525fe62aced39f SystemUUID:ec20b085-6e40-23d9-2652-5fe62aced39f BootID:e92f1dbc-5cb0-4908-a2bd-5927dbebbac5 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:fd:19:bc:13 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:fd:19:bc:13 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:72:cf:b0:75:8c:54 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 17:57:43.259261 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.259250 2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 17:57:43.259347 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.259335 2565 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 17:57:43.260400 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.260372 2565 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 17:57:43.260543 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.260404 2565 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-176.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 17:57:43.260586 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.260546 2565 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 17:57:43.260586 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.260554 2565 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 17:57:43.260638 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.260590 2565 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 17:57:43.260638 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.260608 2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 17:57:43.261317 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.261307 2565 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 17:57:43.265690 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.265624 2565 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 17:57:43.265820 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.265808 2565 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 17:57:43.266953 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.266562 2565 kubelet.go:493] "Attempting to sync node with API server" Oct 30 17:57:43.266953 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.266587 2565 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 17:57:43.266953 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.266606 2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 17:57:43.266953 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.266620 2565 kubelet.go:405] "Adding apiserver pod source" Oct 30 17:57:43.266953 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.266633 2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 17:57:43.267217 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.267198 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 17:57:43.269472 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.269448 2565 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 17:57:43.274260 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.274245 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 17:57:43.274966 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.274953 2565 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 17:57:43.275013 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.274985 2565 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 17:57:43.275179 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.275169 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 17:57:43.275219 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.275185 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 17:57:43.275219 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.275191 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 17:57:43.275219 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.275197 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 17:57:43.275219 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.275202 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 17:57:43.275219 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.275208 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 17:57:43.275671 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.275663 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 17:57:43.275700 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.275673 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 17:57:43.275700 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.275680 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 17:57:43.275700 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.275686 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 17:57:43.275700 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.275694 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 17:57:43.275817 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.275704 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 17:57:43.276745 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.276734 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 17:57:43.276781 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.276747 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 17:57:43.280395 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.280382 2565 server.go:1263] "Started kubelet" Oct 30 17:57:43.280519 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.280477 2565 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 17:57:43.280595 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.280472 2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 17:57:43.280643 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.280612 2565 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 17:57:43.281074 ip-10-0-138-176 systemd[1]: Started Kubernetes Kubelet. Oct 30 17:57:43.281618 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.281564 2565 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 17:57:43.283237 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.283213 2565 server.go:317] "Adding debug handlers to kubelet server" Oct 30 17:57:43.287708 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.287690 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 17:57:43.288097 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.288077 2565 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 17:57:43.288279 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:43.288257 2565 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-176.ec2.internal\" not found" Oct 30 17:57:43.288793 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.288644 2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 17:57:43.288793 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.288746 2565 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 17:57:43.288793 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.288760 2565 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 17:57:43.288938 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.288822 2565 reconstruct.go:97] "Volume reconstruction finished" Oct 30 17:57:43.288938 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.288831 2565 reconciler.go:29] "Reconciler: start to sync state" Oct 30 17:57:43.289024 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:43.288983 2565 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 17:57:43.289115 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.289100 2565 factory.go:55] Registering systemd factory Oct 30 17:57:43.289171 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.289120 2565 factory.go:223] Registration of the systemd container factory successfully Oct 30 17:57:43.289358 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.289345 2565 factory.go:153] Registering CRI-O factory Oct 30 17:57:43.289415 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.289360 2565 factory.go:223] Registration of the crio container factory successfully Oct 30 17:57:43.289415 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.289403 2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 17:57:43.289415 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.289417 2565 factory.go:103] Registering Raw factory Oct 30 17:57:43.289527 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.289428 2565 manager.go:1199] Started watching for new ooms in manager Oct 30 17:57:43.290418 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.290403 2565 manager.go:319] Starting recovery of all containers Oct 30 17:57:43.302787 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.302770 2565 manager.go:324] Recovery completed Oct 30 17:57:43.307383 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.307368 2565 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:57:43.309259 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.309239 2565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:57:43.309319 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.309280 2565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:57:43.309319 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.309303 2565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:57:43.309924 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.309907 2565 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:57:43.309978 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.309901 2565 cpu_manager.go:226] "Starting" policy="none" Oct 30 17:57:43.309978 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.309940 2565 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 17:57:43.310033 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.309970 2565 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 17:57:43.310680 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.310663 2565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:57:43.310719 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.310698 2565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:57:43.310719 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.310707 2565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:57:43.312374 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.312358 2565 policy_none.go:50] "Start" Oct 30 17:57:43.312437 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.312377 2565 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 17:57:43.312437 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.312387 2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 17:57:43.314512 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.314496 2565 policy_none.go:44] "Start" Oct 30 17:57:43.318735 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.318609 2565 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 17:57:43.322287 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.322267 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4f6cz" Oct 30 17:57:43.324878 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:43.324860 2565 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-176.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 17:57:43.324948 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.324935 2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-176.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 17:57:43.325738 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:43.324814 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-176.ec2.internal.18e36064f106fa0b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-176.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-176.ec2.internal,},FirstTimestamp:2026-10-30 17:57:43.280351755 +0000 UTC m=+0.341730198,LastTimestamp:2026-10-30 17:57:43.280351755 +0000 UTC m=+0.341730198,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-176.ec2.internal,}" Oct 30 17:57:43.331910 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.331889 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4f6cz" Oct 30 17:57:43.333609 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:43.333585 2565 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 17:57:43.333680 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:43.333567 2565 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-176.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 17:57:43.333680 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:43.333587 2565 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 17:57:43.350744 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.350702 2565 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 17:57:43.354642 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.354628 2565 manager.go:341] "Starting Device Plugin manager" Oct 30 17:57:43.354704 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:43.354662 2565 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 17:57:43.354704 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.354672 2565 server.go:87] "Starting device plugin registration server" Oct 30 17:57:43.354966 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.354954 2565 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 17:57:43.355007 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.354969 2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 17:57:43.355128 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.355108 2565 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 17:57:43.355219 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.355205 2565 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 17:57:43.355219 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.355216 2565 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 17:57:43.355683 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:43.355667 2565 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 17:57:43.355749 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:43.355710 2565 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-176.ec2.internal\" not found" Oct 30 17:57:43.439050 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.439007 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 17:57:43.441175 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.440313 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 17:57:43.441175 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.440336 2565 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 17:57:43.441175 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.440355 2565 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 17:57:43.441175 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:43.440388 2565 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 17:57:43.453542 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.453520 2565 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:57:43.455630 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.455615 2565 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:57:43.456387 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.456372 2565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:57:43.456451 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.456400 2565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:57:43.456451 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.456410 2565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:57:43.456451 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.456440 2565 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-176.ec2.internal" Oct 30 17:57:43.475123 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.475079 2565 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-176.ec2.internal" Oct 30 17:57:43.475123 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:43.475103 2565 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-176.ec2.internal\": node \"ip-10-0-138-176.ec2.internal\" not found" Oct 30 17:57:43.530406 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:43.530380 2565 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-176.ec2.internal\" not found" Oct 30 17:57:43.541968 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.541945 2565 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-176.ec2.internal"] Oct 30 17:57:43.542022 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.542011 2565 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:57:43.543014 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.542996 2565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:57:43.543126 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.543026 2565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:57:43.543126 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.543039 2565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:57:43.545128 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.545116 2565 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:57:43.545284 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.545271 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal" Oct 30 17:57:43.545319 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.545302 2565 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:57:43.545875 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.545861 2565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:57:43.545948 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.545888 2565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:57:43.545948 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.545901 2565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:57:43.545948 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.545913 2565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:57:43.545948 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.545932 2565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:57:43.545948 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.545942 2565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:57:43.548043 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.548029 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-176.ec2.internal" Oct 30 17:57:43.548086 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.548057 2565 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:57:43.548802 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.548787 2565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:57:43.548874 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.548813 2565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:57:43.548874 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.548826 2565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:57:43.567490 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:43.567472 2565 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-176.ec2.internal\" not found" node="ip-10-0-138-176.ec2.internal" Oct 30 17:57:43.570411 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:43.570391 2565 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-176.ec2.internal\" not found" node="ip-10-0-138-176.ec2.internal" Oct 30 17:57:43.590861 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.590830 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bfd89a876f67f9d878ac0d1431582688-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal\" (UID: \"bfd89a876f67f9d878ac0d1431582688\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal" Oct 30 17:57:43.590947 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.590864 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfd89a876f67f9d878ac0d1431582688-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal\" (UID: \"bfd89a876f67f9d878ac0d1431582688\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal" Oct 30 17:57:43.590947 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.590885 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9e81bd78300430e51b01d30caa1123e7-config\") pod \"kube-apiserver-proxy-ip-10-0-138-176.ec2.internal\" (UID: \"9e81bd78300430e51b01d30caa1123e7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-176.ec2.internal" Oct 30 17:57:43.630547 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:43.630514 2565 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-176.ec2.internal\" not found" Oct 30 17:57:43.691418 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.691386 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bfd89a876f67f9d878ac0d1431582688-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal\" (UID: \"bfd89a876f67f9d878ac0d1431582688\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal" Oct 30 17:57:43.691507 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.691423 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfd89a876f67f9d878ac0d1431582688-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal\" (UID: \"bfd89a876f67f9d878ac0d1431582688\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal" Oct 30 17:57:43.691507 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.691442 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9e81bd78300430e51b01d30caa1123e7-config\") pod \"kube-apiserver-proxy-ip-10-0-138-176.ec2.internal\" (UID: \"9e81bd78300430e51b01d30caa1123e7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-176.ec2.internal" Oct 30 17:57:43.691507 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.691496 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfd89a876f67f9d878ac0d1431582688-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal\" (UID: \"bfd89a876f67f9d878ac0d1431582688\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal" Oct 30 17:57:43.691597 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.691502 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bfd89a876f67f9d878ac0d1431582688-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal\" (UID: \"bfd89a876f67f9d878ac0d1431582688\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal" Oct 30 17:57:43.691597 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.691557 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9e81bd78300430e51b01d30caa1123e7-config\") pod \"kube-apiserver-proxy-ip-10-0-138-176.ec2.internal\" (UID: \"9e81bd78300430e51b01d30caa1123e7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-176.ec2.internal" Oct 30 17:57:43.730744 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:43.730683 2565 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-176.ec2.internal\" not found" Oct 30 17:57:43.831531 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:43.831500 2565 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-176.ec2.internal\" not found" Oct 30 17:57:43.870716 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.870679 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal" Oct 30 17:57:43.873732 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:43.873717 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-176.ec2.internal" Oct 30 17:57:43.931946 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:43.931903 2565 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-176.ec2.internal\" not found" Oct 30 17:57:44.032490 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:44.032400 2565 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-176.ec2.internal\" not found" Oct 30 17:57:44.132996 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:44.132963 2565 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-176.ec2.internal\" not found" Oct 30 17:57:44.232519 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:44.232482 2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 17:57:44.232938 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:44.232643 2565 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 17:57:44.233574 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:44.233556 2565 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-176.ec2.internal\" not found" Oct 30 17:57:44.334464 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:44.334397 2565 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-176.ec2.internal\" not found" Oct 30 17:57:44.337576 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:44.337546 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 17:52:43 +0000 UTC" deadline="2028-06-03 21:34:40.778342218 +0000 UTC" Oct 30 17:57:44.337576 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:44.337572 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13971h36m56.440773549s" Oct 30 17:57:44.352348 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:44.352329 2565 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:57:44.388685 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:44.388666 2565 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:57:44.393521 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:44.393480 2565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfd89a876f67f9d878ac0d1431582688.slice/crio-561db2dcb46df96c6309155b3811410b63e8b552e88a200ed32e8949742b7aa8 WatchSource:0}: Error finding container 561db2dcb46df96c6309155b3811410b63e8b552e88a200ed32e8949742b7aa8: Status 404 returned error can't find the container with id 561db2dcb46df96c6309155b3811410b63e8b552e88a200ed32e8949742b7aa8 Oct 30 17:57:44.393950 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:44.393929 2565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e81bd78300430e51b01d30caa1123e7.slice/crio-6639644759f355b3688140ef37b276a99cc9aaae3564b1358f929987a4e0af0e WatchSource:0}: Error finding container 6639644759f355b3688140ef37b276a99cc9aaae3564b1358f929987a4e0af0e: Status 404 returned error can't find the container with id 6639644759f355b3688140ef37b276a99cc9aaae3564b1358f929987a4e0af0e Oct 30 17:57:44.399117 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:44.399105 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:57:44.434663 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:44.434628 2565 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-176.ec2.internal\" not found" Oct 30 17:57:44.444114 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:44.444070 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal" event={"ID":"bfd89a876f67f9d878ac0d1431582688","Type":"ContainerStarted","Data":"561db2dcb46df96c6309155b3811410b63e8b552e88a200ed32e8949742b7aa8"} Oct 30 17:57:44.445207 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:44.445187 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-176.ec2.internal" event={"ID":"9e81bd78300430e51b01d30caa1123e7","Type":"ContainerStarted","Data":"6639644759f355b3688140ef37b276a99cc9aaae3564b1358f929987a4e0af0e"} Oct 30 17:57:44.457077 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:44.457057 2565 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:57:44.489148 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:44.489091 2565 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal" Oct 30 17:57:44.512337 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:44.512310 2565 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:57:44.513315 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:44.513298 2565 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-176.ec2.internal" Oct 30 17:57:44.534220 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:44.534197 2565 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:57:45.268687 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.268597 2565 apiserver.go:52] "Watching apiserver" Oct 30 17:57:45.289516 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.289479 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 17:57:45.313867 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.313831 2565 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 17:57:45.314327 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.314296 2565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-78h8z","openshift-dns/node-resolver-vp6gj","openshift-multus/network-metrics-daemon-9hrlx","openshift-network-operator/iptables-alerter-rxpmp","kube-system/konnectivity-agent-c84zp","kube-system/kube-apiserver-proxy-ip-10-0-138-176.ec2.internal","openshift-image-registry/node-ca-tsfch","openshift-multus/multus-wvg8l","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal","openshift-network-diagnostics/network-check-target-zpsg9","openshift-ovn-kubernetes/ovnkube-node-52rn6","kube-system/global-pull-secret-syncer-srgjm","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh","openshift-cluster-node-tuning-operator/tuned-6qd9s"] Oct 30 17:57:45.317294 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.317266 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tsfch" Oct 30 17:57:45.319659 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.319629 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" Oct 30 17:57:45.322468 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.322450 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9hrlx" Oct 30 17:57:45.322580 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:45.322522 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9hrlx" podUID="568da9c4-2c4f-4020-95ca-9f7ac7db7c00" Oct 30 17:57:45.325013 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.324991 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rxpmp" Oct 30 17:57:45.328251 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.328229 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-c84zp" Oct 30 17:57:45.332148 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.332125 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:57:45.334102 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.334082 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.337366 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.337338 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 17:57:45.337461 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.337436 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 17:57:45.337509 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.337493 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 17:57:45.337543 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.337437 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 17:57:45.337543 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.337519 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 17:57:45.337613 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.337562 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vzq26\"" Oct 30 17:57:45.337652 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.337627 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-78h8z" Oct 30 17:57:45.339390 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.339370 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vp6gj" Oct 30 17:57:45.342240 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.342192 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zpsg9" Oct 30 17:57:45.342418 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:45.342392 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zpsg9" podUID="bf5ca486-37a8-4da0-acbf-d389caf9f3e6" Oct 30 17:57:45.345121 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.345090 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.347752 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.347728 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-srgjm" Oct 30 17:57:45.347853 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:45.347803 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-srgjm" podUID="a24e24cc-a321-4a62-b023-7b015f155d55" Oct 30 17:57:45.350403 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.349940 2565 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 17:57:45.350403 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.350017 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kqfx2" Oct 30 17:57:45.389803 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.389781 2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 17:57:45.402313 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.402280 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-hostroot\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.402313 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.402310 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-run-systemd\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.402562 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.402328 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-host-cni-bin\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.402562 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.402353 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0af98be4-3c2f-413e-838e-f842854411f2-cni-binary-copy\") pod \"multus-additional-cni-plugins-78h8z\" (UID: \"0af98be4-3c2f-413e-838e-f842854411f2\") " pod="openshift-multus/multus-additional-cni-plugins-78h8z" Oct 30 17:57:45.402562 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.402380 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-run-ovn\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.402562 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.402413 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.402562 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.402484 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-ovn-node-metrics-cert\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.402562 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.402524 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a24e24cc-a321-4a62-b023-7b015f155d55-dbus\") pod \"global-pull-secret-syncer-srgjm\" (UID: \"a24e24cc-a321-4a62-b023-7b015f155d55\") " pod="kube-system/global-pull-secret-syncer-srgjm" Oct 30 17:57:45.402562 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.402554 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a24e24cc-a321-4a62-b023-7b015f155d55-original-pull-secret\") pod \"global-pull-secret-syncer-srgjm\" (UID: \"a24e24cc-a321-4a62-b023-7b015f155d55\") " pod="kube-system/global-pull-secret-syncer-srgjm" Oct 30 17:57:45.402932 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.402605 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/59229234-921c-4177-8b85-3e982194f625-agent-certs\") pod \"konnectivity-agent-c84zp\" (UID: \"59229234-921c-4177-8b85-3e982194f625\") " pod="kube-system/konnectivity-agent-c84zp" Oct 30 17:57:45.402932 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.402637 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-system-cni-dir\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.402932 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.402684 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-systemd-units\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.402932 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.402719 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a28935a6-1847-4f43-9e7a-332c6b3ce49f-iptables-alerter-script\") pod \"iptables-alerter-rxpmp\" (UID: \"a28935a6-1847-4f43-9e7a-332c6b3ce49f\") " pod="openshift-network-operator/iptables-alerter-rxpmp" Oct 30 17:57:45.402932 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.402745 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5sr2\" (UniqueName: \"kubernetes.io/projected/a28935a6-1847-4f43-9e7a-332c6b3ce49f-kube-api-access-m5sr2\") pod \"iptables-alerter-rxpmp\" (UID: \"a28935a6-1847-4f43-9e7a-332c6b3ce49f\") " pod="openshift-network-operator/iptables-alerter-rxpmp" Oct 30 17:57:45.402932 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.402775 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0af98be4-3c2f-413e-838e-f842854411f2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-78h8z\" (UID: \"0af98be4-3c2f-413e-838e-f842854411f2\") " pod="openshift-multus/multus-additional-cni-plugins-78h8z" Oct 30 17:57:45.402932 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.402801 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr4zd\" (UniqueName: \"kubernetes.io/projected/0772096e-be2c-449d-9bf6-c56cbd6c203a-kube-api-access-cr4zd\") pod \"node-resolver-vp6gj\" (UID: \"0772096e-be2c-449d-9bf6-c56cbd6c203a\") " pod="openshift-dns/node-resolver-vp6gj" Oct 30 17:57:45.402932 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.402853 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-multus-daemon-config\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.402932 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.402882 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-var-lib-openvswitch\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.402932 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.402909 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d37ce412-3bd6-4093-9d21-3b72d396c1bd-socket-dir\") pod \"aws-ebs-csi-driver-node-8wmlh\" (UID: \"d37ce412-3bd6-4093-9d21-3b72d396c1bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" Oct 30 17:57:45.402932 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.402933 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d37ce412-3bd6-4093-9d21-3b72d396c1bd-registration-dir\") pod \"aws-ebs-csi-driver-node-8wmlh\" (UID: \"d37ce412-3bd6-4093-9d21-3b72d396c1bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" Oct 30 17:57:45.403455 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.402958 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a28935a6-1847-4f43-9e7a-332c6b3ce49f-host-slash\") pod \"iptables-alerter-rxpmp\" (UID: \"a28935a6-1847-4f43-9e7a-332c6b3ce49f\") " pod="openshift-network-operator/iptables-alerter-rxpmp" Oct 30 17:57:45.403455 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.402985 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-etc-sysctl-d\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:57:45.403455 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.403027 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-run\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:57:45.403455 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.403053 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44e585a6-cc1a-4ebc-a592-907ccee3729f-tmp\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:57:45.403455 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.403120 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0af98be4-3c2f-413e-838e-f842854411f2-cnibin\") pod \"multus-additional-cni-plugins-78h8z\" (UID: \"0af98be4-3c2f-413e-838e-f842854411f2\") " pod="openshift-multus/multus-additional-cni-plugins-78h8z" Oct 30 17:57:45.403455 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.403146 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wchpl\" (UniqueName: \"kubernetes.io/projected/e0a64047-334d-4df8-83de-3a211200ea62-kube-api-access-wchpl\") pod \"node-ca-tsfch\" (UID: \"e0a64047-334d-4df8-83de-3a211200ea62\") " pod="openshift-image-registry/node-ca-tsfch" Oct 30 17:57:45.403455 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.403172 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-host-run-ovn-kubernetes\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.403455 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.403237 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a24e24cc-a321-4a62-b023-7b015f155d55-kubelet-config\") pod \"global-pull-secret-syncer-srgjm\" (UID: \"a24e24cc-a321-4a62-b023-7b015f155d55\") " pod="kube-system/global-pull-secret-syncer-srgjm" Oct 30 17:57:45.403455 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.403283 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d37ce412-3bd6-4093-9d21-3b72d396c1bd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8wmlh\" (UID: \"d37ce412-3bd6-4093-9d21-3b72d396c1bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" Oct 30 17:57:45.403761 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.403453 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-etc-systemd\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:57:45.403761 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.403562 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0af98be4-3c2f-413e-838e-f842854411f2-system-cni-dir\") pod \"multus-additional-cni-plugins-78h8z\" (UID: \"0af98be4-3c2f-413e-838e-f842854411f2\") " pod="openshift-multus/multus-additional-cni-plugins-78h8z" Oct 30 17:57:45.403761 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.403615 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvntm\" (UniqueName: \"kubernetes.io/projected/0af98be4-3c2f-413e-838e-f842854411f2-kube-api-access-zvntm\") pod \"multus-additional-cni-plugins-78h8z\" (UID: \"0af98be4-3c2f-413e-838e-f842854411f2\") " pod="openshift-multus/multus-additional-cni-plugins-78h8z" Oct 30 17:57:45.403761 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.403671 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-host-var-lib-cni-bin\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.403761 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.403724 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-env-overrides\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.403939 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.403766 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h9kb\" (UniqueName: \"kubernetes.io/projected/568da9c4-2c4f-4020-95ca-9f7ac7db7c00-kube-api-access-7h9kb\") pod \"network-metrics-daemon-9hrlx\" (UID: \"568da9c4-2c4f-4020-95ca-9f7ac7db7c00\") " pod="openshift-multus/network-metrics-daemon-9hrlx" Oct 30 17:57:45.403939 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.403804 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-etc-modprobe-d\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:57:45.403939 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.403859 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-etc-kubernetes\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:57:45.403939 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.403885 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-lib-modules\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:57:45.403939 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.403926 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-cni-binary-copy\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.404139 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.403944 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-host-kubelet\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.404139 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.403976 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-host-run-netns\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.404139 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.404034 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-host-run-multus-certs\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.404139 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.404100 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84lr2\" (UniqueName: \"kubernetes.io/projected/bf5ca486-37a8-4da0-acbf-d389caf9f3e6-kube-api-access-84lr2\") pod \"network-check-target-zpsg9\" (UID: \"bf5ca486-37a8-4da0-acbf-d389caf9f3e6\") " pod="openshift-network-diagnostics/network-check-target-zpsg9" Oct 30 17:57:45.404321 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.404146 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65887\" (UniqueName: \"kubernetes.io/projected/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-kube-api-access-65887\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.404321 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.404183 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-sys\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:57:45.404321 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.404263 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n82w\" (UniqueName: \"kubernetes.io/projected/44e585a6-cc1a-4ebc-a592-907ccee3729f-kube-api-access-9n82w\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:57:45.404321 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.404282 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-multus-cni-dir\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.404321 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.404314 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt5ws\" (UniqueName: \"kubernetes.io/projected/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-kube-api-access-wt5ws\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.404530 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.404352 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-ovnkube-script-lib\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.404530 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.404395 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d37ce412-3bd6-4093-9d21-3b72d396c1bd-sys-fs\") pod \"aws-ebs-csi-driver-node-8wmlh\" (UID: \"d37ce412-3bd6-4093-9d21-3b72d396c1bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" Oct 30 17:57:45.404530 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.404423 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-etc-sysctl-conf\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:57:45.404530 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.404445 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-multus-conf-dir\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.404530 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.404468 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-node-log\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.404530 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.404496 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-ovnkube-config\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.404530 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.404522 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-var-lib-kubelet\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:57:45.404822 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.404551 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-host\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:57:45.404822 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.404605 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-etc-kubernetes\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.404822 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.404625 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-run-openvswitch\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.404822 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.404640 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/59229234-921c-4177-8b85-3e982194f625-konnectivity-ca\") pod \"konnectivity-agent-c84zp\" (UID: \"59229234-921c-4177-8b85-3e982194f625\") " pod="kube-system/konnectivity-agent-c84zp" Oct 30 17:57:45.404822 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.404656 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e0a64047-334d-4df8-83de-3a211200ea62-serviceca\") pod \"node-ca-tsfch\" (UID: \"e0a64047-334d-4df8-83de-3a211200ea62\") " pod="openshift-image-registry/node-ca-tsfch" Oct 30 17:57:45.404822 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.404671 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-host-var-lib-cni-multus\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.404822 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.404686 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-host-slash\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.404822 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.404708 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-host-cni-netd\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.404822 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.404744 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh2xp\" (UniqueName: \"kubernetes.io/projected/d37ce412-3bd6-4093-9d21-3b72d396c1bd-kube-api-access-nh2xp\") pod \"aws-ebs-csi-driver-node-8wmlh\" (UID: \"d37ce412-3bd6-4093-9d21-3b72d396c1bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" Oct 30 17:57:45.404822 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.404784 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-cnibin\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.404822 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.404811 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-host-var-lib-kubelet\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.405340 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.404834 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d37ce412-3bd6-4093-9d21-3b72d396c1bd-etc-selinux\") pod \"aws-ebs-csi-driver-node-8wmlh\" (UID: \"d37ce412-3bd6-4093-9d21-3b72d396c1bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" Oct 30 17:57:45.405340 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.404877 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/568da9c4-2c4f-4020-95ca-9f7ac7db7c00-metrics-certs\") pod \"network-metrics-daemon-9hrlx\" (UID: \"568da9c4-2c4f-4020-95ca-9f7ac7db7c00\") " pod="openshift-multus/network-metrics-daemon-9hrlx" Oct 30 17:57:45.405340 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.404918 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0af98be4-3c2f-413e-838e-f842854411f2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-78h8z\" (UID: \"0af98be4-3c2f-413e-838e-f842854411f2\") " pod="openshift-multus/multus-additional-cni-plugins-78h8z" Oct 30 17:57:45.405340 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.404958 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e0a64047-334d-4df8-83de-3a211200ea62-host\") pod \"node-ca-tsfch\" (UID: \"e0a64047-334d-4df8-83de-3a211200ea62\") " pod="openshift-image-registry/node-ca-tsfch" Oct 30 17:57:45.405340 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.404988 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-host-run-k8s-cni-cncf-io\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.405340 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.405014 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-etc-openvswitch\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.405340 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.405070 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-log-socket\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.405340 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.405099 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/44e585a6-cc1a-4ebc-a592-907ccee3729f-etc-tuned\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:57:45.405340 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.405174 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-host-run-netns\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.405340 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.405284 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d37ce412-3bd6-4093-9d21-3b72d396c1bd-device-dir\") pod \"aws-ebs-csi-driver-node-8wmlh\" (UID: \"d37ce412-3bd6-4093-9d21-3b72d396c1bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" Oct 30 17:57:45.405340 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.405323 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-etc-sysconfig\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:57:45.405809 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.405367 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0af98be4-3c2f-413e-838e-f842854411f2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-78h8z\" (UID: \"0af98be4-3c2f-413e-838e-f842854411f2\") " pod="openshift-multus/multus-additional-cni-plugins-78h8z" Oct 30 17:57:45.405809 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.405450 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0772096e-be2c-449d-9bf6-c56cbd6c203a-hosts-file\") pod \"node-resolver-vp6gj\" (UID: \"0772096e-be2c-449d-9bf6-c56cbd6c203a\") " pod="openshift-dns/node-resolver-vp6gj" Oct 30 17:57:45.405809 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.405527 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0772096e-be2c-449d-9bf6-c56cbd6c203a-tmp-dir\") pod \"node-resolver-vp6gj\" (UID: \"0772096e-be2c-449d-9bf6-c56cbd6c203a\") " pod="openshift-dns/node-resolver-vp6gj" Oct 30 17:57:45.405809 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.405553 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-multus-socket-dir-parent\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.445354 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.445317 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 17:57:45.445532 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.445390 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wj5nd\"" Oct 30 17:57:45.445532 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.445392 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-p48xz\"" Oct 30 17:57:45.446001 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.445980 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 17:57:45.446165 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.446035 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 17:57:45.446258 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.446207 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pt2qr\"" Oct 30 17:57:45.446318 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.446260 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 17:57:45.446318 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.446270 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 17:57:45.446405 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.446375 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 17:57:45.446790 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.446662 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-jcq9r\"" Oct 30 17:57:45.446790 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.446690 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 17:57:45.446790 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.446705 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 17:57:45.446790 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.446695 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-c6627\"" Oct 30 17:57:45.446790 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.446723 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-nwsqb\"" Oct 30 17:57:45.446790 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.446722 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:57:45.446790 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.446752 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9m88r\"" Oct 30 17:57:45.446790 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.446768 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 17:57:45.459034 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.458966 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 17:57:45.459034 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.459000 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 17:57:45.459649 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.459626 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:57:45.459649 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.459640 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 17:57:45.459815 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.459695 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 17:57:45.459815 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.459745 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 17:57:45.459815 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.459767 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 17:57:45.459815 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.459812 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 17:57:45.459953 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.459830 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 17:57:45.459953 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.459847 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 17:57:45.459953 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.459867 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 17:57:45.459953 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.459880 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 17:57:45.459953 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.459931 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fh48h\"" Oct 30 17:57:45.506332 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.506297 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-etc-sysctl-d\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:57:45.506332 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.506343 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-run\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:57:45.506556 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.506374 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44e585a6-cc1a-4ebc-a592-907ccee3729f-tmp\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:57:45.506556 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.506398 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0af98be4-3c2f-413e-838e-f842854411f2-cnibin\") pod \"multus-additional-cni-plugins-78h8z\" (UID: \"0af98be4-3c2f-413e-838e-f842854411f2\") " pod="openshift-multus/multus-additional-cni-plugins-78h8z" Oct 30 17:57:45.506556 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.506417 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wchpl\" (UniqueName: \"kubernetes.io/projected/e0a64047-334d-4df8-83de-3a211200ea62-kube-api-access-wchpl\") pod \"node-ca-tsfch\" (UID: \"e0a64047-334d-4df8-83de-3a211200ea62\") " pod="openshift-image-registry/node-ca-tsfch" Oct 30 17:57:45.506556 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.506437 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-host-run-ovn-kubernetes\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.506556 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.506465 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a24e24cc-a321-4a62-b023-7b015f155d55-kubelet-config\") pod \"global-pull-secret-syncer-srgjm\" (UID: \"a24e24cc-a321-4a62-b023-7b015f155d55\") " pod="kube-system/global-pull-secret-syncer-srgjm" Oct 30 17:57:45.506556 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.506486 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d37ce412-3bd6-4093-9d21-3b72d396c1bd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8wmlh\" (UID: \"d37ce412-3bd6-4093-9d21-3b72d396c1bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" Oct 30 17:57:45.506556 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.506513 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-etc-systemd\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:57:45.506556 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.506541 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0af98be4-3c2f-413e-838e-f842854411f2-system-cni-dir\") pod \"multus-additional-cni-plugins-78h8z\" (UID: \"0af98be4-3c2f-413e-838e-f842854411f2\") " pod="openshift-multus/multus-additional-cni-plugins-78h8z" Oct 30 17:57:45.506847 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.506561 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvntm\" (UniqueName: \"kubernetes.io/projected/0af98be4-3c2f-413e-838e-f842854411f2-kube-api-access-zvntm\") pod \"multus-additional-cni-plugins-78h8z\" (UID: \"0af98be4-3c2f-413e-838e-f842854411f2\") " pod="openshift-multus/multus-additional-cni-plugins-78h8z" Oct 30 17:57:45.506847 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.506579 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-host-var-lib-cni-bin\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.506847 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.506600 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-env-overrides\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.506847 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.506629 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h9kb\" (UniqueName: \"kubernetes.io/projected/568da9c4-2c4f-4020-95ca-9f7ac7db7c00-kube-api-access-7h9kb\") pod \"network-metrics-daemon-9hrlx\" (UID: \"568da9c4-2c4f-4020-95ca-9f7ac7db7c00\") " pod="openshift-multus/network-metrics-daemon-9hrlx" Oct 30 17:57:45.506847 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.506662 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-etc-modprobe-d\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:57:45.506847 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.506689 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-etc-kubernetes\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:57:45.506847 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.506705 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-lib-modules\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:57:45.506847 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.506720 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-cni-binary-copy\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.506847 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.506737 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-host-kubelet\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.506847 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.506754 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-host-run-netns\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.506847 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.506770 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-host-run-multus-certs\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.506847 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.506790 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84lr2\" (UniqueName: \"kubernetes.io/projected/bf5ca486-37a8-4da0-acbf-d389caf9f3e6-kube-api-access-84lr2\") pod \"network-check-target-zpsg9\" (UID: \"bf5ca486-37a8-4da0-acbf-d389caf9f3e6\") " pod="openshift-network-diagnostics/network-check-target-zpsg9" Oct 30 17:57:45.506847 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.506821 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65887\" (UniqueName: \"kubernetes.io/projected/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-kube-api-access-65887\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.506847 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.506839 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-sys\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:57:45.507427 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.506861 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n82w\" (UniqueName: \"kubernetes.io/projected/44e585a6-cc1a-4ebc-a592-907ccee3729f-kube-api-access-9n82w\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:57:45.507427 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.506882 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-multus-cni-dir\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.507427 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.506911 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt5ws\" (UniqueName: \"kubernetes.io/projected/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-kube-api-access-wt5ws\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.507427 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.506930 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-ovnkube-script-lib\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.507427 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.506946 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d37ce412-3bd6-4093-9d21-3b72d396c1bd-sys-fs\") pod \"aws-ebs-csi-driver-node-8wmlh\" (UID: \"d37ce412-3bd6-4093-9d21-3b72d396c1bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" Oct 30 17:57:45.507427 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.506963 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-etc-sysctl-conf\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:57:45.507427 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.506991 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-multus-conf-dir\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.507427 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.507017 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-node-log\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.507427 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.507041 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-ovnkube-config\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.507427 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.507061 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-var-lib-kubelet\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:57:45.507427 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.507084 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-host\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:57:45.507427 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.507105 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-etc-kubernetes\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.507427 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.507124 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-run-openvswitch\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.507427 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.507156 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/59229234-921c-4177-8b85-3e982194f625-konnectivity-ca\") pod \"konnectivity-agent-c84zp\" (UID: \"59229234-921c-4177-8b85-3e982194f625\") " pod="kube-system/konnectivity-agent-c84zp" Oct 30 17:57:45.507427 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.507179 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e0a64047-334d-4df8-83de-3a211200ea62-serviceca\") pod \"node-ca-tsfch\" (UID: \"e0a64047-334d-4df8-83de-3a211200ea62\") " pod="openshift-image-registry/node-ca-tsfch" Oct 30 17:57:45.507427 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.507195 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-host-var-lib-cni-multus\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.507427 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.507240 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-host-slash\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.507427 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.507271 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-host-cni-netd\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.508231 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.507301 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh2xp\" (UniqueName: \"kubernetes.io/projected/d37ce412-3bd6-4093-9d21-3b72d396c1bd-kube-api-access-nh2xp\") pod \"aws-ebs-csi-driver-node-8wmlh\" (UID: \"d37ce412-3bd6-4093-9d21-3b72d396c1bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" Oct 30 17:57:45.508231 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.507333 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-cnibin\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.508231 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.507361 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-host-var-lib-kubelet\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.508231 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.507395 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d37ce412-3bd6-4093-9d21-3b72d396c1bd-etc-selinux\") pod \"aws-ebs-csi-driver-node-8wmlh\" (UID: \"d37ce412-3bd6-4093-9d21-3b72d396c1bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" Oct 30 17:57:45.508231 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.507426 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/568da9c4-2c4f-4020-95ca-9f7ac7db7c00-metrics-certs\") pod \"network-metrics-daemon-9hrlx\" (UID: \"568da9c4-2c4f-4020-95ca-9f7ac7db7c00\") " pod="openshift-multus/network-metrics-daemon-9hrlx" Oct 30 17:57:45.508231 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.507460 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0af98be4-3c2f-413e-838e-f842854411f2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-78h8z\" (UID: \"0af98be4-3c2f-413e-838e-f842854411f2\") " pod="openshift-multus/multus-additional-cni-plugins-78h8z" Oct 30 17:57:45.508231 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.507489 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e0a64047-334d-4df8-83de-3a211200ea62-host\") pod \"node-ca-tsfch\" (UID: \"e0a64047-334d-4df8-83de-3a211200ea62\") " pod="openshift-image-registry/node-ca-tsfch" Oct 30 17:57:45.508231 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.507517 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-host-run-k8s-cni-cncf-io\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.508231 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.507543 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-etc-openvswitch\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.508231 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.507568 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-log-socket\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.508231 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.507596 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/44e585a6-cc1a-4ebc-a592-907ccee3729f-etc-tuned\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:57:45.508231 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.507629 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-host-run-netns\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.508231 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.507657 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d37ce412-3bd6-4093-9d21-3b72d396c1bd-device-dir\") pod \"aws-ebs-csi-driver-node-8wmlh\" (UID: \"d37ce412-3bd6-4093-9d21-3b72d396c1bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" Oct 30 17:57:45.508231 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.507684 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-etc-sysconfig\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:57:45.508231 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.507712 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0af98be4-3c2f-413e-838e-f842854411f2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-78h8z\" (UID: \"0af98be4-3c2f-413e-838e-f842854411f2\") " pod="openshift-multus/multus-additional-cni-plugins-78h8z" Oct 30 17:57:45.508231 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.507745 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0772096e-be2c-449d-9bf6-c56cbd6c203a-hosts-file\") pod \"node-resolver-vp6gj\" (UID: \"0772096e-be2c-449d-9bf6-c56cbd6c203a\") " pod="openshift-dns/node-resolver-vp6gj" Oct 30 17:57:45.508231 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.507773 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0772096e-be2c-449d-9bf6-c56cbd6c203a-tmp-dir\") pod \"node-resolver-vp6gj\" (UID: \"0772096e-be2c-449d-9bf6-c56cbd6c203a\") " pod="openshift-dns/node-resolver-vp6gj" Oct 30 17:57:45.509108 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.507802 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-multus-socket-dir-parent\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.509108 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.507832 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-hostroot\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.509108 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.507860 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-run-systemd\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.509108 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.507889 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-host-cni-bin\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.509108 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.507915 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0af98be4-3c2f-413e-838e-f842854411f2-cni-binary-copy\") pod \"multus-additional-cni-plugins-78h8z\" (UID: \"0af98be4-3c2f-413e-838e-f842854411f2\") " pod="openshift-multus/multus-additional-cni-plugins-78h8z" Oct 30 17:57:45.509108 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.507945 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-run-ovn\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.509108 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.507974 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.509108 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.508005 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-ovn-node-metrics-cert\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.509108 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.508034 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a24e24cc-a321-4a62-b023-7b015f155d55-dbus\") pod \"global-pull-secret-syncer-srgjm\" (UID: \"a24e24cc-a321-4a62-b023-7b015f155d55\") " pod="kube-system/global-pull-secret-syncer-srgjm" Oct 30 17:57:45.509108 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.508070 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a24e24cc-a321-4a62-b023-7b015f155d55-original-pull-secret\") pod \"global-pull-secret-syncer-srgjm\" (UID: \"a24e24cc-a321-4a62-b023-7b015f155d55\") " pod="kube-system/global-pull-secret-syncer-srgjm" Oct 30 17:57:45.509108 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.508118 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/59229234-921c-4177-8b85-3e982194f625-agent-certs\") pod \"konnectivity-agent-c84zp\" (UID: \"59229234-921c-4177-8b85-3e982194f625\") " pod="kube-system/konnectivity-agent-c84zp" Oct 30 17:57:45.509108 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.508151 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-system-cni-dir\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.509108 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.508184 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-systemd-units\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.509108 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.508229 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a28935a6-1847-4f43-9e7a-332c6b3ce49f-iptables-alerter-script\") pod \"iptables-alerter-rxpmp\" (UID: \"a28935a6-1847-4f43-9e7a-332c6b3ce49f\") " pod="openshift-network-operator/iptables-alerter-rxpmp" Oct 30 17:57:45.509108 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.508239 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-host\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:57:45.509108 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.508263 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5sr2\" (UniqueName: \"kubernetes.io/projected/a28935a6-1847-4f43-9e7a-332c6b3ce49f-kube-api-access-m5sr2\") pod \"iptables-alerter-rxpmp\" (UID: \"a28935a6-1847-4f43-9e7a-332c6b3ce49f\") " pod="openshift-network-operator/iptables-alerter-rxpmp" Oct 30 17:57:45.509108 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.508311 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0af98be4-3c2f-413e-838e-f842854411f2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-78h8z\" (UID: \"0af98be4-3c2f-413e-838e-f842854411f2\") " pod="openshift-multus/multus-additional-cni-plugins-78h8z" Oct 30 17:57:45.509858 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.508354 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr4zd\" (UniqueName: \"kubernetes.io/projected/0772096e-be2c-449d-9bf6-c56cbd6c203a-kube-api-access-cr4zd\") pod \"node-resolver-vp6gj\" (UID: \"0772096e-be2c-449d-9bf6-c56cbd6c203a\") " pod="openshift-dns/node-resolver-vp6gj" Oct 30 17:57:45.509858 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.508389 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-multus-daemon-config\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.509858 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.508418 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-var-lib-openvswitch\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.509858 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.508451 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d37ce412-3bd6-4093-9d21-3b72d396c1bd-socket-dir\") pod \"aws-ebs-csi-driver-node-8wmlh\" (UID: \"d37ce412-3bd6-4093-9d21-3b72d396c1bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" Oct 30 17:57:45.509858 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.508481 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d37ce412-3bd6-4093-9d21-3b72d396c1bd-registration-dir\") pod \"aws-ebs-csi-driver-node-8wmlh\" (UID: \"d37ce412-3bd6-4093-9d21-3b72d396c1bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" Oct 30 17:57:45.509858 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.508509 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a28935a6-1847-4f43-9e7a-332c6b3ce49f-host-slash\") pod \"iptables-alerter-rxpmp\" (UID: \"a28935a6-1847-4f43-9e7a-332c6b3ce49f\") " pod="openshift-network-operator/iptables-alerter-rxpmp" Oct 30 17:57:45.509858 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.508620 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a28935a6-1847-4f43-9e7a-332c6b3ce49f-host-slash\") pod \"iptables-alerter-rxpmp\" (UID: \"a28935a6-1847-4f43-9e7a-332c6b3ce49f\") " pod="openshift-network-operator/iptables-alerter-rxpmp" Oct 30 17:57:45.509858 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.508651 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-etc-kubernetes\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.509858 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.508717 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-run-openvswitch\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.509858 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.508789 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-etc-sysctl-d\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:57:45.509858 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.508893 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-run\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:57:45.509858 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.509244 2565 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 17:57:45.509858 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.509454 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0af98be4-3c2f-413e-838e-f842854411f2-cnibin\") pod \"multus-additional-cni-plugins-78h8z\" (UID: \"0af98be4-3c2f-413e-838e-f842854411f2\") " pod="openshift-multus/multus-additional-cni-plugins-78h8z" Oct 30 17:57:45.509858 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.509673 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-host-run-ovn-kubernetes\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.509858 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.509729 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a24e24cc-a321-4a62-b023-7b015f155d55-kubelet-config\") pod \"global-pull-secret-syncer-srgjm\" (UID: \"a24e24cc-a321-4a62-b023-7b015f155d55\") " pod="kube-system/global-pull-secret-syncer-srgjm" Oct 30 17:57:45.509858 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.509766 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d37ce412-3bd6-4093-9d21-3b72d396c1bd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8wmlh\" (UID: \"d37ce412-3bd6-4093-9d21-3b72d396c1bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" Oct 30 17:57:45.509858 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.509799 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-etc-systemd\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:57:45.509858 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.509825 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0af98be4-3c2f-413e-838e-f842854411f2-system-cni-dir\") pod \"multus-additional-cni-plugins-78h8z\" (UID: \"0af98be4-3c2f-413e-838e-f842854411f2\") " pod="openshift-multus/multus-additional-cni-plugins-78h8z" Oct 30 17:57:45.510603 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.509943 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/59229234-921c-4177-8b85-3e982194f625-konnectivity-ca\") pod \"konnectivity-agent-c84zp\" (UID: \"59229234-921c-4177-8b85-3e982194f625\") " pod="kube-system/konnectivity-agent-c84zp" Oct 30 17:57:45.510603 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.510261 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-etc-sysconfig\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:57:45.510603 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.510306 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-run-ovn\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.510603 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.510304 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-host-run-netns\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.510603 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.510336 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-host-var-lib-cni-multus\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.510603 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.510387 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-host-slash\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.510603 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.510441 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-host-cni-netd\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.510603 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.510551 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0af98be4-3c2f-413e-838e-f842854411f2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-78h8z\" (UID: \"0af98be4-3c2f-413e-838e-f842854411f2\") " pod="openshift-multus/multus-additional-cni-plugins-78h8z" Oct 30 17:57:45.510603 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.510561 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-env-overrides\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.511044 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.510631 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-sys\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:57:45.511044 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.510682 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e0a64047-334d-4df8-83de-3a211200ea62-serviceca\") pod \"node-ca-tsfch\" (UID: \"e0a64047-334d-4df8-83de-3a211200ea62\") " pod="openshift-image-registry/node-ca-tsfch" Oct 30 17:57:45.511044 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.510691 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0772096e-be2c-449d-9bf6-c56cbd6c203a-hosts-file\") pod \"node-resolver-vp6gj\" (UID: \"0772096e-be2c-449d-9bf6-c56cbd6c203a\") " pod="openshift-dns/node-resolver-vp6gj" Oct 30 17:57:45.511044 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.510691 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-etc-modprobe-d\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:57:45.511044 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.510750 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d37ce412-3bd6-4093-9d21-3b72d396c1bd-sys-fs\") pod \"aws-ebs-csi-driver-node-8wmlh\" (UID: \"d37ce412-3bd6-4093-9d21-3b72d396c1bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" Oct 30 17:57:45.511044 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.510803 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-cnibin\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.511044 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.510810 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-lib-modules\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:57:45.511044 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.510861 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-host-var-lib-kubelet\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.511044 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.510865 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0772096e-be2c-449d-9bf6-c56cbd6c203a-tmp-dir\") pod \"node-resolver-vp6gj\" (UID: \"0772096e-be2c-449d-9bf6-c56cbd6c203a\") " pod="openshift-dns/node-resolver-vp6gj" Oct 30 17:57:45.511044 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.510881 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-etc-sysctl-conf\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:57:45.511044 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.510936 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-multus-conf-dir\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.511044 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.510943 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-multus-cni-dir\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.511044 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.510963 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d37ce412-3bd6-4093-9d21-3b72d396c1bd-etc-selinux\") pod \"aws-ebs-csi-driver-node-8wmlh\" (UID: \"d37ce412-3bd6-4093-9d21-3b72d396c1bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" Oct 30 17:57:45.511044 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.510995 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-node-log\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.511044 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.511011 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-var-lib-kubelet\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:57:45.511044 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.511042 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-host-kubelet\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.511820 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.511094 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-run-systemd\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.511820 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.511143 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-multus-socket-dir-parent\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.511820 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.511186 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-hostroot\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.511820 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.511515 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-etc-openvswitch\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.511820 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.511608 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e0a64047-334d-4df8-83de-3a211200ea62-host\") pod \"node-ca-tsfch\" (UID: \"e0a64047-334d-4df8-83de-3a211200ea62\") " pod="openshift-image-registry/node-ca-tsfch" Oct 30 17:57:45.511820 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:45.511679 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:57:45.511820 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.510552 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-host-run-multus-certs\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.511820 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:45.511746 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/568da9c4-2c4f-4020-95ca-9f7ac7db7c00-metrics-certs podName:568da9c4-2c4f-4020-95ca-9f7ac7db7c00 nodeName:}" failed. No retries permitted until 2026-10-30 17:57:46.011726082 +0000 UTC m=+3.073104539 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/568da9c4-2c4f-4020-95ca-9f7ac7db7c00-metrics-certs") pod "network-metrics-daemon-9hrlx" (UID: "568da9c4-2c4f-4020-95ca-9f7ac7db7c00") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:57:45.511820 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.511747 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-host-run-k8s-cni-cncf-io\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.511820 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.511751 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-log-socket\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.511820 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.511809 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-host-cni-bin\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.512392 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.511884 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-host-run-netns\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.512392 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.511944 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d37ce412-3bd6-4093-9d21-3b72d396c1bd-device-dir\") pod \"aws-ebs-csi-driver-node-8wmlh\" (UID: \"d37ce412-3bd6-4093-9d21-3b72d396c1bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" Oct 30 17:57:45.512392 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.511969 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-ovnkube-script-lib\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.512392 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.512049 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d37ce412-3bd6-4093-9d21-3b72d396c1bd-socket-dir\") pod \"aws-ebs-csi-driver-node-8wmlh\" (UID: \"d37ce412-3bd6-4093-9d21-3b72d396c1bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" Oct 30 17:57:45.512392 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.512094 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-var-lib-openvswitch\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.512392 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.512097 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-cni-binary-copy\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.512392 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.512144 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d37ce412-3bd6-4093-9d21-3b72d396c1bd-registration-dir\") pod \"aws-ebs-csi-driver-node-8wmlh\" (UID: \"d37ce412-3bd6-4093-9d21-3b72d396c1bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" Oct 30 17:57:45.512392 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.512252 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.512778 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.512694 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0af98be4-3c2f-413e-838e-f842854411f2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-78h8z\" (UID: \"0af98be4-3c2f-413e-838e-f842854411f2\") " pod="openshift-multus/multus-additional-cni-plugins-78h8z" Oct 30 17:57:45.512778 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.512720 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-multus-daemon-config\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.512778 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.512728 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a28935a6-1847-4f43-9e7a-332c6b3ce49f-iptables-alerter-script\") pod \"iptables-alerter-rxpmp\" (UID: \"a28935a6-1847-4f43-9e7a-332c6b3ce49f\") " pod="openshift-network-operator/iptables-alerter-rxpmp" Oct 30 17:57:45.512778 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.512745 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0af98be4-3c2f-413e-838e-f842854411f2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-78h8z\" (UID: \"0af98be4-3c2f-413e-838e-f842854411f2\") " pod="openshift-multus/multus-additional-cni-plugins-78h8z" Oct 30 17:57:45.512945 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.512791 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-system-cni-dir\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.512945 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.512849 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-systemd-units\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.512945 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.512915 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a24e24cc-a321-4a62-b023-7b015f155d55-dbus\") pod \"global-pull-secret-syncer-srgjm\" (UID: \"a24e24cc-a321-4a62-b023-7b015f155d55\") " pod="kube-system/global-pull-secret-syncer-srgjm" Oct 30 17:57:45.513055 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.513023 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-host-var-lib-cni-bin\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.513111 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:45.513056 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:57:45.513111 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:45.513109 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a24e24cc-a321-4a62-b023-7b015f155d55-original-pull-secret podName:a24e24cc-a321-4a62-b023-7b015f155d55 nodeName:}" failed. No retries permitted until 2026-10-30 17:57:46.013088191 +0000 UTC m=+3.074466623 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a24e24cc-a321-4a62-b023-7b015f155d55-original-pull-secret") pod "global-pull-secret-syncer-srgjm" (UID: "a24e24cc-a321-4a62-b023-7b015f155d55") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:57:45.513230 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.513121 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-etc-kubernetes\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:57:45.513230 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.513143 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-ovnkube-config\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.513230 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.513148 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0af98be4-3c2f-413e-838e-f842854411f2-cni-binary-copy\") pod \"multus-additional-cni-plugins-78h8z\" (UID: \"0af98be4-3c2f-413e-838e-f842854411f2\") " pod="openshift-multus/multus-additional-cni-plugins-78h8z" Oct 30 17:57:45.513541 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.513496 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44e585a6-cc1a-4ebc-a592-907ccee3729f-tmp\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:57:45.516850 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.516678 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-ovn-node-metrics-cert\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.516850 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.516771 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/59229234-921c-4177-8b85-3e982194f625-agent-certs\") pod \"konnectivity-agent-c84zp\" (UID: \"59229234-921c-4177-8b85-3e982194f625\") " pod="kube-system/konnectivity-agent-c84zp" Oct 30 17:57:45.517030 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.516955 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/44e585a6-cc1a-4ebc-a592-907ccee3729f-etc-tuned\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:57:45.528439 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.528388 2565 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:57:45.532263 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.532195 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5sr2\" (UniqueName: \"kubernetes.io/projected/a28935a6-1847-4f43-9e7a-332c6b3ce49f-kube-api-access-m5sr2\") pod \"iptables-alerter-rxpmp\" (UID: \"a28935a6-1847-4f43-9e7a-332c6b3ce49f\") " pod="openshift-network-operator/iptables-alerter-rxpmp" Oct 30 17:57:45.532370 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:45.532273 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:57:45.532370 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:45.532296 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:57:45.532370 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:45.532312 2565 projected.go:196] Error preparing data for projected volume kube-api-access-84lr2 for pod openshift-network-diagnostics/network-check-target-zpsg9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:57:45.532506 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:45.532379 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf5ca486-37a8-4da0-acbf-d389caf9f3e6-kube-api-access-84lr2 podName:bf5ca486-37a8-4da0-acbf-d389caf9f3e6 nodeName:}" failed. No retries permitted until 2026-10-30 17:57:46.032360143 +0000 UTC m=+3.093738600 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-84lr2" (UniqueName: "kubernetes.io/projected/bf5ca486-37a8-4da0-acbf-d389caf9f3e6-kube-api-access-84lr2") pod "network-check-target-zpsg9" (UID: "bf5ca486-37a8-4da0-acbf-d389caf9f3e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:57:45.534596 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.534569 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h9kb\" (UniqueName: \"kubernetes.io/projected/568da9c4-2c4f-4020-95ca-9f7ac7db7c00-kube-api-access-7h9kb\") pod \"network-metrics-daemon-9hrlx\" (UID: \"568da9c4-2c4f-4020-95ca-9f7ac7db7c00\") " pod="openshift-multus/network-metrics-daemon-9hrlx" Oct 30 17:57:45.535415 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.535366 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh2xp\" (UniqueName: \"kubernetes.io/projected/d37ce412-3bd6-4093-9d21-3b72d396c1bd-kube-api-access-nh2xp\") pod \"aws-ebs-csi-driver-node-8wmlh\" (UID: \"d37ce412-3bd6-4093-9d21-3b72d396c1bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" Oct 30 17:57:45.536627 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.536082 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wchpl\" (UniqueName: \"kubernetes.io/projected/e0a64047-334d-4df8-83de-3a211200ea62-kube-api-access-wchpl\") pod \"node-ca-tsfch\" (UID: \"e0a64047-334d-4df8-83de-3a211200ea62\") " pod="openshift-image-registry/node-ca-tsfch" Oct 30 17:57:45.536627 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.536576 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr4zd\" (UniqueName: \"kubernetes.io/projected/0772096e-be2c-449d-9bf6-c56cbd6c203a-kube-api-access-cr4zd\") pod \"node-resolver-vp6gj\" (UID: \"0772096e-be2c-449d-9bf6-c56cbd6c203a\") " pod="openshift-dns/node-resolver-vp6gj" Oct 30 17:57:45.537382 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.537343 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvntm\" (UniqueName: \"kubernetes.io/projected/0af98be4-3c2f-413e-838e-f842854411f2-kube-api-access-zvntm\") pod \"multus-additional-cni-plugins-78h8z\" (UID: \"0af98be4-3c2f-413e-838e-f842854411f2\") " pod="openshift-multus/multus-additional-cni-plugins-78h8z" Oct 30 17:57:45.538837 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.538813 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65887\" (UniqueName: \"kubernetes.io/projected/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-kube-api-access-65887\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.540179 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.540142 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n82w\" (UniqueName: \"kubernetes.io/projected/44e585a6-cc1a-4ebc-a592-907ccee3729f-kube-api-access-9n82w\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:57:45.541523 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.541474 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt5ws\" (UniqueName: \"kubernetes.io/projected/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-kube-api-access-wt5ws\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.633854 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.633815 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tsfch" Oct 30 17:57:45.644185 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:45.644129 2565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0a64047_334d_4df8_83de_3a211200ea62.slice/crio-c41265ebc3199e6664c64099e6b1826ef30d8ab567d887f01bc7fd8e37cfde4d WatchSource:0}: Error finding container c41265ebc3199e6664c64099e6b1826ef30d8ab567d887f01bc7fd8e37cfde4d: Status 404 returned error can't find the container with id c41265ebc3199e6664c64099e6b1826ef30d8ab567d887f01bc7fd8e37cfde4d Oct 30 17:57:45.649650 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.649623 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" Oct 30 17:57:45.656736 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.656711 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rxpmp" Oct 30 17:57:45.660971 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.660930 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-c84zp" Oct 30 17:57:45.667052 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:45.667011 2565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda28935a6_1847_4f43_9e7a_332c6b3ce49f.slice/crio-f894d624a353c3f8e2c9cdb15ecfa62e4288061b1f6ce1662e6245642069e557 WatchSource:0}: Error finding container f894d624a353c3f8e2c9cdb15ecfa62e4288061b1f6ce1662e6245642069e557: Status 404 returned error can't find the container with id f894d624a353c3f8e2c9cdb15ecfa62e4288061b1f6ce1662e6245642069e557 Oct 30 17:57:45.669739 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.669714 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:57:45.672556 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:45.672527 2565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59229234_921c_4177_8b85_3e982194f625.slice/crio-eeeb661dbbd8a1ec768602d833f2feab2fa496aa751dbe4ae51816fb5ffc8de4 WatchSource:0}: Error finding container eeeb661dbbd8a1ec768602d833f2feab2fa496aa751dbe4ae51816fb5ffc8de4: Status 404 returned error can't find the container with id eeeb661dbbd8a1ec768602d833f2feab2fa496aa751dbe4ae51816fb5ffc8de4 Oct 30 17:57:45.678347 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.678327 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wvg8l" Oct 30 17:57:45.684108 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:45.684077 2565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44e585a6_cc1a_4ebc_a592_907ccee3729f.slice/crio-317a5c369cacbf121db08726e9b826a994b0bfea845e53b886c218a20b000ef6 WatchSource:0}: Error finding container 317a5c369cacbf121db08726e9b826a994b0bfea845e53b886c218a20b000ef6: Status 404 returned error can't find the container with id 317a5c369cacbf121db08726e9b826a994b0bfea845e53b886c218a20b000ef6 Oct 30 17:57:45.691167 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:45.691115 2565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ba0b8dc_0459_440d_91dc_ad18f4b279fa.slice/crio-3bc6aeac772dd708ce9046b0304b23ae422455f1543d097553d14f687d51d86f WatchSource:0}: Error finding container 3bc6aeac772dd708ce9046b0304b23ae422455f1543d097553d14f687d51d86f: Status 404 returned error can't find the container with id 3bc6aeac772dd708ce9046b0304b23ae422455f1543d097553d14f687d51d86f Oct 30 17:57:45.700720 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.700523 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vp6gj" Oct 30 17:57:45.707294 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.707271 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-78h8z" Oct 30 17:57:45.712455 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:45.712416 2565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0772096e_be2c_449d_9bf6_c56cbd6c203a.slice/crio-91820cfa6dc81f9c6c366883a9c69fbc51bd5b4d7cf837fb6a1d97a111addfdf WatchSource:0}: Error finding container 91820cfa6dc81f9c6c366883a9c69fbc51bd5b4d7cf837fb6a1d97a111addfdf: Status 404 returned error can't find the container with id 91820cfa6dc81f9c6c366883a9c69fbc51bd5b4d7cf837fb6a1d97a111addfdf Oct 30 17:57:45.720776 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:45.720740 2565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0af98be4_3c2f_413e_838e_f842854411f2.slice/crio-5b19ec34f9443e71dd23a1c4adebb7a8fdcfc9a9c2790cb54ca232ad88ccfebd WatchSource:0}: Error finding container 5b19ec34f9443e71dd23a1c4adebb7a8fdcfc9a9c2790cb54ca232ad88ccfebd: Status 404 returned error can't find the container with id 5b19ec34f9443e71dd23a1c4adebb7a8fdcfc9a9c2790cb54ca232ad88ccfebd Oct 30 17:57:45.724214 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:45.724189 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:57:45.736815 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:57:45.736783 2565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod516c6b6c_fc40_49cf_93b5_1c4471b02cf0.slice/crio-cdd236e6bad8950bc60a0cf0d2be119eff0267273b1421a7a22fed2b7c70001e WatchSource:0}: Error finding container cdd236e6bad8950bc60a0cf0d2be119eff0267273b1421a7a22fed2b7c70001e: Status 404 returned error can't find the container with id cdd236e6bad8950bc60a0cf0d2be119eff0267273b1421a7a22fed2b7c70001e Oct 30 17:57:46.017791 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:46.017750 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/568da9c4-2c4f-4020-95ca-9f7ac7db7c00-metrics-certs\") pod \"network-metrics-daemon-9hrlx\" (UID: \"568da9c4-2c4f-4020-95ca-9f7ac7db7c00\") " pod="openshift-multus/network-metrics-daemon-9hrlx" Oct 30 17:57:46.017957 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:46.017825 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a24e24cc-a321-4a62-b023-7b015f155d55-original-pull-secret\") pod \"global-pull-secret-syncer-srgjm\" (UID: \"a24e24cc-a321-4a62-b023-7b015f155d55\") " pod="kube-system/global-pull-secret-syncer-srgjm" Oct 30 17:57:46.018021 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:46.017997 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:57:46.018080 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:46.018067 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a24e24cc-a321-4a62-b023-7b015f155d55-original-pull-secret podName:a24e24cc-a321-4a62-b023-7b015f155d55 nodeName:}" failed. No retries permitted until 2026-10-30 17:57:47.018049206 +0000 UTC m=+4.079427658 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a24e24cc-a321-4a62-b023-7b015f155d55-original-pull-secret") pod "global-pull-secret-syncer-srgjm" (UID: "a24e24cc-a321-4a62-b023-7b015f155d55") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:57:46.018809 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:46.018153 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:57:46.018809 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:46.018202 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/568da9c4-2c4f-4020-95ca-9f7ac7db7c00-metrics-certs podName:568da9c4-2c4f-4020-95ca-9f7ac7db7c00 nodeName:}" failed. No retries permitted until 2026-10-30 17:57:47.018187544 +0000 UTC m=+4.079565978 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/568da9c4-2c4f-4020-95ca-9f7ac7db7c00-metrics-certs") pod "network-metrics-daemon-9hrlx" (UID: "568da9c4-2c4f-4020-95ca-9f7ac7db7c00") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:57:46.119282 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:46.119217 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84lr2\" (UniqueName: \"kubernetes.io/projected/bf5ca486-37a8-4da0-acbf-d389caf9f3e6-kube-api-access-84lr2\") pod \"network-check-target-zpsg9\" (UID: \"bf5ca486-37a8-4da0-acbf-d389caf9f3e6\") " pod="openshift-network-diagnostics/network-check-target-zpsg9" Oct 30 17:57:46.119583 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:46.119452 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:57:46.119583 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:46.119473 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:57:46.119583 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:46.119487 2565 projected.go:196] Error preparing data for projected volume kube-api-access-84lr2 for pod openshift-network-diagnostics/network-check-target-zpsg9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:57:46.119583 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:46.119552 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf5ca486-37a8-4da0-acbf-d389caf9f3e6-kube-api-access-84lr2 podName:bf5ca486-37a8-4da0-acbf-d389caf9f3e6 nodeName:}" failed. No retries permitted until 2026-10-30 17:57:47.119532596 +0000 UTC m=+4.180911048 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-84lr2" (UniqueName: "kubernetes.io/projected/bf5ca486-37a8-4da0-acbf-d389caf9f3e6-kube-api-access-84lr2") pod "network-check-target-zpsg9" (UID: "bf5ca486-37a8-4da0-acbf-d389caf9f3e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:57:46.353810 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:46.353725 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 17:52:45 +0000 UTC" deadline="2028-06-08 17:20:15.56374279 +0000 UTC" Oct 30 17:57:46.353810 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:46.353809 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14087h22m29.209938005s" Oct 30 17:57:46.441662 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:46.441618 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zpsg9" Oct 30 17:57:46.441837 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:46.441777 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zpsg9" podUID="bf5ca486-37a8-4da0-acbf-d389caf9f3e6" Oct 30 17:57:46.454008 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:46.453969 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" event={"ID":"516c6b6c-fc40-49cf-93b5-1c4471b02cf0","Type":"ContainerStarted","Data":"cdd236e6bad8950bc60a0cf0d2be119eff0267273b1421a7a22fed2b7c70001e"} Oct 30 17:57:46.456197 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:46.456146 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vp6gj" event={"ID":"0772096e-be2c-449d-9bf6-c56cbd6c203a","Type":"ContainerStarted","Data":"91820cfa6dc81f9c6c366883a9c69fbc51bd5b4d7cf837fb6a1d97a111addfdf"} Oct 30 17:57:46.459400 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:46.459344 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wvg8l" event={"ID":"8ba0b8dc-0459-440d-91dc-ad18f4b279fa","Type":"ContainerStarted","Data":"3bc6aeac772dd708ce9046b0304b23ae422455f1543d097553d14f687d51d86f"} Oct 30 17:57:46.462397 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:46.462371 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" event={"ID":"44e585a6-cc1a-4ebc-a592-907ccee3729f","Type":"ContainerStarted","Data":"317a5c369cacbf121db08726e9b826a994b0bfea845e53b886c218a20b000ef6"} Oct 30 17:57:46.470536 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:46.470506 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" event={"ID":"d37ce412-3bd6-4093-9d21-3b72d396c1bd","Type":"ContainerStarted","Data":"90c5a4d78f5e69d3d3d7d2e7a3303f8ed64b8a08cdcc180430b36f637af24dee"} Oct 30 17:57:46.473006 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:46.472978 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tsfch" event={"ID":"e0a64047-334d-4df8-83de-3a211200ea62","Type":"ContainerStarted","Data":"c41265ebc3199e6664c64099e6b1826ef30d8ab567d887f01bc7fd8e37cfde4d"} Oct 30 17:57:46.477352 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:46.477235 2565 generic.go:356] "Generic (PLEG): container finished" podID="bfd89a876f67f9d878ac0d1431582688" containerID="fa6b0527e674aaca86c5e6f2aa27874e46e1076527a3cb31dd7a4f70c1b4d580" exitCode=0 Oct 30 17:57:46.477352 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:46.477308 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal" event={"ID":"bfd89a876f67f9d878ac0d1431582688","Type":"ContainerDied","Data":"fa6b0527e674aaca86c5e6f2aa27874e46e1076527a3cb31dd7a4f70c1b4d580"} Oct 30 17:57:46.483479 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:46.482964 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-176.ec2.internal" event={"ID":"9e81bd78300430e51b01d30caa1123e7","Type":"ContainerStarted","Data":"7b7254c145c86f7687ed674a24901d52ef6550fc1901906d81acdb64bfe0c264"} Oct 30 17:57:46.493223 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:46.493178 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-78h8z" event={"ID":"0af98be4-3c2f-413e-838e-f842854411f2","Type":"ContainerStarted","Data":"5b19ec34f9443e71dd23a1c4adebb7a8fdcfc9a9c2790cb54ca232ad88ccfebd"} Oct 30 17:57:46.503622 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:46.503593 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-c84zp" event={"ID":"59229234-921c-4177-8b85-3e982194f625","Type":"ContainerStarted","Data":"eeeb661dbbd8a1ec768602d833f2feab2fa496aa751dbe4ae51816fb5ffc8de4"} Oct 30 17:57:46.521871 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:46.521754 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rxpmp" event={"ID":"a28935a6-1847-4f43-9e7a-332c6b3ce49f","Type":"ContainerStarted","Data":"f894d624a353c3f8e2c9cdb15ecfa62e4288061b1f6ce1662e6245642069e557"} Oct 30 17:57:46.561888 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:46.561818 2565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-176.ec2.internal" podStartSLOduration=2.561796152 podStartE2EDuration="2.561796152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:57:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:46.561596347 +0000 UTC m=+3.622974806" watchObservedRunningTime="2026-10-30 17:57:46.561796152 +0000 UTC m=+3.623174612" Oct 30 17:57:47.026074 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:47.026036 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/568da9c4-2c4f-4020-95ca-9f7ac7db7c00-metrics-certs\") pod \"network-metrics-daemon-9hrlx\" (UID: \"568da9c4-2c4f-4020-95ca-9f7ac7db7c00\") " pod="openshift-multus/network-metrics-daemon-9hrlx" Oct 30 17:57:47.026264 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:47.026098 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a24e24cc-a321-4a62-b023-7b015f155d55-original-pull-secret\") pod \"global-pull-secret-syncer-srgjm\" (UID: \"a24e24cc-a321-4a62-b023-7b015f155d55\") " pod="kube-system/global-pull-secret-syncer-srgjm" Oct 30 17:57:47.026344 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:47.026280 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:57:47.026395 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:47.026348 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a24e24cc-a321-4a62-b023-7b015f155d55-original-pull-secret podName:a24e24cc-a321-4a62-b023-7b015f155d55 nodeName:}" failed. No retries permitted until 2026-10-30 17:57:49.026328294 +0000 UTC m=+6.087706741 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a24e24cc-a321-4a62-b023-7b015f155d55-original-pull-secret") pod "global-pull-secret-syncer-srgjm" (UID: "a24e24cc-a321-4a62-b023-7b015f155d55") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:57:47.026453 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:47.026416 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:57:47.026453 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:47.026449 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/568da9c4-2c4f-4020-95ca-9f7ac7db7c00-metrics-certs podName:568da9c4-2c4f-4020-95ca-9f7ac7db7c00 nodeName:}" failed. No retries permitted until 2026-10-30 17:57:49.026436887 +0000 UTC m=+6.087815335 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/568da9c4-2c4f-4020-95ca-9f7ac7db7c00-metrics-certs") pod "network-metrics-daemon-9hrlx" (UID: "568da9c4-2c4f-4020-95ca-9f7ac7db7c00") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:57:47.127301 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:47.127261 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84lr2\" (UniqueName: \"kubernetes.io/projected/bf5ca486-37a8-4da0-acbf-d389caf9f3e6-kube-api-access-84lr2\") pod \"network-check-target-zpsg9\" (UID: \"bf5ca486-37a8-4da0-acbf-d389caf9f3e6\") " pod="openshift-network-diagnostics/network-check-target-zpsg9" Oct 30 17:57:47.127469 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:47.127419 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:57:47.127469 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:47.127438 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:57:47.127469 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:47.127451 2565 projected.go:196] Error preparing data for projected volume kube-api-access-84lr2 for pod openshift-network-diagnostics/network-check-target-zpsg9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:57:47.127636 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:47.127508 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf5ca486-37a8-4da0-acbf-d389caf9f3e6-kube-api-access-84lr2 podName:bf5ca486-37a8-4da0-acbf-d389caf9f3e6 nodeName:}" failed. No retries permitted until 2026-10-30 17:57:49.127491168 +0000 UTC m=+6.188869605 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-84lr2" (UniqueName: "kubernetes.io/projected/bf5ca486-37a8-4da0-acbf-d389caf9f3e6-kube-api-access-84lr2") pod "network-check-target-zpsg9" (UID: "bf5ca486-37a8-4da0-acbf-d389caf9f3e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:57:47.354130 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:47.353989 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 17:52:45 +0000 UTC" deadline="2028-06-13 18:11:43.387807437 +0000 UTC" Oct 30 17:57:47.354130 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:47.354031 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14208h13m56.033781545s" Oct 30 17:57:47.445345 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:47.445282 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-srgjm" Oct 30 17:57:47.445508 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:47.445416 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-srgjm" podUID="a24e24cc-a321-4a62-b023-7b015f155d55" Oct 30 17:57:47.445819 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:47.445795 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9hrlx" Oct 30 17:57:47.445920 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:47.445902 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9hrlx" podUID="568da9c4-2c4f-4020-95ca-9f7ac7db7c00" Oct 30 17:57:47.539018 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:47.538974 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal" event={"ID":"bfd89a876f67f9d878ac0d1431582688","Type":"ContainerStarted","Data":"40da92af59281f382128981d7f620727c98b6e5e7fe95406343d7dba1de6a3cf"} Oct 30 17:57:48.441163 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:48.440918 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zpsg9" Oct 30 17:57:48.441163 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:48.441077 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zpsg9" podUID="bf5ca486-37a8-4da0-acbf-d389caf9f3e6" Oct 30 17:57:49.045994 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:49.045914 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a24e24cc-a321-4a62-b023-7b015f155d55-original-pull-secret\") pod \"global-pull-secret-syncer-srgjm\" (UID: \"a24e24cc-a321-4a62-b023-7b015f155d55\") " pod="kube-system/global-pull-secret-syncer-srgjm" Oct 30 17:57:49.046260 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:49.046014 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/568da9c4-2c4f-4020-95ca-9f7ac7db7c00-metrics-certs\") pod \"network-metrics-daemon-9hrlx\" (UID: \"568da9c4-2c4f-4020-95ca-9f7ac7db7c00\") " pod="openshift-multus/network-metrics-daemon-9hrlx" Oct 30 17:57:49.046260 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:49.046151 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:57:49.046260 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:49.046215 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/568da9c4-2c4f-4020-95ca-9f7ac7db7c00-metrics-certs podName:568da9c4-2c4f-4020-95ca-9f7ac7db7c00 nodeName:}" failed. No retries permitted until 2026-10-30 17:57:53.046195184 +0000 UTC m=+10.107573620 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/568da9c4-2c4f-4020-95ca-9f7ac7db7c00-metrics-certs") pod "network-metrics-daemon-9hrlx" (UID: "568da9c4-2c4f-4020-95ca-9f7ac7db7c00") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:57:49.046569 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:49.046326 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:57:49.046569 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:49.046398 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a24e24cc-a321-4a62-b023-7b015f155d55-original-pull-secret podName:a24e24cc-a321-4a62-b023-7b015f155d55 nodeName:}" failed. No retries permitted until 2026-10-30 17:57:53.046380989 +0000 UTC m=+10.107759440 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a24e24cc-a321-4a62-b023-7b015f155d55-original-pull-secret") pod "global-pull-secret-syncer-srgjm" (UID: "a24e24cc-a321-4a62-b023-7b015f155d55") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:57:49.146566 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:49.146524 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84lr2\" (UniqueName: \"kubernetes.io/projected/bf5ca486-37a8-4da0-acbf-d389caf9f3e6-kube-api-access-84lr2\") pod \"network-check-target-zpsg9\" (UID: \"bf5ca486-37a8-4da0-acbf-d389caf9f3e6\") " pod="openshift-network-diagnostics/network-check-target-zpsg9" Oct 30 17:57:49.146739 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:49.146721 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:57:49.146739 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:49.146740 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:57:49.146834 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:49.146752 2565 projected.go:196] Error preparing data for projected volume kube-api-access-84lr2 for pod openshift-network-diagnostics/network-check-target-zpsg9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:57:49.146834 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:49.146807 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf5ca486-37a8-4da0-acbf-d389caf9f3e6-kube-api-access-84lr2 podName:bf5ca486-37a8-4da0-acbf-d389caf9f3e6 nodeName:}" failed. No retries permitted until 2026-10-30 17:57:53.146788956 +0000 UTC m=+10.208167405 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-84lr2" (UniqueName: "kubernetes.io/projected/bf5ca486-37a8-4da0-acbf-d389caf9f3e6-kube-api-access-84lr2") pod "network-check-target-zpsg9" (UID: "bf5ca486-37a8-4da0-acbf-d389caf9f3e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:57:49.444597 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:49.443846 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-srgjm" Oct 30 17:57:49.444597 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:49.443968 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-srgjm" podUID="a24e24cc-a321-4a62-b023-7b015f155d55" Oct 30 17:57:49.444597 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:49.444384 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9hrlx" Oct 30 17:57:49.444597 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:49.444502 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9hrlx" podUID="568da9c4-2c4f-4020-95ca-9f7ac7db7c00" Oct 30 17:57:49.567384 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:49.566336 2565 generic.go:356] "Generic (PLEG): container finished" podID="0af98be4-3c2f-413e-838e-f842854411f2" containerID="224a70994571cf40b7ac77cd6dc844d93099de98517fb65cad6f113641d9a662" exitCode=0 Oct 30 17:57:49.567384 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:49.566393 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-78h8z" event={"ID":"0af98be4-3c2f-413e-838e-f842854411f2","Type":"ContainerDied","Data":"224a70994571cf40b7ac77cd6dc844d93099de98517fb65cad6f113641d9a662"} Oct 30 17:57:49.636445 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:49.636346 2565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal" podStartSLOduration=5.636326606 podStartE2EDuration="5.636326606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:57:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:47.5812766 +0000 UTC m=+4.642655057" watchObservedRunningTime="2026-10-30 17:57:49.636326606 +0000 UTC m=+6.697705060" Oct 30 17:57:50.441050 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:50.441009 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zpsg9" Oct 30 17:57:50.441223 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:50.441174 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zpsg9" podUID="bf5ca486-37a8-4da0-acbf-d389caf9f3e6" Oct 30 17:57:50.577082 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:50.576371 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tsfch" event={"ID":"e0a64047-334d-4df8-83de-3a211200ea62","Type":"ContainerStarted","Data":"ff8ee2e57c626e8fc19ae46fb2bc2a8fea315e05b6ac8c719580e4792effdd79"} Oct 30 17:57:50.581601 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:50.581090 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-c84zp" event={"ID":"59229234-921c-4177-8b85-3e982194f625","Type":"ContainerStarted","Data":"05aea6fd1ad87d99eb6ece3ff227f7e28f4940beb34846cb49f1771b2490fd21"} Oct 30 17:57:50.655731 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:50.655191 2565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tsfch" podStartSLOduration=2.921774311 podStartE2EDuration="7.655168034s" totalImagesPullingTime="4.733393723s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:57:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:50.616601574 +0000 UTC m=+7.677980031" watchObservedRunningTime="2026-10-30 17:57:50.655168034 +0000 UTC m=+7.716546487" Oct 30 17:57:51.441765 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:51.441402 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-srgjm" Oct 30 17:57:51.441765 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:51.441557 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-srgjm" podUID="a24e24cc-a321-4a62-b023-7b015f155d55" Oct 30 17:57:51.441765 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:51.441633 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9hrlx" Oct 30 17:57:51.441765 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:51.441721 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9hrlx" podUID="568da9c4-2c4f-4020-95ca-9f7ac7db7c00" Oct 30 17:57:51.600113 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:51.600029 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" event={"ID":"d37ce412-3bd6-4093-9d21-3b72d396c1bd","Type":"ContainerStarted","Data":"39817ba188cbb4ed30b7386e36452a8dade6a0a2268b8a7560e37cfd340802c8"} Oct 30 17:57:52.441456 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:52.440769 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zpsg9" Oct 30 17:57:52.441456 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:52.440916 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zpsg9" podUID="bf5ca486-37a8-4da0-acbf-d389caf9f3e6" Oct 30 17:57:52.761475 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:52.760751 2565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-c84zp" Oct 30 17:57:52.761926 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:52.761663 2565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-c84zp" Oct 30 17:57:52.800407 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:52.799703 2565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-c84zp" podStartSLOduration=5.688994485 podStartE2EDuration="9.799683987s" totalImagesPullingTime="4.110689502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:57:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:50.655666812 +0000 UTC m=+7.717045265" watchObservedRunningTime="2026-10-30 17:57:52.799683987 +0000 UTC m=+9.861062441" Oct 30 17:57:53.072192 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:53.071651 2565 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 17:57:53.079207 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:53.079173 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/568da9c4-2c4f-4020-95ca-9f7ac7db7c00-metrics-certs\") pod \"network-metrics-daemon-9hrlx\" (UID: \"568da9c4-2c4f-4020-95ca-9f7ac7db7c00\") " pod="openshift-multus/network-metrics-daemon-9hrlx" Oct 30 17:57:53.079338 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:53.079237 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a24e24cc-a321-4a62-b023-7b015f155d55-original-pull-secret\") pod \"global-pull-secret-syncer-srgjm\" (UID: \"a24e24cc-a321-4a62-b023-7b015f155d55\") " pod="kube-system/global-pull-secret-syncer-srgjm" Oct 30 17:57:53.079400 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:53.079372 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:57:53.079449 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:53.079441 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a24e24cc-a321-4a62-b023-7b015f155d55-original-pull-secret podName:a24e24cc-a321-4a62-b023-7b015f155d55 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:01.079422637 +0000 UTC m=+18.140801073 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a24e24cc-a321-4a62-b023-7b015f155d55-original-pull-secret") pod "global-pull-secret-syncer-srgjm" (UID: "a24e24cc-a321-4a62-b023-7b015f155d55") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:57:53.079530 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:53.079518 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:57:53.079569 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:53.079557 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/568da9c4-2c4f-4020-95ca-9f7ac7db7c00-metrics-certs podName:568da9c4-2c4f-4020-95ca-9f7ac7db7c00 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:01.079544818 +0000 UTC m=+18.140923250 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/568da9c4-2c4f-4020-95ca-9f7ac7db7c00-metrics-certs") pod "network-metrics-daemon-9hrlx" (UID: "568da9c4-2c4f-4020-95ca-9f7ac7db7c00") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:57:53.180369 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:53.180328 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84lr2\" (UniqueName: \"kubernetes.io/projected/bf5ca486-37a8-4da0-acbf-d389caf9f3e6-kube-api-access-84lr2\") pod \"network-check-target-zpsg9\" (UID: \"bf5ca486-37a8-4da0-acbf-d389caf9f3e6\") " pod="openshift-network-diagnostics/network-check-target-zpsg9" Oct 30 17:57:53.180602 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:53.180580 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:57:53.180675 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:53.180609 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:57:53.180675 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:53.180624 2565 projected.go:196] Error preparing data for projected volume kube-api-access-84lr2 for pod openshift-network-diagnostics/network-check-target-zpsg9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:57:53.180769 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:53.180684 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf5ca486-37a8-4da0-acbf-d389caf9f3e6-kube-api-access-84lr2 podName:bf5ca486-37a8-4da0-acbf-d389caf9f3e6 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:01.180668132 +0000 UTC m=+18.242046563 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-84lr2" (UniqueName: "kubernetes.io/projected/bf5ca486-37a8-4da0-acbf-d389caf9f3e6-kube-api-access-84lr2") pod "network-check-target-zpsg9" (UID: "bf5ca486-37a8-4da0-acbf-d389caf9f3e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:57:53.359755 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:53.359587 2565 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T17:57:53.071672272Z","UUID":"bf8eb9db-3e2c-4b5f-8511-ddc4aa2845c2","Handler":null,"Name":"","Endpoint":""} Oct 30 17:57:53.361117 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:53.361093 2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 17:57:53.361285 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:53.361126 2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 17:57:53.444153 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:53.444110 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-srgjm" Oct 30 17:57:53.444294 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:53.444254 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-srgjm" podUID="a24e24cc-a321-4a62-b023-7b015f155d55" Oct 30 17:57:53.444644 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:53.444621 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9hrlx" Oct 30 17:57:53.444741 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:53.444724 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9hrlx" podUID="568da9c4-2c4f-4020-95ca-9f7ac7db7c00" Oct 30 17:57:53.608083 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:53.608052 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wvg8l" event={"ID":"8ba0b8dc-0459-440d-91dc-ad18f4b279fa","Type":"ContainerStarted","Data":"6347c8f2e3e22d928f788a175862deed288b63fcc7e1bb98066547b4c23ee0b8"} Oct 30 17:57:53.611591 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:53.611531 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" event={"ID":"d37ce412-3bd6-4093-9d21-3b72d396c1bd","Type":"ContainerStarted","Data":"66593cd40f5f2ac6c1438e33150ff8681b321c02b993213ae10c1a9cccdf75f7"} Oct 30 17:57:53.611760 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:53.611741 2565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-c84zp" Oct 30 17:57:53.612414 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:53.612369 2565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-c84zp" Oct 30 17:57:53.709618 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:53.709357 2565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-wvg8l" podStartSLOduration=3.235542116 podStartE2EDuration="10.709337675s" totalImagesPullingTime="7.473795559s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:57:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:53.670962894 +0000 UTC m=+10.732341348" watchObservedRunningTime="2026-10-30 17:57:53.709337675 +0000 UTC m=+10.770716134" Oct 30 17:57:54.440987 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:54.440912 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zpsg9" Oct 30 17:57:54.441521 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:54.441039 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zpsg9" podUID="bf5ca486-37a8-4da0-acbf-d389caf9f3e6" Oct 30 17:57:54.616007 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:54.615962 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vp6gj" event={"ID":"0772096e-be2c-449d-9bf6-c56cbd6c203a","Type":"ContainerStarted","Data":"c4837ed1ac24f1c989f2d20e86f95ba4538ad7dd60ed339061f11f1145b8db87"} Oct 30 17:57:54.618939 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:54.618904 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" event={"ID":"d37ce412-3bd6-4093-9d21-3b72d396c1bd","Type":"ContainerStarted","Data":"4ee38e492937232123d5c834fa113c7f72c3f832ffee2a70dfd0e4252360e886"} Oct 30 17:57:54.629707 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:54.629675 2565 generic.go:356] "Generic (PLEG): container finished" podID="0af98be4-3c2f-413e-838e-f842854411f2" containerID="57903c2e169bad6ae7ed053a689ab29d81c12c4faa1aeed18f156f9adb8b8d13" exitCode=0 Oct 30 17:57:54.629880 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:54.629744 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-78h8z" event={"ID":"0af98be4-3c2f-413e-838e-f842854411f2","Type":"ContainerDied","Data":"57903c2e169bad6ae7ed053a689ab29d81c12c4faa1aeed18f156f9adb8b8d13"} Oct 30 17:57:54.655576 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:54.655524 2565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-vp6gj" podStartSLOduration=3.795419866 podStartE2EDuration="11.655506593s" totalImagesPullingTime="7.860086727s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:57:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:54.655126145 +0000 UTC m=+11.716504599" watchObservedRunningTime="2026-10-30 17:57:54.655506593 +0000 UTC m=+11.716885046" Oct 30 17:57:54.698598 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:54.698502 2565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" podStartSLOduration=3.703506632 podStartE2EDuration="11.698489586s" totalImagesPullingTime="7.994982954s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:57:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:54.697888044 +0000 UTC m=+11.759266497" watchObservedRunningTime="2026-10-30 17:57:54.698489586 +0000 UTC m=+11.759868080" Oct 30 17:57:55.441322 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:55.441286 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9hrlx" Oct 30 17:57:55.441847 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:55.441348 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-srgjm" Oct 30 17:57:55.441847 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:55.441499 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9hrlx" podUID="568da9c4-2c4f-4020-95ca-9f7ac7db7c00" Oct 30 17:57:55.441847 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:55.441613 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-srgjm" podUID="a24e24cc-a321-4a62-b023-7b015f155d55" Oct 30 17:57:55.634456 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:55.634422 2565 generic.go:356] "Generic (PLEG): container finished" podID="0af98be4-3c2f-413e-838e-f842854411f2" containerID="f2ecd408dbe1af1877160ffe7b9865b538c56c7bdd2a760aaca8ce8e38036d59" exitCode=0 Oct 30 17:57:55.634602 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:55.634509 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-78h8z" event={"ID":"0af98be4-3c2f-413e-838e-f842854411f2","Type":"ContainerDied","Data":"f2ecd408dbe1af1877160ffe7b9865b538c56c7bdd2a760aaca8ce8e38036d59"} Oct 30 17:57:55.636992 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:55.636959 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rxpmp" event={"ID":"a28935a6-1847-4f43-9e7a-332c6b3ce49f","Type":"ContainerStarted","Data":"45347a0124065a731c21fb350a2a200482923399bf73b3fe8b59b6f3d003f2fa"} Oct 30 17:57:55.716841 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:55.716735 2565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rxpmp" podStartSLOduration=4.96707056 podStartE2EDuration="12.716719736s" totalImagesPullingTime="7.749649176s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:57:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:55.716281195 +0000 UTC m=+12.777659650" watchObservedRunningTime="2026-10-30 17:57:55.716719736 +0000 UTC m=+12.778098189" Oct 30 17:57:56.441215 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:56.441182 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zpsg9" Oct 30 17:57:56.441359 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:56.441318 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zpsg9" podUID="bf5ca486-37a8-4da0-acbf-d389caf9f3e6" Oct 30 17:57:56.641245 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:56.641215 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" event={"ID":"44e585a6-cc1a-4ebc-a592-907ccee3729f","Type":"ContainerStarted","Data":"35cd7f8308c96c1e6dc6409c9cb5a9f86fbff4e6cb15ae0febafee1c3293f3be"} Oct 30 17:57:56.689836 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:56.689794 2565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" podStartSLOduration=3.812539841 podStartE2EDuration="13.689780715s" totalImagesPullingTime="9.877240874s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:57:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:56.689198871 +0000 UTC m=+13.750577323" watchObservedRunningTime="2026-10-30 17:57:56.689780715 +0000 UTC m=+13.751159167" Oct 30 17:57:57.441017 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:57.440974 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-srgjm" Oct 30 17:57:57.441227 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:57.441182 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-srgjm" podUID="a24e24cc-a321-4a62-b023-7b015f155d55" Oct 30 17:57:57.441365 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:57.441234 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9hrlx" Oct 30 17:57:57.441868 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:57.441369 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9hrlx" podUID="568da9c4-2c4f-4020-95ca-9f7ac7db7c00" Oct 30 17:57:57.650586 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:57.650554 2565 generic.go:356] "Generic (PLEG): container finished" podID="0af98be4-3c2f-413e-838e-f842854411f2" containerID="22df64c713e5a45f72b39a6b008db1e3894fbcb6737678ed96cb0793b046a9cd" exitCode=0 Oct 30 17:57:57.650758 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:57.650645 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-78h8z" event={"ID":"0af98be4-3c2f-413e-838e-f842854411f2","Type":"ContainerDied","Data":"22df64c713e5a45f72b39a6b008db1e3894fbcb6737678ed96cb0793b046a9cd"} Oct 30 17:57:58.441531 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:58.441340 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zpsg9" Oct 30 17:57:58.441944 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:58.441608 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zpsg9" podUID="bf5ca486-37a8-4da0-acbf-d389caf9f3e6" Oct 30 17:57:59.440970 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:59.440945 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-srgjm" Oct 30 17:57:59.441101 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:59.441073 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-srgjm" podUID="a24e24cc-a321-4a62-b023-7b015f155d55" Oct 30 17:57:59.441527 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:59.441491 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9hrlx" Oct 30 17:57:59.441639 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:57:59.441615 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9hrlx" podUID="568da9c4-2c4f-4020-95ca-9f7ac7db7c00" Oct 30 17:57:59.657805 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:59.657718 2565 generic.go:356] "Generic (PLEG): container finished" podID="0af98be4-3c2f-413e-838e-f842854411f2" containerID="485e737be18e64e9db94e7449f813ef1dd328082a95375b286ae2fac48b6337b" exitCode=0 Oct 30 17:57:59.657805 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:57:59.657783 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-78h8z" event={"ID":"0af98be4-3c2f-413e-838e-f842854411f2","Type":"ContainerDied","Data":"485e737be18e64e9db94e7449f813ef1dd328082a95375b286ae2fac48b6337b"} Oct 30 17:58:00.441457 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:00.441285 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zpsg9" Oct 30 17:58:00.441617 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:00.441538 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zpsg9" podUID="bf5ca486-37a8-4da0-acbf-d389caf9f3e6" Oct 30 17:58:00.663102 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:00.663012 2565 generic.go:356] "Generic (PLEG): container finished" podID="0af98be4-3c2f-413e-838e-f842854411f2" containerID="28de92773fb6953ef372ce1bcd8a6793e6b0bfdf9b22287c4530b13caedd3e54" exitCode=0 Oct 30 17:58:00.663547 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:00.663097 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-78h8z" event={"ID":"0af98be4-3c2f-413e-838e-f842854411f2","Type":"ContainerDied","Data":"28de92773fb6953ef372ce1bcd8a6793e6b0bfdf9b22287c4530b13caedd3e54"} Oct 30 17:58:00.665991 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:00.665970 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52rn6_516c6b6c-fc40-49cf-93b5-1c4471b02cf0/ovn-acl-logging/0.log" Oct 30 17:58:00.666373 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:00.666352 2565 generic.go:356] "Generic (PLEG): container finished" podID="516c6b6c-fc40-49cf-93b5-1c4471b02cf0" containerID="d07de1ff4897c756613da5d24473653c246a3684d6e33058b75bc48eab16cc08" exitCode=1 Oct 30 17:58:00.666454 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:00.666401 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" event={"ID":"516c6b6c-fc40-49cf-93b5-1c4471b02cf0","Type":"ContainerStarted","Data":"4f89321f385a75758f3e48dd250df95e52d33517257d465c5f52a0ef4279336e"} Oct 30 17:58:00.666506 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:00.666473 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" event={"ID":"516c6b6c-fc40-49cf-93b5-1c4471b02cf0","Type":"ContainerStarted","Data":"fdcf3c36ae76660b422d80673992f99ca302c7019dd14732bf54249d0dc71a0d"} Oct 30 17:58:00.666554 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:00.666525 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" event={"ID":"516c6b6c-fc40-49cf-93b5-1c4471b02cf0","Type":"ContainerStarted","Data":"bf9dcd02ba8adf0216c3275d4626faf3a9a5d401ed78be96f90374e4c4e7080b"} Oct 30 17:58:00.666608 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:00.666574 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" event={"ID":"516c6b6c-fc40-49cf-93b5-1c4471b02cf0","Type":"ContainerStarted","Data":"85a2b1272b3567a664b8c3e88fde7ba2c9d8d9611a091e00d51044465f894ec9"} Oct 30 17:58:00.666655 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:00.666624 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" event={"ID":"516c6b6c-fc40-49cf-93b5-1c4471b02cf0","Type":"ContainerDied","Data":"d07de1ff4897c756613da5d24473653c246a3684d6e33058b75bc48eab16cc08"} Oct 30 17:58:00.666708 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:00.666675 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" event={"ID":"516c6b6c-fc40-49cf-93b5-1c4471b02cf0","Type":"ContainerStarted","Data":"90b649d17d9c1e64eee92cbabaa9be7555d6ad943f03d35c32776e2616af6e2d"} Oct 30 17:58:01.141772 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:01.141703 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/568da9c4-2c4f-4020-95ca-9f7ac7db7c00-metrics-certs\") pod \"network-metrics-daemon-9hrlx\" (UID: \"568da9c4-2c4f-4020-95ca-9f7ac7db7c00\") " pod="openshift-multus/network-metrics-daemon-9hrlx" Oct 30 17:58:01.141772 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:01.141746 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a24e24cc-a321-4a62-b023-7b015f155d55-original-pull-secret\") pod \"global-pull-secret-syncer-srgjm\" (UID: \"a24e24cc-a321-4a62-b023-7b015f155d55\") " pod="kube-system/global-pull-secret-syncer-srgjm" Oct 30 17:58:01.141933 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:01.141853 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:58:01.141933 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:01.141853 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:58:01.141933 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:01.141914 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a24e24cc-a321-4a62-b023-7b015f155d55-original-pull-secret podName:a24e24cc-a321-4a62-b023-7b015f155d55 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:17.141901327 +0000 UTC m=+34.203279777 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a24e24cc-a321-4a62-b023-7b015f155d55-original-pull-secret") pod "global-pull-secret-syncer-srgjm" (UID: "a24e24cc-a321-4a62-b023-7b015f155d55") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:58:01.141933 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:01.141928 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/568da9c4-2c4f-4020-95ca-9f7ac7db7c00-metrics-certs podName:568da9c4-2c4f-4020-95ca-9f7ac7db7c00 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:17.141921904 +0000 UTC m=+34.203300377 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/568da9c4-2c4f-4020-95ca-9f7ac7db7c00-metrics-certs") pod "network-metrics-daemon-9hrlx" (UID: "568da9c4-2c4f-4020-95ca-9f7ac7db7c00") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:58:01.242526 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:01.242479 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84lr2\" (UniqueName: \"kubernetes.io/projected/bf5ca486-37a8-4da0-acbf-d389caf9f3e6-kube-api-access-84lr2\") pod \"network-check-target-zpsg9\" (UID: \"bf5ca486-37a8-4da0-acbf-d389caf9f3e6\") " pod="openshift-network-diagnostics/network-check-target-zpsg9" Oct 30 17:58:01.242692 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:01.242617 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:58:01.242692 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:01.242632 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:58:01.242692 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:01.242641 2565 projected.go:196] Error preparing data for projected volume kube-api-access-84lr2 for pod openshift-network-diagnostics/network-check-target-zpsg9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:58:01.242692 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:01.242687 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf5ca486-37a8-4da0-acbf-d389caf9f3e6-kube-api-access-84lr2 podName:bf5ca486-37a8-4da0-acbf-d389caf9f3e6 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:17.242676525 +0000 UTC m=+34.304054969 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-84lr2" (UniqueName: "kubernetes.io/projected/bf5ca486-37a8-4da0-acbf-d389caf9f3e6-kube-api-access-84lr2") pod "network-check-target-zpsg9" (UID: "bf5ca486-37a8-4da0-acbf-d389caf9f3e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:58:01.441169 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:01.441052 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-srgjm" Oct 30 17:58:01.441169 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:01.441086 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9hrlx" Oct 30 17:58:01.441344 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:01.441209 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-srgjm" podUID="a24e24cc-a321-4a62-b023-7b015f155d55" Oct 30 17:58:01.441344 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:01.441332 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9hrlx" podUID="568da9c4-2c4f-4020-95ca-9f7ac7db7c00" Oct 30 17:58:01.671883 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:01.671844 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-78h8z" event={"ID":"0af98be4-3c2f-413e-838e-f842854411f2","Type":"ContainerStarted","Data":"3dff139613af4722ac44559432c2ae89f197f5da01df105caa73739909ed6f39"} Oct 30 17:58:01.721992 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:01.721867 2565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-78h8z" podStartSLOduration=8.671236215 podStartE2EDuration="18.721849717s" totalImagesPullingTime="10.050613502s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:57:43 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:01.720955453 +0000 UTC m=+18.782333906" watchObservedRunningTime="2026-10-30 17:58:01.721849717 +0000 UTC m=+18.783228170" Oct 30 17:58:02.441274 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:02.441246 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zpsg9" Oct 30 17:58:02.441427 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:02.441343 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zpsg9" podUID="bf5ca486-37a8-4da0-acbf-d389caf9f3e6" Oct 30 17:58:02.676723 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:02.676644 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52rn6_516c6b6c-fc40-49cf-93b5-1c4471b02cf0/ovn-acl-logging/0.log" Oct 30 17:58:02.677121 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:02.677052 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" event={"ID":"516c6b6c-fc40-49cf-93b5-1c4471b02cf0","Type":"ContainerStarted","Data":"217c272eb222880b02797fe087efcdcbec2ab2b14f280062b0ab50fc833c49ce"} Oct 30 17:58:03.441636 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:03.441602 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-srgjm" Oct 30 17:58:03.441772 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:03.441712 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-srgjm" podUID="a24e24cc-a321-4a62-b023-7b015f155d55" Oct 30 17:58:03.441772 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:03.441763 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9hrlx" Oct 30 17:58:03.441884 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:03.441867 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9hrlx" podUID="568da9c4-2c4f-4020-95ca-9f7ac7db7c00" Oct 30 17:58:04.441059 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:04.441038 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zpsg9" Oct 30 17:58:04.441536 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:04.441167 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zpsg9" podUID="bf5ca486-37a8-4da0-acbf-d389caf9f3e6" Oct 30 17:58:04.687437 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:04.687350 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52rn6_516c6b6c-fc40-49cf-93b5-1c4471b02cf0/ovn-acl-logging/0.log" Oct 30 17:58:04.687718 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:04.687693 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" event={"ID":"516c6b6c-fc40-49cf-93b5-1c4471b02cf0","Type":"ContainerStarted","Data":"ec7d022f53245c5c82731a767a203fde76daf681ecd80ff7c3ad77796ac234cf"} Oct 30 17:58:04.687998 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:04.687979 2565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:04.688099 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:04.688081 2565 scope.go:122] "RemoveContainer" containerID="d07de1ff4897c756613da5d24473653c246a3684d6e33058b75bc48eab16cc08" Oct 30 17:58:04.705352 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:04.705206 2565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:05.441554 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:05.441530 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-srgjm" Oct 30 17:58:05.442009 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:05.441664 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-srgjm" podUID="a24e24cc-a321-4a62-b023-7b015f155d55" Oct 30 17:58:05.442009 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:05.441708 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9hrlx" Oct 30 17:58:05.442009 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:05.441804 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9hrlx" podUID="568da9c4-2c4f-4020-95ca-9f7ac7db7c00" Oct 30 17:58:05.694486 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:05.694414 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52rn6_516c6b6c-fc40-49cf-93b5-1c4471b02cf0/ovn-acl-logging/0.log" Oct 30 17:58:05.694826 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:05.694792 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" event={"ID":"516c6b6c-fc40-49cf-93b5-1c4471b02cf0","Type":"ContainerStarted","Data":"dfceb06a9664bd026439e356932e03cc427ab283c7390f969bf2d2beae17828a"} Oct 30 17:58:05.694931 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:05.694897 2565 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:58:05.695205 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:05.695172 2565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:05.717936 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:05.717905 2565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:05.788001 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:05.787951 2565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" podStartSLOduration=8.537244907 podStartE2EDuration="22.787934748s" totalImagesPullingTime="14.250689841s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:57:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:05.787640472 +0000 UTC m=+22.849018922" watchObservedRunningTime="2026-10-30 17:58:05.787934748 +0000 UTC m=+22.849313200" Oct 30 17:58:06.441488 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:06.441459 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zpsg9" Oct 30 17:58:06.441649 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:06.441561 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zpsg9" podUID="bf5ca486-37a8-4da0-acbf-d389caf9f3e6" Oct 30 17:58:06.553943 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:06.553871 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-srgjm"] Oct 30 17:58:06.554076 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:06.553986 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-srgjm" Oct 30 17:58:06.554076 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:06.554065 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-srgjm" podUID="a24e24cc-a321-4a62-b023-7b015f155d55" Oct 30 17:58:06.557710 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:06.557682 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9hrlx"] Oct 30 17:58:06.557841 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:06.557808 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9hrlx" Oct 30 17:58:06.557956 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:06.557927 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9hrlx" podUID="568da9c4-2c4f-4020-95ca-9f7ac7db7c00" Oct 30 17:58:06.558414 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:06.558393 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zpsg9"] Oct 30 17:58:06.696931 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:06.696899 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zpsg9" Oct 30 17:58:06.697096 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:06.696975 2565 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:58:06.697096 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:06.696994 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zpsg9" podUID="bf5ca486-37a8-4da0-acbf-d389caf9f3e6" Oct 30 17:58:07.698610 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:07.698585 2565 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:58:08.441316 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:08.441281 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9hrlx" Oct 30 17:58:08.441506 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:08.441281 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-srgjm" Oct 30 17:58:08.441506 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:08.441389 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9hrlx" podUID="568da9c4-2c4f-4020-95ca-9f7ac7db7c00" Oct 30 17:58:08.441506 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:08.441468 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-srgjm" podUID="a24e24cc-a321-4a62-b023-7b015f155d55" Oct 30 17:58:08.441506 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:08.441281 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zpsg9" Oct 30 17:58:08.441712 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:08.441546 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zpsg9" podUID="bf5ca486-37a8-4da0-acbf-d389caf9f3e6" Oct 30 17:58:09.718255 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:09.718227 2565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:09.718657 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:09.718412 2565 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:58:09.729550 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:09.729515 2565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" podUID="516c6b6c-fc40-49cf-93b5-1c4471b02cf0" containerName="ovnkube-controller" probeResult="failure" output="" Oct 30 17:58:09.738445 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:09.738416 2565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" podUID="516c6b6c-fc40-49cf-93b5-1c4471b02cf0" containerName="ovnkube-controller" probeResult="failure" output="" Oct 30 17:58:10.441437 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:10.441408 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-srgjm" Oct 30 17:58:10.441615 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:10.441486 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zpsg9" Oct 30 17:58:10.441615 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:10.441567 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9hrlx" Oct 30 17:58:10.441615 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:10.441579 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-srgjm" podUID="a24e24cc-a321-4a62-b023-7b015f155d55" Oct 30 17:58:10.441731 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:10.441639 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9hrlx" podUID="568da9c4-2c4f-4020-95ca-9f7ac7db7c00" Oct 30 17:58:10.441731 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:10.441687 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zpsg9" podUID="bf5ca486-37a8-4da0-acbf-d389caf9f3e6" Oct 30 17:58:12.440600 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:12.440572 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9hrlx" Oct 30 17:58:12.441033 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:12.440676 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zpsg9" Oct 30 17:58:12.441033 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:12.440690 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9hrlx" podUID="568da9c4-2c4f-4020-95ca-9f7ac7db7c00" Oct 30 17:58:12.441033 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:12.440748 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zpsg9" podUID="bf5ca486-37a8-4da0-acbf-d389caf9f3e6" Oct 30 17:58:12.441033 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:12.440759 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-srgjm" Oct 30 17:58:12.441033 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:12.440840 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-srgjm" podUID="a24e24cc-a321-4a62-b023-7b015f155d55" Oct 30 17:58:12.744345 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:12.744107 2565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeReady" Oct 30 17:58:12.744473 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:12.744420 2565 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 17:58:12.821398 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:12.821360 2565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4qz5q"] Oct 30 17:58:12.824738 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:12.824722 2565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2vkgd"] Oct 30 17:58:12.824869 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:12.824852 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4qz5q" Oct 30 17:58:12.827545 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:12.827528 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2vkgd" Oct 30 17:58:12.848485 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:12.848398 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2vkgd"] Oct 30 17:58:12.848632 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:12.848608 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 17:58:12.848699 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:12.848684 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 17:58:12.849071 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:12.849044 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4qz5q"] Oct 30 17:58:12.859484 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:12.859453 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-8mdkh\"" Oct 30 17:58:12.859484 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:12.859470 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 17:58:12.859684 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:12.859469 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 17:58:12.859684 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:12.859565 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rdnw2\"" Oct 30 17:58:12.859831 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:12.859809 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 17:58:12.916487 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:12.916448 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6164286-3eb9-4e3c-82bc-4886de22b3fb-cert\") pod \"ingress-canary-2vkgd\" (UID: \"d6164286-3eb9-4e3c-82bc-4886de22b3fb\") " pod="openshift-ingress-canary/ingress-canary-2vkgd" Oct 30 17:58:12.916684 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:12.916505 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8215ee49-dab0-40f1-af73-3d4493e3bc04-tmp-dir\") pod \"dns-default-4qz5q\" (UID: \"8215ee49-dab0-40f1-af73-3d4493e3bc04\") " pod="openshift-dns/dns-default-4qz5q" Oct 30 17:58:12.916684 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:12.916591 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqmfq\" (UniqueName: \"kubernetes.io/projected/8215ee49-dab0-40f1-af73-3d4493e3bc04-kube-api-access-gqmfq\") pod \"dns-default-4qz5q\" (UID: \"8215ee49-dab0-40f1-af73-3d4493e3bc04\") " pod="openshift-dns/dns-default-4qz5q" Oct 30 17:58:12.916684 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:12.916677 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8215ee49-dab0-40f1-af73-3d4493e3bc04-metrics-tls\") pod \"dns-default-4qz5q\" (UID: \"8215ee49-dab0-40f1-af73-3d4493e3bc04\") " pod="openshift-dns/dns-default-4qz5q" Oct 30 17:58:12.916851 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:12.916727 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckxh9\" (UniqueName: \"kubernetes.io/projected/d6164286-3eb9-4e3c-82bc-4886de22b3fb-kube-api-access-ckxh9\") pod \"ingress-canary-2vkgd\" (UID: \"d6164286-3eb9-4e3c-82bc-4886de22b3fb\") " pod="openshift-ingress-canary/ingress-canary-2vkgd" Oct 30 17:58:12.916851 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:12.916802 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8215ee49-dab0-40f1-af73-3d4493e3bc04-config-volume\") pod \"dns-default-4qz5q\" (UID: \"8215ee49-dab0-40f1-af73-3d4493e3bc04\") " pod="openshift-dns/dns-default-4qz5q" Oct 30 17:58:13.017844 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:13.017724 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6164286-3eb9-4e3c-82bc-4886de22b3fb-cert\") pod \"ingress-canary-2vkgd\" (UID: \"d6164286-3eb9-4e3c-82bc-4886de22b3fb\") " pod="openshift-ingress-canary/ingress-canary-2vkgd" Oct 30 17:58:13.017844 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:13.017812 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8215ee49-dab0-40f1-af73-3d4493e3bc04-tmp-dir\") pod \"dns-default-4qz5q\" (UID: \"8215ee49-dab0-40f1-af73-3d4493e3bc04\") " pod="openshift-dns/dns-default-4qz5q" Oct 30 17:58:13.017844 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:13.017834 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqmfq\" (UniqueName: \"kubernetes.io/projected/8215ee49-dab0-40f1-af73-3d4493e3bc04-kube-api-access-gqmfq\") pod \"dns-default-4qz5q\" (UID: \"8215ee49-dab0-40f1-af73-3d4493e3bc04\") " pod="openshift-dns/dns-default-4qz5q" Oct 30 17:58:13.018117 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:13.017874 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8215ee49-dab0-40f1-af73-3d4493e3bc04-metrics-tls\") pod \"dns-default-4qz5q\" (UID: \"8215ee49-dab0-40f1-af73-3d4493e3bc04\") " pod="openshift-dns/dns-default-4qz5q" Oct 30 17:58:13.018117 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:13.017918 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckxh9\" (UniqueName: \"kubernetes.io/projected/d6164286-3eb9-4e3c-82bc-4886de22b3fb-kube-api-access-ckxh9\") pod \"ingress-canary-2vkgd\" (UID: \"d6164286-3eb9-4e3c-82bc-4886de22b3fb\") " pod="openshift-ingress-canary/ingress-canary-2vkgd" Oct 30 17:58:13.018117 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:13.018046 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8215ee49-dab0-40f1-af73-3d4493e3bc04-config-volume\") pod \"dns-default-4qz5q\" (UID: \"8215ee49-dab0-40f1-af73-3d4493e3bc04\") " pod="openshift-dns/dns-default-4qz5q" Oct 30 17:58:13.018117 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:13.018049 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:58:13.018328 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:13.018180 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8215ee49-dab0-40f1-af73-3d4493e3bc04-metrics-tls podName:8215ee49-dab0-40f1-af73-3d4493e3bc04 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:13.518158832 +0000 UTC m=+30.579537280 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8215ee49-dab0-40f1-af73-3d4493e3bc04-metrics-tls") pod "dns-default-4qz5q" (UID: "8215ee49-dab0-40f1-af73-3d4493e3bc04") : secret "dns-default-metrics-tls" not found Oct 30 17:58:13.018443 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:13.018424 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8215ee49-dab0-40f1-af73-3d4493e3bc04-tmp-dir\") pod \"dns-default-4qz5q\" (UID: \"8215ee49-dab0-40f1-af73-3d4493e3bc04\") " pod="openshift-dns/dns-default-4qz5q" Oct 30 17:58:13.018662 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:13.018638 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8215ee49-dab0-40f1-af73-3d4493e3bc04-config-volume\") pod \"dns-default-4qz5q\" (UID: \"8215ee49-dab0-40f1-af73-3d4493e3bc04\") " pod="openshift-dns/dns-default-4qz5q" Oct 30 17:58:13.022066 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:13.022040 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6164286-3eb9-4e3c-82bc-4886de22b3fb-cert\") pod \"ingress-canary-2vkgd\" (UID: \"d6164286-3eb9-4e3c-82bc-4886de22b3fb\") " pod="openshift-ingress-canary/ingress-canary-2vkgd" Oct 30 17:58:13.038953 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:13.038914 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqmfq\" (UniqueName: \"kubernetes.io/projected/8215ee49-dab0-40f1-af73-3d4493e3bc04-kube-api-access-gqmfq\") pod \"dns-default-4qz5q\" (UID: \"8215ee49-dab0-40f1-af73-3d4493e3bc04\") " pod="openshift-dns/dns-default-4qz5q" Oct 30 17:58:13.039086 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:13.039067 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckxh9\" (UniqueName: \"kubernetes.io/projected/d6164286-3eb9-4e3c-82bc-4886de22b3fb-kube-api-access-ckxh9\") pod \"ingress-canary-2vkgd\" (UID: \"d6164286-3eb9-4e3c-82bc-4886de22b3fb\") " pod="openshift-ingress-canary/ingress-canary-2vkgd" Oct 30 17:58:13.141345 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:13.141307 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2vkgd" Oct 30 17:58:13.451539 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:13.451436 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2vkgd"] Oct 30 17:58:13.456261 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:58:13.456236 2565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6164286_3eb9_4e3c_82bc_4886de22b3fb.slice/crio-af185b4ea7b5d4e8cc75783cd205502cd36711430d5772eced1ea92c6bec8bf1 WatchSource:0}: Error finding container af185b4ea7b5d4e8cc75783cd205502cd36711430d5772eced1ea92c6bec8bf1: Status 404 returned error can't find the container with id af185b4ea7b5d4e8cc75783cd205502cd36711430d5772eced1ea92c6bec8bf1 Oct 30 17:58:13.522521 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:13.522482 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8215ee49-dab0-40f1-af73-3d4493e3bc04-metrics-tls\") pod \"dns-default-4qz5q\" (UID: \"8215ee49-dab0-40f1-af73-3d4493e3bc04\") " pod="openshift-dns/dns-default-4qz5q" Oct 30 17:58:13.522690 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:13.522654 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:58:13.522732 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:13.522722 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8215ee49-dab0-40f1-af73-3d4493e3bc04-metrics-tls podName:8215ee49-dab0-40f1-af73-3d4493e3bc04 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:14.522704128 +0000 UTC m=+31.584082577 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8215ee49-dab0-40f1-af73-3d4493e3bc04-metrics-tls") pod "dns-default-4qz5q" (UID: "8215ee49-dab0-40f1-af73-3d4493e3bc04") : secret "dns-default-metrics-tls" not found Oct 30 17:58:13.713896 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:13.713819 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2vkgd" event={"ID":"d6164286-3eb9-4e3c-82bc-4886de22b3fb","Type":"ContainerStarted","Data":"af185b4ea7b5d4e8cc75783cd205502cd36711430d5772eced1ea92c6bec8bf1"} Oct 30 17:58:14.441610 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:14.441570 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9hrlx" Oct 30 17:58:14.441792 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:14.441611 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-srgjm" Oct 30 17:58:14.441792 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:14.441570 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zpsg9" Oct 30 17:58:14.457276 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:14.457249 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 17:58:14.457661 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:14.457556 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 17:58:14.457661 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:14.457636 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 17:58:14.458047 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:14.458022 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xl6hf\"" Oct 30 17:58:14.458166 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:14.458109 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-2xlsx\"" Oct 30 17:58:14.469873 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:14.469847 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6fpb9\"" Oct 30 17:58:14.469980 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:14.469935 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 17:58:14.531148 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:14.531098 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8215ee49-dab0-40f1-af73-3d4493e3bc04-metrics-tls\") pod \"dns-default-4qz5q\" (UID: \"8215ee49-dab0-40f1-af73-3d4493e3bc04\") " pod="openshift-dns/dns-default-4qz5q" Oct 30 17:58:14.531303 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:14.531270 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:58:14.531368 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:14.531352 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8215ee49-dab0-40f1-af73-3d4493e3bc04-metrics-tls podName:8215ee49-dab0-40f1-af73-3d4493e3bc04 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:16.53133013 +0000 UTC m=+33.592708565 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8215ee49-dab0-40f1-af73-3d4493e3bc04-metrics-tls") pod "dns-default-4qz5q" (UID: "8215ee49-dab0-40f1-af73-3d4493e3bc04") : secret "dns-default-metrics-tls" not found Oct 30 17:58:16.549900 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:16.547543 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8215ee49-dab0-40f1-af73-3d4493e3bc04-metrics-tls\") pod \"dns-default-4qz5q\" (UID: \"8215ee49-dab0-40f1-af73-3d4493e3bc04\") " pod="openshift-dns/dns-default-4qz5q" Oct 30 17:58:16.549900 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:16.547829 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:58:16.549900 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:16.547920 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8215ee49-dab0-40f1-af73-3d4493e3bc04-metrics-tls podName:8215ee49-dab0-40f1-af73-3d4493e3bc04 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:20.547900032 +0000 UTC m=+37.609278480 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8215ee49-dab0-40f1-af73-3d4493e3bc04-metrics-tls") pod "dns-default-4qz5q" (UID: "8215ee49-dab0-40f1-af73-3d4493e3bc04") : secret "dns-default-metrics-tls" not found Oct 30 17:58:16.722353 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:16.722313 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2vkgd" event={"ID":"d6164286-3eb9-4e3c-82bc-4886de22b3fb","Type":"ContainerStarted","Data":"e3fb319da8a17d93712546dcd3c04016d743c6b8a354ac483ecae5dbd40c6d52"} Oct 30 17:58:16.762656 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:16.762561 2565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2vkgd" podStartSLOduration=2.008185894 podStartE2EDuration="4.762548845s" totalImagesPullingTime="2.754362951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:58:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:16.761893434 +0000 UTC m=+33.823271885" watchObservedRunningTime="2026-10-30 17:58:16.762548845 +0000 UTC m=+33.823927343" Oct 30 17:58:17.152842 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:17.152758 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a24e24cc-a321-4a62-b023-7b015f155d55-original-pull-secret\") pod \"global-pull-secret-syncer-srgjm\" (UID: \"a24e24cc-a321-4a62-b023-7b015f155d55\") " pod="kube-system/global-pull-secret-syncer-srgjm" Oct 30 17:58:17.152842 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:17.152823 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/568da9c4-2c4f-4020-95ca-9f7ac7db7c00-metrics-certs\") pod \"network-metrics-daemon-9hrlx\" (UID: \"568da9c4-2c4f-4020-95ca-9f7ac7db7c00\") " pod="openshift-multus/network-metrics-daemon-9hrlx" Oct 30 17:58:17.153059 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:17.152913 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 17:58:17.153059 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:17.152966 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/568da9c4-2c4f-4020-95ca-9f7ac7db7c00-metrics-certs podName:568da9c4-2c4f-4020-95ca-9f7ac7db7c00 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:49.152953039 +0000 UTC m=+66.214331484 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/568da9c4-2c4f-4020-95ca-9f7ac7db7c00-metrics-certs") pod "network-metrics-daemon-9hrlx" (UID: "568da9c4-2c4f-4020-95ca-9f7ac7db7c00") : secret "metrics-daemon-secret" not found Oct 30 17:58:17.155303 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:17.155278 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a24e24cc-a321-4a62-b023-7b015f155d55-original-pull-secret\") pod \"global-pull-secret-syncer-srgjm\" (UID: \"a24e24cc-a321-4a62-b023-7b015f155d55\") " pod="kube-system/global-pull-secret-syncer-srgjm" Oct 30 17:58:17.198877 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:17.198847 2565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-97w68"] Oct 30 17:58:17.222579 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:17.222547 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-srgjm" Oct 30 17:58:17.253830 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:17.253797 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84lr2\" (UniqueName: \"kubernetes.io/projected/bf5ca486-37a8-4da0-acbf-d389caf9f3e6-kube-api-access-84lr2\") pod \"network-check-target-zpsg9\" (UID: \"bf5ca486-37a8-4da0-acbf-d389caf9f3e6\") " pod="openshift-network-diagnostics/network-check-target-zpsg9" Oct 30 17:58:17.256452 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:17.256424 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84lr2\" (UniqueName: \"kubernetes.io/projected/bf5ca486-37a8-4da0-acbf-d389caf9f3e6-kube-api-access-84lr2\") pod \"network-check-target-zpsg9\" (UID: \"bf5ca486-37a8-4da0-acbf-d389caf9f3e6\") " pod="openshift-network-diagnostics/network-check-target-zpsg9" Oct 30 17:58:17.257492 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:17.257475 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-97w68"] Oct 30 17:58:17.257569 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:17.257559 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-97w68" Oct 30 17:58:17.314653 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:17.314627 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 17:58:17.314833 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:17.314817 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-t4kbs\"" Oct 30 17:58:17.314883 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:17.314862 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 17:58:17.355177 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:17.355128 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2wnc\" (UniqueName: \"kubernetes.io/projected/a6842e7e-ad53-424d-be89-f6b29f22ea24-kube-api-access-j2wnc\") pod \"migrator-66df5d496b-97w68\" (UID: \"a6842e7e-ad53-424d-be89-f6b29f22ea24\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-97w68" Oct 30 17:58:17.366660 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:17.366631 2565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vlnv2"] Oct 30 17:58:17.387884 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:17.387860 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vlnv2" Oct 30 17:58:17.392029 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:17.392004 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vlnv2"] Oct 30 17:58:17.406604 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:17.406556 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 17:58:17.406778 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:17.406761 2565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mngtd\"" Oct 30 17:58:17.415302 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:17.415281 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 17:58:17.415418 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:17.415313 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 17:58:17.415572 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:17.415559 2565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 17:58:17.456149 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:17.456099 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99kgv\" (UniqueName: \"kubernetes.io/projected/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-kube-api-access-99kgv\") pod \"insights-runtime-extractor-vlnv2\" (UID: \"245afcc3-9c86-4c4e-ac8f-9f7d427c5920\") " pod="openshift-insights/insights-runtime-extractor-vlnv2" Oct 30 17:58:17.456267 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:17.456157 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2wnc\" (UniqueName: \"kubernetes.io/projected/a6842e7e-ad53-424d-be89-f6b29f22ea24-kube-api-access-j2wnc\") pod \"migrator-66df5d496b-97w68\" (UID: \"a6842e7e-ad53-424d-be89-f6b29f22ea24\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-97w68" Oct 30 17:58:17.456340 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:17.456319 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vlnv2\" (UID: \"245afcc3-9c86-4c4e-ac8f-9f7d427c5920\") " pod="openshift-insights/insights-runtime-extractor-vlnv2" Oct 30 17:58:17.456434 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:17.456388 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vlnv2\" (UID: \"245afcc3-9c86-4c4e-ac8f-9f7d427c5920\") " pod="openshift-insights/insights-runtime-extractor-vlnv2" Oct 30 17:58:17.456504 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:17.456481 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-crio-socket\") pod \"insights-runtime-extractor-vlnv2\" (UID: \"245afcc3-9c86-4c4e-ac8f-9f7d427c5920\") " pod="openshift-insights/insights-runtime-extractor-vlnv2" Oct 30 17:58:17.456537 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:17.456522 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-data-volume\") pod \"insights-runtime-extractor-vlnv2\" (UID: \"245afcc3-9c86-4c4e-ac8f-9f7d427c5920\") " pod="openshift-insights/insights-runtime-extractor-vlnv2" Oct 30 17:58:17.477797 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:17.477778 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2wnc\" (UniqueName: \"kubernetes.io/projected/a6842e7e-ad53-424d-be89-f6b29f22ea24-kube-api-access-j2wnc\") pod \"migrator-66df5d496b-97w68\" (UID: \"a6842e7e-ad53-424d-be89-f6b29f22ea24\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-97w68" Oct 30 17:58:17.541062 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:17.541027 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zpsg9" Oct 30 17:58:17.553953 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:17.553925 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-srgjm"] Oct 30 17:58:17.556624 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:58:17.556599 2565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda24e24cc_a321_4a62_b023_7b015f155d55.slice/crio-368fe8a43f1f9d87c0fd555c1f310648af3f9c8792bd44010dde89dd3412a7de WatchSource:0}: Error finding container 368fe8a43f1f9d87c0fd555c1f310648af3f9c8792bd44010dde89dd3412a7de: Status 404 returned error can't find the container with id 368fe8a43f1f9d87c0fd555c1f310648af3f9c8792bd44010dde89dd3412a7de Oct 30 17:58:17.556840 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:17.556823 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vlnv2\" (UID: \"245afcc3-9c86-4c4e-ac8f-9f7d427c5920\") " pod="openshift-insights/insights-runtime-extractor-vlnv2" Oct 30 17:58:17.556907 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:17.556873 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vlnv2\" (UID: \"245afcc3-9c86-4c4e-ac8f-9f7d427c5920\") " pod="openshift-insights/insights-runtime-extractor-vlnv2" Oct 30 17:58:17.556962 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:17.556927 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-crio-socket\") pod \"insights-runtime-extractor-vlnv2\" (UID: \"245afcc3-9c86-4c4e-ac8f-9f7d427c5920\") " pod="openshift-insights/insights-runtime-extractor-vlnv2" Oct 30 17:58:17.557016 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:17.556966 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-data-volume\") pod \"insights-runtime-extractor-vlnv2\" (UID: \"245afcc3-9c86-4c4e-ac8f-9f7d427c5920\") " pod="openshift-insights/insights-runtime-extractor-vlnv2" Oct 30 17:58:17.557016 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:17.556980 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:58:17.557016 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:17.556991 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 17:58:17.557016 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:17.556999 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99kgv\" (UniqueName: \"kubernetes.io/projected/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-kube-api-access-99kgv\") pod \"insights-runtime-extractor-vlnv2\" (UID: \"245afcc3-9c86-4c4e-ac8f-9f7d427c5920\") " pod="openshift-insights/insights-runtime-extractor-vlnv2" Oct 30 17:58:17.557220 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:17.557044 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-insights-runtime-extractor-tls podName:245afcc3-9c86-4c4e-ac8f-9f7d427c5920 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:18.057024346 +0000 UTC m=+35.118402798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vlnv2" (UID: "245afcc3-9c86-4c4e-ac8f-9f7d427c5920") : secret "insights-runtime-extractor-tls" not found Oct 30 17:58:17.557220 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:17.557064 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-kube-rbac-proxy-cm podName:245afcc3-9c86-4c4e-ac8f-9f7d427c5920 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:18.057054706 +0000 UTC m=+35.118433139 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vlnv2" (UID: "245afcc3-9c86-4c4e-ac8f-9f7d427c5920") : configmap "kube-rbac-proxy" not found Oct 30 17:58:17.557220 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:17.557115 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-crio-socket\") pod \"insights-runtime-extractor-vlnv2\" (UID: \"245afcc3-9c86-4c4e-ac8f-9f7d427c5920\") " pod="openshift-insights/insights-runtime-extractor-vlnv2" Oct 30 17:58:17.557345 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:17.557306 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-data-volume\") pod \"insights-runtime-extractor-vlnv2\" (UID: \"245afcc3-9c86-4c4e-ac8f-9f7d427c5920\") " pod="openshift-insights/insights-runtime-extractor-vlnv2" Oct 30 17:58:17.577060 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:17.577028 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99kgv\" (UniqueName: \"kubernetes.io/projected/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-kube-api-access-99kgv\") pod \"insights-runtime-extractor-vlnv2\" (UID: \"245afcc3-9c86-4c4e-ac8f-9f7d427c5920\") " pod="openshift-insights/insights-runtime-extractor-vlnv2" Oct 30 17:58:17.589233 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:17.582854 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-97w68" Oct 30 17:58:17.726204 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:17.726168 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-srgjm" event={"ID":"a24e24cc-a321-4a62-b023-7b015f155d55","Type":"ContainerStarted","Data":"368fe8a43f1f9d87c0fd555c1f310648af3f9c8792bd44010dde89dd3412a7de"} Oct 30 17:58:17.851966 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:17.851937 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zpsg9"] Oct 30 17:58:17.854941 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:58:17.854920 2565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf5ca486_37a8_4da0_acbf_d389caf9f3e6.slice/crio-507d24b53aa3c10ca3e0817caae8defea37f8597936fa8f0a8633359eaa7a42d WatchSource:0}: Error finding container 507d24b53aa3c10ca3e0817caae8defea37f8597936fa8f0a8633359eaa7a42d: Status 404 returned error can't find the container with id 507d24b53aa3c10ca3e0817caae8defea37f8597936fa8f0a8633359eaa7a42d Oct 30 17:58:17.899067 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:17.899043 2565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-97w68"] Oct 30 17:58:17.901700 ip-10-0-138-176 kubenswrapper[2565]: W1030 17:58:17.901678 2565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6842e7e_ad53_424d_be89_f6b29f22ea24.slice/crio-7541c221aa4e62cb01d078764e9f799e80eb222aa78fabb01004b143f09a8534 WatchSource:0}: Error finding container 7541c221aa4e62cb01d078764e9f799e80eb222aa78fabb01004b143f09a8534: Status 404 returned error can't find the container with id 7541c221aa4e62cb01d078764e9f799e80eb222aa78fabb01004b143f09a8534 Oct 30 17:58:18.060111 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:18.060027 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vlnv2\" (UID: \"245afcc3-9c86-4c4e-ac8f-9f7d427c5920\") " pod="openshift-insights/insights-runtime-extractor-vlnv2" Oct 30 17:58:18.060111 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:18.060083 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vlnv2\" (UID: \"245afcc3-9c86-4c4e-ac8f-9f7d427c5920\") " pod="openshift-insights/insights-runtime-extractor-vlnv2" Oct 30 17:58:18.060344 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:18.060190 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 17:58:18.060344 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:18.060218 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:58:18.060344 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:18.060243 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-kube-rbac-proxy-cm podName:245afcc3-9c86-4c4e-ac8f-9f7d427c5920 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:19.060230622 +0000 UTC m=+36.121609054 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vlnv2" (UID: "245afcc3-9c86-4c4e-ac8f-9f7d427c5920") : configmap "kube-rbac-proxy" not found Oct 30 17:58:18.060344 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:18.060284 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-insights-runtime-extractor-tls podName:245afcc3-9c86-4c4e-ac8f-9f7d427c5920 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:19.060265131 +0000 UTC m=+36.121643575 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vlnv2" (UID: "245afcc3-9c86-4c4e-ac8f-9f7d427c5920") : secret "insights-runtime-extractor-tls" not found Oct 30 17:58:18.730452 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:18.730417 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-97w68" event={"ID":"a6842e7e-ad53-424d-be89-f6b29f22ea24","Type":"ContainerStarted","Data":"7541c221aa4e62cb01d078764e9f799e80eb222aa78fabb01004b143f09a8534"} Oct 30 17:58:18.731794 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:18.731765 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zpsg9" event={"ID":"bf5ca486-37a8-4da0-acbf-d389caf9f3e6","Type":"ContainerStarted","Data":"507d24b53aa3c10ca3e0817caae8defea37f8597936fa8f0a8633359eaa7a42d"} Oct 30 17:58:19.067907 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:19.067805 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vlnv2\" (UID: \"245afcc3-9c86-4c4e-ac8f-9f7d427c5920\") " pod="openshift-insights/insights-runtime-extractor-vlnv2" Oct 30 17:58:19.067907 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:19.067876 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vlnv2\" (UID: \"245afcc3-9c86-4c4e-ac8f-9f7d427c5920\") " pod="openshift-insights/insights-runtime-extractor-vlnv2" Oct 30 17:58:19.068174 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:19.067999 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 17:58:19.068174 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:19.068005 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:58:19.068174 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:19.068084 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-kube-rbac-proxy-cm podName:245afcc3-9c86-4c4e-ac8f-9f7d427c5920 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:21.068055627 +0000 UTC m=+38.129434073 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vlnv2" (UID: "245afcc3-9c86-4c4e-ac8f-9f7d427c5920") : configmap "kube-rbac-proxy" not found Oct 30 17:58:19.068174 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:19.068104 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-insights-runtime-extractor-tls podName:245afcc3-9c86-4c4e-ac8f-9f7d427c5920 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:21.068093775 +0000 UTC m=+38.129472208 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vlnv2" (UID: "245afcc3-9c86-4c4e-ac8f-9f7d427c5920") : secret "insights-runtime-extractor-tls" not found Oct 30 17:58:19.737784 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:19.737746 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-97w68" event={"ID":"a6842e7e-ad53-424d-be89-f6b29f22ea24","Type":"ContainerStarted","Data":"78544087e462630d88baf14a4674bac4ef5b76fc2842bee72ec6d70ce84c2dc6"} Oct 30 17:58:19.789292 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:19.789186 2565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-97w68" podStartSLOduration=1.164966951 podStartE2EDuration="2.789166547s" totalImagesPullingTime="1.624199596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:58:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:19.787165941 +0000 UTC m=+36.848544407" watchObservedRunningTime="2026-10-30 17:58:19.789166547 +0000 UTC m=+36.850545002" Oct 30 17:58:20.580334 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:20.580292 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8215ee49-dab0-40f1-af73-3d4493e3bc04-metrics-tls\") pod \"dns-default-4qz5q\" (UID: \"8215ee49-dab0-40f1-af73-3d4493e3bc04\") " pod="openshift-dns/dns-default-4qz5q" Oct 30 17:58:20.580516 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:20.580457 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:58:20.580516 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:20.580510 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8215ee49-dab0-40f1-af73-3d4493e3bc04-metrics-tls podName:8215ee49-dab0-40f1-af73-3d4493e3bc04 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:28.58049246 +0000 UTC m=+45.641870892 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8215ee49-dab0-40f1-af73-3d4493e3bc04-metrics-tls") pod "dns-default-4qz5q" (UID: "8215ee49-dab0-40f1-af73-3d4493e3bc04") : secret "dns-default-metrics-tls" not found Oct 30 17:58:21.084101 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:21.084063 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vlnv2\" (UID: \"245afcc3-9c86-4c4e-ac8f-9f7d427c5920\") " pod="openshift-insights/insights-runtime-extractor-vlnv2" Oct 30 17:58:21.084524 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:21.084126 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vlnv2\" (UID: \"245afcc3-9c86-4c4e-ac8f-9f7d427c5920\") " pod="openshift-insights/insights-runtime-extractor-vlnv2" Oct 30 17:58:21.084524 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:21.084223 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 17:58:21.084524 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:21.084238 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:58:21.084524 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:21.084297 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-insights-runtime-extractor-tls podName:245afcc3-9c86-4c4e-ac8f-9f7d427c5920 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:25.084279339 +0000 UTC m=+42.145657775 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vlnv2" (UID: "245afcc3-9c86-4c4e-ac8f-9f7d427c5920") : secret "insights-runtime-extractor-tls" not found Oct 30 17:58:21.084524 ip-10-0-138-176 kubenswrapper[2565]: E1030 17:58:21.084317 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-kube-rbac-proxy-cm podName:245afcc3-9c86-4c4e-ac8f-9f7d427c5920 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:25.084308459 +0000 UTC m=+42.145686891 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vlnv2" (UID: "245afcc3-9c86-4c4e-ac8f-9f7d427c5920") : configmap "kube-rbac-proxy" not found Oct 30 17:58:21.745967 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:21.745888 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zpsg9" event={"ID":"bf5ca486-37a8-4da0-acbf-d389caf9f3e6","Type":"ContainerStarted","Data":"f165540fe9ee0d162e2803457e495f991b6d7de26842c12906bcb030ef756d9f"} Oct 30 17:58:21.746230 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:21.746047 2565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zpsg9" Oct 30 17:58:21.789249 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:21.789193 2565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zpsg9" podStartSLOduration=35.337102913 podStartE2EDuration="38.78917915s" totalImagesPullingTime="3.452076237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:57:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:21.787955262 +0000 UTC m=+38.849333717" watchObservedRunningTime="2026-10-30 17:58:21.78917915 +0000 UTC m=+38.850557602" Oct 30 17:58:22.750488 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:22.750452 2565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-srgjm" event={"ID":"a24e24cc-a321-4a62-b023-7b015f155d55","Type":"ContainerStarted","Data":"4752be286cbd9cbffc444fcd53312f6b10da0dbe35049d7bb0fb824aadc63359"} Oct 30 17:58:22.795260 ip-10-0-138-176 kubenswrapper[2565]: I1030 17:58:22.795161 2565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-srgjm" podStartSLOduration=34.81838835 podStartE2EDuration="39.795145174s" totalImagesPullingTime="4.976756824s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:57:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:22.795024869 +0000 UTC m=+39.856403324" watchObservedRunningTime="2026-10-30 17:58:22.795145174 +0000 UTC m=+39.856523617" Oct 30 17:58:22.849550 ip-10-0-138-176 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 17:58:22.860566 ip-10-0-138-176 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 17:58:22.860741 ip-10-0-138-176 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 17:58:22.860790 ip-10-0-138-176 systemd[1]: kubelet.service: Consumed 2.373s CPU time. Oct 30 17:58:22.869758 ip-10-0-138-176 systemd[1]: Starting Kubernetes Kubelet... Oct 30 17:58:22.977527 ip-10-0-138-176 kubenswrapper[4442]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:58:22.977527 ip-10-0-138-176 kubenswrapper[4442]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 17:58:22.977527 ip-10-0-138-176 kubenswrapper[4442]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:58:22.977527 ip-10-0-138-176 kubenswrapper[4442]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:58:22.979984 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.979879 4442 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:58:22.979984 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.979984 4442 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:58:22.979984 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.979988 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:58:22.980088 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.979992 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:58:22.980088 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.979995 4442 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:58:22.980088 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.979998 4442 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:58:22.980088 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980001 4442 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:58:22.980088 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980004 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:58:22.980088 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980008 4442 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:58:22.980088 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980013 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:58:22.980088 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980017 4442 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:58:22.980088 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980020 4442 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:58:22.980088 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980023 4442 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:58:22.980088 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980026 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:58:22.980088 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980029 4442 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:58:22.980088 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980031 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:58:22.980088 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980034 4442 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:58:22.980088 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980037 4442 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:58:22.980088 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980040 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:58:22.980088 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980042 4442 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:58:22.980088 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980045 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:58:22.980088 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980054 4442 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:58:22.980624 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980057 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:58:22.980624 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980059 4442 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:58:22.980624 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980062 4442 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:58:22.980624 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980065 4442 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:58:22.980624 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980068 4442 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:58:22.980624 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980071 4442 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:58:22.980624 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980074 4442 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:58:22.980624 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980077 4442 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:58:22.980624 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980080 4442 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:58:22.980624 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980082 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:58:22.980624 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980085 4442 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:58:22.980624 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980088 4442 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:58:22.980624 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980090 4442 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:58:22.980624 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980093 4442 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:58:22.980624 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980097 4442 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:58:22.980624 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980100 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:58:22.980624 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980103 4442 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:58:22.980624 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980105 4442 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:58:22.980624 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980108 4442 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:58:22.980624 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980111 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:58:22.981127 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980114 4442 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:58:22.981127 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980116 4442 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:58:22.981127 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980119 4442 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:58:22.981127 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980122 4442 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:58:22.981127 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980125 4442 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:58:22.981127 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980128 4442 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:58:22.981127 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980145 4442 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:58:22.981127 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980147 4442 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:58:22.981127 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980150 4442 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:58:22.981127 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980152 4442 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:58:22.981127 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980155 4442 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:58:22.981127 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980159 4442 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:58:22.981127 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980162 4442 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:58:22.981127 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980166 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:58:22.981127 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980169 4442 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:58:22.981127 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980171 4442 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:58:22.981127 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980174 4442 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:58:22.981127 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980177 4442 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:58:22.981127 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980180 4442 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:58:22.981613 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980183 4442 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:58:22.981613 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980186 4442 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:58:22.981613 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980188 4442 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:58:22.981613 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980191 4442 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:58:22.981613 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980194 4442 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:58:22.981613 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980196 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:58:22.981613 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980199 4442 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:58:22.981613 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980202 4442 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:58:22.981613 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980204 4442 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:58:22.981613 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980207 4442 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:58:22.981613 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980210 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:58:22.981613 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980213 4442 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:58:22.981613 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980215 4442 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:58:22.981613 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980218 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:58:22.981613 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980221 4442 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:58:22.981613 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980223 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:58:22.981613 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980226 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:58:22.981613 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980229 4442 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:58:22.981613 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980232 4442 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:58:22.981613 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980235 4442 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:58:22.982110 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980237 4442 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:58:22.982110 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980240 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:58:22.982110 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980243 4442 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:58:22.982110 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980245 4442 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:58:22.982110 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980248 4442 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:58:22.982110 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980251 4442 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:58:22.982110 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980255 4442 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:58:22.982110 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980259 4442 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:58:22.982110 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980264 4442 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:58:22.982110 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980267 4442 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:58:22.982110 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980270 4442 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:58:22.982110 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980273 4442 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:58:22.982110 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980277 4442 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:58:22.982110 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980279 4442 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:58:22.982110 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980282 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:58:22.982110 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980284 4442 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:58:22.982110 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980288 4442 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:58:22.982110 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980291 4442 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:58:22.982110 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980293 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:58:22.982110 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980296 4442 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:58:22.982651 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980298 4442 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:58:22.982651 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980301 4442 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:58:22.982651 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980303 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:58:22.982651 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980306 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:58:22.982651 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.980309 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:58:22.982651 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981215 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:58:22.982651 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981227 4442 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:58:22.982651 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981232 4442 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:58:22.982651 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981237 4442 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:58:22.982651 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981243 4442 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:58:22.982651 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981249 4442 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:58:22.982651 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981253 4442 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:58:22.982651 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981258 4442 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:58:22.982651 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981262 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:58:22.982651 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981266 4442 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:58:22.982651 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981271 4442 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:58:22.982651 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981276 4442 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:58:22.982651 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981281 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:58:22.982651 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981286 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:58:22.982651 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981291 4442 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:58:22.983193 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981297 4442 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:58:22.983193 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981301 4442 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:58:22.983193 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981304 4442 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:58:22.983193 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981306 4442 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:58:22.983193 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981310 4442 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:58:22.983193 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981312 4442 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:58:22.983193 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981315 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:58:22.983193 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981318 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:58:22.983193 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981320 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:58:22.983193 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981323 4442 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:58:22.983193 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981326 4442 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:58:22.983193 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981328 4442 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:58:22.983193 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981331 4442 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:58:22.983193 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981336 4442 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:58:22.983193 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981340 4442 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:58:22.983193 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981343 4442 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:58:22.983193 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981346 4442 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:58:22.983193 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981348 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:58:22.983193 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981351 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:58:22.983679 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981354 4442 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:58:22.983679 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981358 4442 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:58:22.983679 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981363 4442 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:58:22.983679 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981367 4442 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:58:22.983679 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981371 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:58:22.983679 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981376 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:58:22.983679 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981381 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:58:22.983679 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981385 4442 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:58:22.983679 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981389 4442 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:58:22.983679 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981394 4442 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:58:22.983679 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981399 4442 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:58:22.983679 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981408 4442 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:58:22.983679 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981417 4442 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:58:22.983679 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981424 4442 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:58:22.983679 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981429 4442 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:58:22.983679 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981434 4442 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:58:22.983679 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981439 4442 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:58:22.983679 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981444 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:58:22.983679 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981448 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:58:22.984435 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981453 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:58:22.984435 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981458 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:58:22.984435 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981463 4442 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:58:22.984435 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981468 4442 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:58:22.984435 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981473 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:58:22.984435 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981476 4442 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:58:22.984435 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981479 4442 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:58:22.984435 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981482 4442 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:58:22.984435 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981485 4442 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:58:22.984435 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981487 4442 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:58:22.984435 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981490 4442 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:58:22.984435 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981493 4442 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:58:22.984435 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981496 4442 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:58:22.984435 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981498 4442 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:58:22.984435 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981501 4442 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:58:22.984435 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981504 4442 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:58:22.984435 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981506 4442 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:58:22.984435 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981509 4442 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:58:22.984435 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981511 4442 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:58:22.984435 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981515 4442 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:58:22.985043 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981517 4442 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:58:22.985043 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981520 4442 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:58:22.985043 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981523 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:58:22.985043 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981525 4442 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:58:22.985043 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981528 4442 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:58:22.985043 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981531 4442 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:58:22.985043 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981535 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:58:22.985043 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981540 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:58:22.985043 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981544 4442 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:58:22.985043 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981548 4442 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:58:22.985043 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981552 4442 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:58:22.985043 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981561 4442 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:58:22.985043 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981566 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:58:22.985043 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981571 4442 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:58:22.985043 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981576 4442 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:58:22.985043 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981579 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:58:22.985043 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981581 4442 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:58:22.985043 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981585 4442 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:58:22.985043 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981588 4442 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:58:22.985043 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981591 4442 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:58:22.985562 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981593 4442 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:58:22.985562 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981596 4442 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:58:22.985562 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981598 4442 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:58:22.985562 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981602 4442 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:58:22.985562 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981607 4442 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:58:22.985562 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981612 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:58:22.985562 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981616 4442 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:58:22.985562 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981621 4442 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:58:22.985562 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981625 4442 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:58:22.985562 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981630 4442 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:58:22.985562 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981634 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:58:22.985562 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981639 4442 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:58:22.985562 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.981644 4442 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:58:22.985562 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982533 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:58:22.985562 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982543 4442 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:58:22.985562 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982548 4442 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:58:22.985562 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982553 4442 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:58:22.985562 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982558 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:58:22.985562 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982564 4442 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:58:22.985562 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982569 4442 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:58:22.986097 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982574 4442 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:58:22.986097 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982578 4442 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:58:22.986097 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982583 4442 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:58:22.986097 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982590 4442 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:58:22.986097 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982595 4442 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:58:22.986097 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982600 4442 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:58:22.986097 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982604 4442 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:58:22.986097 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982609 4442 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:58:22.986097 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982614 4442 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:58:22.986097 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982619 4442 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:58:22.986097 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982623 4442 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:58:22.986097 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982630 4442 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:58:22.986097 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982637 4442 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:58:22.986097 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982643 4442 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:58:22.986097 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982649 4442 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:58:22.986097 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982654 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:58:22.986097 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982659 4442 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:58:22.986097 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982664 4442 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:58:22.986097 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982668 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:58:22.986595 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982673 4442 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:58:22.986595 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982677 4442 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:58:22.986595 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982682 4442 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:58:22.986595 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982686 4442 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:58:22.986595 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982690 4442 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:58:22.986595 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982695 4442 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:58:22.986595 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982699 4442 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:58:22.986595 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982704 4442 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:58:22.986595 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982708 4442 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:58:22.986595 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982712 4442 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:58:22.986595 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982718 4442 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:58:22.986595 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982721 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:58:22.986595 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982725 4442 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:58:22.986595 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982727 4442 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:58:22.986595 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982730 4442 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:58:22.986595 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982733 4442 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:58:22.986595 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982738 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:58:22.986595 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982741 4442 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:58:22.986595 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982744 4442 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:58:22.987053 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982747 4442 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:58:22.987053 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982750 4442 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:58:22.987053 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982753 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:58:22.987053 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982756 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:58:22.987053 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982759 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:58:22.987053 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982761 4442 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:58:22.987053 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982764 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:58:22.987053 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982767 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:58:22.987053 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982770 4442 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:58:22.987053 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982773 4442 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:58:22.987053 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982775 4442 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:58:22.987053 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982778 4442 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:58:22.987053 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982781 4442 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:58:22.987053 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982784 4442 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:58:22.987053 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982787 4442 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:58:22.987053 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982789 4442 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:58:22.987053 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982792 4442 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:58:22.987053 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982795 4442 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:58:22.987053 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982797 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:58:22.987053 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982800 4442 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:58:22.987576 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982803 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:58:22.987576 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982805 4442 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:58:22.987576 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982808 4442 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:58:22.987576 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982810 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:58:22.987576 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982813 4442 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:58:22.987576 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982816 4442 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:58:22.987576 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982819 4442 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:58:22.987576 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982822 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:58:22.987576 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982825 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:58:22.987576 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982830 4442 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:58:22.987576 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982832 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:58:22.987576 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982835 4442 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:58:22.987576 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982837 4442 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:58:22.987576 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982840 4442 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:58:22.987576 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982843 4442 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:58:22.987576 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982846 4442 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:58:22.987576 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982849 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:58:22.987576 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982851 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:58:22.987576 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982854 4442 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:58:22.988081 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982857 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:58:22.988081 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982861 4442 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:58:22.988081 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982865 4442 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:58:22.988081 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982868 4442 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:58:22.988081 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982871 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:58:22.988081 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982874 4442 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:58:22.988081 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982877 4442 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:58:22.988081 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982879 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:58:22.988081 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982882 4442 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:58:22.988081 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982885 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:58:22.988081 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982887 4442 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:58:22.988081 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982890 4442 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:58:22.988081 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982892 4442 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:58:22.988081 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982895 4442 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:58:22.988081 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982897 4442 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:58:22.988081 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982900 4442 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:58:22.988081 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982903 4442 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:58:22.988081 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982905 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:58:22.988081 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982908 4442 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:58:22.988577 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982910 4442 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:58:22.988577 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982913 4442 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:58:22.988577 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.982916 4442 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:58:22.988577 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983029 4442 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 17:58:22.988577 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983038 4442 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 17:58:22.988577 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983044 4442 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 17:58:22.988577 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983049 4442 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 17:58:22.988577 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983054 4442 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 17:58:22.988577 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983058 4442 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 17:58:22.988577 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983062 4442 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 17:58:22.988577 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983068 4442 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 17:58:22.988577 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983071 4442 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 17:58:22.988577 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983074 4442 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 17:58:22.988577 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983078 4442 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 17:58:22.988577 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983082 4442 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 17:58:22.988577 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983085 4442 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 17:58:22.988577 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983088 4442 flags.go:64] FLAG: --cgroup-root="" Oct 30 17:58:22.988577 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983091 4442 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 17:58:22.988577 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983094 4442 flags.go:64] FLAG: --client-ca-file="" Oct 30 17:58:22.988577 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983097 4442 flags.go:64] FLAG: --cloud-provider="external" Oct 30 17:58:22.988577 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983101 4442 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 17:58:22.988577 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983106 4442 flags.go:64] FLAG: --cluster-domain="" Oct 30 17:58:22.988577 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983109 4442 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 17:58:22.989147 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983112 4442 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 17:58:22.989147 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983115 4442 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 17:58:22.989147 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983119 4442 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 17:58:22.989147 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983123 4442 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 17:58:22.989147 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983126 4442 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 17:58:22.989147 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983149 4442 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 17:58:22.989147 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983155 4442 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 17:58:22.989147 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983160 4442 flags.go:64] FLAG: --contention-profiling="false" Oct 30 17:58:22.989147 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983166 4442 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 17:58:22.989147 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983169 4442 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 17:58:22.989147 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983175 4442 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 17:58:22.989147 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983179 4442 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 17:58:22.989147 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983186 4442 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 17:58:22.989147 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983190 4442 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 17:58:22.989147 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983193 4442 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 17:58:22.989147 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983196 4442 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 17:58:22.989147 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983199 4442 flags.go:64] FLAG: --enable-server="true" Oct 30 17:58:22.989147 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983202 4442 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 17:58:22.989147 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983206 4442 flags.go:64] FLAG: --event-burst="100" Oct 30 17:58:22.989147 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983209 4442 flags.go:64] FLAG: --event-qps="50" Oct 30 17:58:22.989147 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983212 4442 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 17:58:22.989147 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983216 4442 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 17:58:22.989147 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983220 4442 flags.go:64] FLAG: --eviction-hard="" Oct 30 17:58:22.989147 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983225 4442 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 17:58:22.989864 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983228 4442 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 17:58:22.989864 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983231 4442 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 17:58:22.989864 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983234 4442 flags.go:64] FLAG: --eviction-soft="" Oct 30 17:58:22.989864 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983238 4442 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 17:58:22.989864 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983241 4442 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 17:58:22.989864 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983244 4442 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 17:58:22.989864 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983247 4442 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 17:58:22.989864 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983250 4442 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 17:58:22.989864 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983253 4442 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 17:58:22.989864 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983256 4442 flags.go:64] FLAG: --feature-gates="" Oct 30 17:58:22.989864 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983261 4442 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 17:58:22.989864 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983265 4442 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 17:58:22.989864 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983268 4442 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 17:58:22.989864 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983271 4442 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 17:58:22.989864 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983274 4442 flags.go:64] FLAG: --healthz-port="10248" Oct 30 17:58:22.989864 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983277 4442 flags.go:64] FLAG: --help="false" Oct 30 17:58:22.989864 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983280 4442 flags.go:64] FLAG: --hostname-override="ip-10-0-138-176.ec2.internal" Oct 30 17:58:22.989864 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983284 4442 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 17:58:22.989864 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983288 4442 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 17:58:22.989864 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983291 4442 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 17:58:22.989864 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983296 4442 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 17:58:22.989864 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983300 4442 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 17:58:22.989864 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983303 4442 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 17:58:22.989864 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983306 4442 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 17:58:22.990489 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983309 4442 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 17:58:22.990489 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983312 4442 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 17:58:22.990489 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983315 4442 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 17:58:22.990489 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983318 4442 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 17:58:22.990489 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983321 4442 flags.go:64] FLAG: --kube-reserved="" Oct 30 17:58:22.990489 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983324 4442 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 17:58:22.990489 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983327 4442 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 17:58:22.990489 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983330 4442 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 17:58:22.990489 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983333 4442 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 17:58:22.990489 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983336 4442 flags.go:64] FLAG: --lock-file="" Oct 30 17:58:22.990489 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983339 4442 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 17:58:22.990489 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983342 4442 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 17:58:22.990489 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983345 4442 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 17:58:22.990489 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983351 4442 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 17:58:22.990489 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983354 4442 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 17:58:22.990489 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983358 4442 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 17:58:22.990489 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983361 4442 flags.go:64] FLAG: --logging-format="text" Oct 30 17:58:22.990489 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983364 4442 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 17:58:22.990489 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983367 4442 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 17:58:22.990489 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983370 4442 flags.go:64] FLAG: --manifest-url="" Oct 30 17:58:22.990489 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983373 4442 flags.go:64] FLAG: --manifest-url-header="" Oct 30 17:58:22.990489 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983378 4442 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 17:58:22.990489 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983382 4442 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 17:58:22.990489 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983386 4442 flags.go:64] FLAG: --max-pods="110" Oct 30 17:58:22.990489 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983389 4442 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 17:58:22.991337 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983392 4442 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 17:58:22.991337 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983397 4442 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 17:58:22.991337 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983400 4442 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 17:58:22.991337 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983403 4442 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 17:58:22.991337 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983406 4442 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 17:58:22.991337 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983410 4442 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 17:58:22.991337 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983418 4442 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 17:58:22.991337 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983422 4442 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 17:58:22.991337 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983425 4442 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 17:58:22.991337 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983428 4442 flags.go:64] FLAG: --pod-cidr="" Oct 30 17:58:22.991337 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983431 4442 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 17:58:22.991337 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983434 4442 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 17:58:22.991337 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983437 4442 flags.go:64] FLAG: --pods-per-core="0" Oct 30 17:58:22.991337 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983440 4442 flags.go:64] FLAG: --port="10250" Oct 30 17:58:22.991337 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983444 4442 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 17:58:22.991337 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983446 4442 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c6984eabe761abec" Oct 30 17:58:22.991337 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983450 4442 flags.go:64] FLAG: --qos-reserved="" Oct 30 17:58:22.991337 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983453 4442 flags.go:64] FLAG: --read-only-port="10255" Oct 30 17:58:22.991337 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983456 4442 flags.go:64] FLAG: --register-node="true" Oct 30 17:58:22.991337 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983459 4442 flags.go:64] FLAG: --register-with-taints="" Oct 30 17:58:22.991337 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983463 4442 flags.go:64] FLAG: --registry-burst="10" Oct 30 17:58:22.991337 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983466 4442 flags.go:64] FLAG: --registry-qps="5" Oct 30 17:58:22.991337 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983469 4442 flags.go:64] FLAG: --reserved-cpus="" Oct 30 17:58:22.991337 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983472 4442 flags.go:64] FLAG: --reserved-memory="" Oct 30 17:58:22.991337 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983479 4442 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 17:58:22.991337 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983483 4442 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 17:58:22.992010 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983486 4442 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 17:58:22.992010 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983489 4442 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 17:58:22.992010 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983492 4442 flags.go:64] FLAG: --runonce="false" Oct 30 17:58:22.992010 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983495 4442 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 17:58:22.992010 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983498 4442 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 17:58:22.992010 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983502 4442 flags.go:64] FLAG: --seccomp-default="false" Oct 30 17:58:22.992010 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983505 4442 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 17:58:22.992010 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983510 4442 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 17:58:22.992010 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983513 4442 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 17:58:22.992010 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983517 4442 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 17:58:22.992010 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983520 4442 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 17:58:22.992010 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983523 4442 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 17:58:22.992010 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983526 4442 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 17:58:22.992010 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983530 4442 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 17:58:22.992010 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983533 4442 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 17:58:22.992010 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983536 4442 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 17:58:22.992010 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983539 4442 flags.go:64] FLAG: --system-cgroups="" Oct 30 17:58:22.992010 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983542 4442 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 17:58:22.992010 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983547 4442 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 17:58:22.992010 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983550 4442 flags.go:64] FLAG: --tls-cert-file="" Oct 30 17:58:22.992010 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983553 4442 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 17:58:22.992010 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983557 4442 flags.go:64] FLAG: --tls-min-version="" Oct 30 17:58:22.992010 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983560 4442 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 17:58:22.992010 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983563 4442 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 17:58:22.992010 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983566 4442 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 17:58:22.992741 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983569 4442 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 17:58:22.992741 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983572 4442 flags.go:64] FLAG: --v="2" Oct 30 17:58:22.992741 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983577 4442 flags.go:64] FLAG: --version="false" Oct 30 17:58:22.992741 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983582 4442 flags.go:64] FLAG: --vmodule="" Oct 30 17:58:22.992741 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983586 4442 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 17:58:22.992741 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.983590 4442 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 17:58:22.992741 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984285 4442 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:58:22.992741 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984302 4442 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:58:22.992741 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984308 4442 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:58:22.992741 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984314 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:58:22.992741 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984319 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:58:22.992741 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984325 4442 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:58:22.992741 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984330 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:58:22.992741 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984336 4442 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:58:22.992741 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984344 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:58:22.992741 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984349 4442 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:58:22.992741 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984354 4442 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:58:22.992741 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984359 4442 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:58:22.992741 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984369 4442 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:58:22.992741 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984375 4442 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:58:22.992741 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984380 4442 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:58:22.992741 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984385 4442 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:58:22.993319 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984390 4442 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:58:22.993319 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984395 4442 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:58:22.993319 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984399 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:58:22.993319 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984404 4442 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:58:22.993319 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984408 4442 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:58:22.993319 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984413 4442 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:58:22.993319 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984418 4442 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:58:22.993319 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984422 4442 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:58:22.993319 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984435 4442 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:58:22.993319 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984440 4442 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:58:22.993319 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984445 4442 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:58:22.993319 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984450 4442 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:58:22.993319 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984455 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:58:22.993319 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984460 4442 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:58:22.993319 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984465 4442 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:58:22.993319 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984470 4442 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:58:22.993319 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984475 4442 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:58:22.993319 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984479 4442 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:58:22.993319 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984484 4442 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:58:22.993787 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984488 4442 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:58:22.993787 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984535 4442 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:58:22.993787 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984545 4442 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:58:22.993787 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984550 4442 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:58:22.993787 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984555 4442 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:58:22.993787 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984562 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:58:22.993787 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984567 4442 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:58:22.993787 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984572 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:58:22.993787 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984576 4442 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:58:22.993787 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984581 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:58:22.993787 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984585 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:58:22.993787 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984591 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:58:22.993787 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984596 4442 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:58:22.993787 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984601 4442 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:58:22.993787 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984605 4442 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:58:22.993787 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984618 4442 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:58:22.993787 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984625 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:58:22.993787 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984630 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:58:22.993787 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984635 4442 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:58:22.994312 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984639 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:58:22.994312 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984644 4442 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:58:22.994312 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984651 4442 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:58:22.994312 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984655 4442 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:58:22.994312 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984660 4442 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:58:22.994312 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984664 4442 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:58:22.994312 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984669 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:58:22.994312 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984673 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:58:22.994312 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984683 4442 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:58:22.994312 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984687 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:58:22.994312 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984692 4442 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:58:22.994312 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984696 4442 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:58:22.994312 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984701 4442 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:58:22.994312 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984705 4442 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:58:22.994312 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984710 4442 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:58:22.994312 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984716 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:58:22.994312 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984720 4442 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:58:22.994312 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984725 4442 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:58:22.994312 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984732 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:58:22.994312 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984737 4442 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:58:22.994837 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984742 4442 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:58:22.994837 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984849 4442 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:58:22.994837 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.984995 4442 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:58:22.994837 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985000 4442 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:58:22.994837 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985003 4442 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:58:22.994837 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985006 4442 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:58:22.994837 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985021 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:58:22.994837 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985025 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:58:22.994837 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985028 4442 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:58:22.994837 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985031 4442 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:58:22.994837 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985034 4442 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:58:22.994837 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985037 4442 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:58:22.994837 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985039 4442 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:58:22.994837 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985042 4442 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:58:22.994837 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985045 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:58:22.994837 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985048 4442 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:58:22.994837 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985051 4442 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:58:22.994837 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985054 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:58:22.994837 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985057 4442 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:58:22.994837 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985059 4442 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:58:22.995445 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985062 4442 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:58:22.995445 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985065 4442 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:58:22.995445 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985070 4442 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:58:22.995445 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985075 4442 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:58:22.995445 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985079 4442 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:58:22.995445 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985083 4442 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:58:22.995445 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985086 4442 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:58:22.995445 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985090 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:58:22.995445 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985093 4442 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:58:22.995445 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985096 4442 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:58:22.995445 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985099 4442 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:58:22.995445 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985101 4442 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:58:22.995445 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.985170 4442 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:58:22.995445 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985219 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:58:22.995445 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985226 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:58:22.995873 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985229 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:58:22.995873 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985232 4442 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:58:22.995873 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985235 4442 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:58:22.995873 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985238 4442 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:58:22.995873 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985243 4442 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:58:22.995873 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985247 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:58:22.995873 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985250 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:58:22.995873 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985253 4442 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:58:22.995873 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985256 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:58:22.995873 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985259 4442 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:58:22.995873 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985261 4442 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:58:22.995873 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985264 4442 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:58:22.995873 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985267 4442 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:58:22.995873 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985270 4442 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:58:22.995873 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985272 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:58:22.995873 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985275 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:58:22.995873 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985279 4442 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:58:22.995873 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985282 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:58:22.995873 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985284 4442 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:58:22.996399 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985287 4442 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:58:22.996399 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985290 4442 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:58:22.996399 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985293 4442 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:58:22.996399 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985296 4442 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:58:22.996399 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985299 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:58:22.996399 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985303 4442 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:58:22.996399 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985306 4442 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:58:22.996399 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985309 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:58:22.996399 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985311 4442 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:58:22.996399 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985314 4442 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:58:22.996399 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985316 4442 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:58:22.996399 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985319 4442 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:58:22.996399 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985322 4442 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:58:22.996399 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985324 4442 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:58:22.996399 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985327 4442 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:58:22.996399 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985330 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:58:22.996399 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985332 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:58:22.996399 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985335 4442 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:58:22.996399 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985337 4442 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:58:22.996399 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985340 4442 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:58:22.996877 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985342 4442 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:58:22.996877 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985345 4442 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:58:22.996877 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985347 4442 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:58:22.996877 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985350 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:58:22.996877 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985352 4442 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:58:22.996877 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985355 4442 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:58:22.996877 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985358 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:58:22.996877 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985360 4442 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:58:22.996877 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985363 4442 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:58:22.996877 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985366 4442 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:58:22.996877 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985368 4442 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:58:22.996877 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985371 4442 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:58:22.996877 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985375 4442 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:58:22.996877 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985379 4442 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:58:22.996877 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985382 4442 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:58:22.996877 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985386 4442 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:58:22.996877 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985389 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:58:22.996877 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985392 4442 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:58:22.996877 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985395 4442 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:58:22.997619 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985397 4442 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:58:22.997619 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985400 4442 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:58:22.997619 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985402 4442 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:58:22.997619 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985405 4442 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:58:22.997619 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985407 4442 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:58:22.997619 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985410 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:58:22.997619 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985412 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:58:22.997619 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985416 4442 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:58:22.997619 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985419 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:58:22.997619 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985422 4442 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:58:22.997619 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985424 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:58:22.997619 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985427 4442 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:58:22.997619 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985430 4442 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:58:22.997619 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985432 4442 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:58:22.997619 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985435 4442 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:58:22.997619 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985437 4442 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:58:22.997619 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985440 4442 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:58:22.997619 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985443 4442 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:58:22.997619 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985445 4442 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:58:22.997619 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985448 4442 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:58:22.998146 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985450 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:58:22.998146 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985456 4442 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:58:22.998146 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985458 4442 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:58:22.998146 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985461 4442 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:58:22.998146 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985464 4442 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:58:22.998146 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985466 4442 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:58:22.998146 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985469 4442 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:58:22.998146 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985471 4442 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:58:22.998146 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985475 4442 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:58:22.998146 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985479 4442 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:58:22.998146 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985482 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:58:22.998146 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985485 4442 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:58:22.998146 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985488 4442 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:58:22.998146 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985490 4442 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:58:22.998146 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985493 4442 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:58:22.998146 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985496 4442 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:58:22.998146 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985498 4442 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:58:22.998146 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985501 4442 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:58:22.998146 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985504 4442 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:58:22.998146 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985506 4442 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:58:22.998696 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985509 4442 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:58:22.998696 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985512 4442 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:58:22.998696 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985514 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:58:22.998696 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985517 4442 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:58:22.998696 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985522 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:58:22.998696 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.985524 4442 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:58:22.998696 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.989644 4442 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 17:58:22.998696 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.989664 4442 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 17:58:22.998696 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.989685 4442 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 17:58:22.998696 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.989689 4442 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 17:58:22.998696 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989757 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:58:22.998696 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989762 4442 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:58:22.998696 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989766 4442 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:58:22.998696 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989769 4442 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:58:22.998696 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989771 4442 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:58:22.998696 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989774 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:58:22.998696 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989777 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:58:22.998696 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989779 4442 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:58:22.998696 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989782 4442 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:58:22.999183 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989785 4442 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:58:22.999183 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989787 4442 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:58:22.999183 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989790 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:58:22.999183 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989792 4442 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:58:22.999183 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989795 4442 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:58:22.999183 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989798 4442 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:58:22.999183 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989800 4442 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:58:22.999183 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989803 4442 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:58:22.999183 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989805 4442 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:58:22.999183 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989808 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:58:22.999183 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989812 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:58:22.999183 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989814 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:58:22.999183 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989817 4442 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:58:22.999183 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989820 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:58:22.999183 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989822 4442 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:58:22.999183 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989825 4442 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:58:22.999183 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989827 4442 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:58:22.999183 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989831 4442 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:58:22.999183 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989835 4442 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:58:22.999655 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989839 4442 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:58:22.999655 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989842 4442 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:58:22.999655 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989846 4442 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:58:22.999655 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989848 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:58:22.999655 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989852 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:58:22.999655 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989854 4442 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:58:22.999655 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989857 4442 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:58:22.999655 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989860 4442 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:58:22.999655 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989862 4442 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:58:22.999655 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989865 4442 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:58:22.999655 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989868 4442 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:58:22.999655 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989870 4442 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:58:22.999655 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989874 4442 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:58:22.999655 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989877 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:58:22.999655 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989879 4442 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:58:22.999655 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989882 4442 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:58:22.999655 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989885 4442 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:58:22.999655 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989887 4442 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:58:22.999655 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989890 4442 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:58:22.999655 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989892 4442 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:58:23.000225 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989895 4442 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:58:23.000225 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989898 4442 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:58:23.000225 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989902 4442 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:58:23.000225 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989904 4442 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:58:23.000225 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989907 4442 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:58:23.000225 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989910 4442 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:58:23.000225 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989913 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:58:23.000225 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989915 4442 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:58:23.000225 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989918 4442 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:58:23.000225 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989920 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:58:23.000225 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989923 4442 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:58:23.000225 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989926 4442 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:58:23.000225 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989928 4442 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:58:23.000225 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989931 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:58:23.000225 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989934 4442 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:58:23.000225 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989938 4442 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:58:23.000225 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989941 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:58:23.000225 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989944 4442 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:58:23.000225 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989946 4442 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:58:23.000225 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989949 4442 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:58:23.000968 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989951 4442 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:58:23.000968 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989954 4442 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:58:23.000968 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989956 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:58:23.000968 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989959 4442 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:58:23.000968 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989962 4442 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:58:23.000968 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989964 4442 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:58:23.000968 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989967 4442 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:58:23.000968 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989969 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:58:23.000968 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989972 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:58:23.000968 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989974 4442 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:58:23.000968 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989977 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:58:23.000968 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989979 4442 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:58:23.000968 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989982 4442 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:58:23.000968 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989984 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:58:23.000968 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989987 4442 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:58:23.000968 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989990 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:58:23.000968 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989992 4442 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:58:23.000968 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989995 4442 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:58:23.000968 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.989998 4442 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:58:23.000968 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990000 4442 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:58:23.001722 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990003 4442 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:58:23.001722 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990005 4442 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:58:23.001722 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990007 4442 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:58:23.001722 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990010 4442 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:58:23.001722 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990013 4442 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:58:23.001722 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990016 4442 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:58:23.001722 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990018 4442 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:58:23.001722 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990021 4442 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:58:23.001722 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990024 4442 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:58:23.001722 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990027 4442 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:58:23.001722 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990030 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:58:23.001722 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990032 4442 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:58:23.001722 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990036 4442 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:58:23.001722 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990041 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:58:23.001722 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990044 4442 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:58:23.001722 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990048 4442 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:58:23.001722 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990051 4442 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:58:23.001722 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990053 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:58:23.002193 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.990086 4442 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:58:23.002193 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990255 4442 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:58:23.002193 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990261 4442 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:58:23.002193 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990265 4442 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:58:23.002193 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990267 4442 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:58:23.002193 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990271 4442 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:58:23.002193 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990273 4442 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:58:23.002193 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990276 4442 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:58:23.002193 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990279 4442 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:58:23.002193 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990282 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:58:23.002193 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990285 4442 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:58:23.002193 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990287 4442 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:58:23.002193 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990290 4442 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:58:23.002193 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990292 4442 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:58:23.002193 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990295 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:58:23.002193 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990297 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:58:23.002573 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990300 4442 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:58:23.002573 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990302 4442 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:58:23.002573 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990306 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:58:23.002573 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990308 4442 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:58:23.002573 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990311 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:58:23.002573 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990314 4442 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:58:23.002573 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990316 4442 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:58:23.002573 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990320 4442 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:58:23.002573 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990323 4442 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:58:23.002573 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990326 4442 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:58:23.002573 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990328 4442 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:58:23.002573 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990331 4442 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:58:23.002573 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990333 4442 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:58:23.002573 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990336 4442 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:58:23.002573 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990338 4442 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:58:23.002573 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990342 4442 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:58:23.002573 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990346 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:58:23.002573 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990349 4442 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:58:23.002573 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990352 4442 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:58:23.003038 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990355 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:58:23.003038 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990358 4442 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:58:23.003038 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990361 4442 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:58:23.003038 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990363 4442 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:58:23.003038 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990366 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:58:23.003038 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990369 4442 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:58:23.003038 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990372 4442 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:58:23.003038 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990375 4442 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:58:23.003038 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990377 4442 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:58:23.003038 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990380 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:58:23.003038 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990382 4442 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:58:23.003038 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990385 4442 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:58:23.003038 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990387 4442 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:58:23.003038 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990390 4442 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:58:23.003038 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990392 4442 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:58:23.003038 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990395 4442 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:58:23.003038 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990398 4442 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:58:23.003038 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990401 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:58:23.003038 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990403 4442 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:58:23.003677 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990406 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:58:23.003677 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990408 4442 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:58:23.003677 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990412 4442 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:58:23.003677 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990415 4442 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:58:23.003677 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990419 4442 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:58:23.003677 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990422 4442 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:58:23.003677 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990425 4442 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:58:23.003677 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990428 4442 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:58:23.003677 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990430 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:58:23.003677 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990433 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:58:23.003677 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990435 4442 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:58:23.003677 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990438 4442 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:58:23.003677 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990440 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:58:23.003677 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990443 4442 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:58:23.003677 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990445 4442 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:58:23.003677 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990447 4442 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:58:23.003677 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990450 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:58:23.003677 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990453 4442 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:58:23.003677 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990455 4442 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:58:23.003677 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990458 4442 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:58:23.004298 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990461 4442 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:58:23.004298 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990463 4442 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:58:23.004298 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990466 4442 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:58:23.004298 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990468 4442 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:58:23.004298 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990471 4442 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:58:23.004298 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990473 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:58:23.004298 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990476 4442 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:58:23.004298 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990478 4442 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:58:23.004298 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990481 4442 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:58:23.004298 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990484 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:58:23.004298 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990486 4442 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:58:23.004298 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990489 4442 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:58:23.004298 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990491 4442 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:58:23.004298 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990494 4442 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:58:23.004298 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990497 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:58:23.004298 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990501 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:58:23.004298 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990503 4442 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:58:23.004298 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990506 4442 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:58:23.004298 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990509 4442 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:58:23.004298 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990511 4442 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:58:23.005014 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990514 4442 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:58:23.005014 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990516 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:58:23.005014 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990519 4442 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:58:23.005014 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990521 4442 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:58:23.005014 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990524 4442 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:58:23.005014 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990526 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:58:23.005014 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990529 4442 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:58:23.005014 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990531 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:58:23.005014 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990534 4442 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:58:23.005014 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990536 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:58:23.005014 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990539 4442 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:58:23.005014 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990541 4442 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:58:23.005014 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990544 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:58:23.005014 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.990568 4442 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:58:23.005014 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990600 4442 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:58:23.005562 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990605 4442 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:58:23.005562 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990609 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:58:23.005562 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990612 4442 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:58:23.005562 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990615 4442 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:58:23.005562 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990618 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:58:23.005562 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990623 4442 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:58:23.005562 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990626 4442 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:58:23.005562 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990629 4442 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:58:23.005562 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990631 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:58:23.005562 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990634 4442 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:58:23.005562 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990636 4442 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:58:23.005562 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990639 4442 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:58:23.005562 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990642 4442 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:58:23.005562 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990649 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:58:23.005562 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990652 4442 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:58:23.005562 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990655 4442 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:58:23.005562 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990657 4442 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:58:23.005562 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990660 4442 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:58:23.005562 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990662 4442 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:58:23.006407 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990665 4442 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:58:23.006407 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990667 4442 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:58:23.006407 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990670 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:58:23.006407 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990672 4442 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:58:23.006407 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990675 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:58:23.006407 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990677 4442 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:58:23.006407 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990679 4442 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:58:23.006407 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990682 4442 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:58:23.006407 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990686 4442 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:58:23.006407 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990689 4442 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:58:23.006407 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990692 4442 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:58:23.006407 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990695 4442 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:58:23.006407 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990698 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:58:23.006407 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990701 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:58:23.006407 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990703 4442 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:58:23.006407 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990706 4442 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:58:23.006407 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990708 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:58:23.006407 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990711 4442 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:58:23.006407 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990718 4442 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:58:23.006407 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990720 4442 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:58:23.006900 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990723 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:58:23.006900 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990725 4442 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:58:23.006900 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990728 4442 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:58:23.006900 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990730 4442 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:58:23.006900 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990733 4442 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:58:23.006900 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990735 4442 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:58:23.006900 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990738 4442 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:58:23.006900 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990741 4442 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:58:23.006900 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990744 4442 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:58:23.006900 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990746 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:58:23.006900 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990749 4442 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:58:23.006900 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990751 4442 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:58:23.006900 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990754 4442 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:58:23.006900 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990756 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:58:23.006900 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990758 4442 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:58:23.006900 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990761 4442 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:58:23.006900 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990763 4442 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:58:23.006900 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990766 4442 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:58:23.006900 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990768 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:58:23.006900 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990771 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:58:23.007433 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990773 4442 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:58:23.007433 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990776 4442 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:58:23.007433 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990778 4442 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:58:23.007433 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990781 4442 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:58:23.007433 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990787 4442 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:58:23.007433 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990789 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:58:23.007433 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990792 4442 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:58:23.007433 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990795 4442 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:58:23.007433 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990798 4442 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:58:23.007433 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990800 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:58:23.007433 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990805 4442 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:58:23.007433 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990807 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:58:23.007433 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990810 4442 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:58:23.007433 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990813 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:58:23.007433 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990815 4442 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:58:23.007433 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990818 4442 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:58:23.007433 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990820 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:58:23.007433 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990823 4442 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:58:23.007433 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990826 4442 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:58:23.007433 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990829 4442 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:58:23.007967 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990835 4442 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:58:23.007967 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990838 4442 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:58:23.007967 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990840 4442 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:58:23.007967 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990843 4442 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:58:23.007967 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990846 4442 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:58:23.007967 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990848 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:58:23.007967 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990851 4442 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:58:23.007967 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990853 4442 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:58:23.007967 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990856 4442 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:58:23.007967 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990858 4442 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:58:23.007967 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990861 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:58:23.007967 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990864 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:58:23.007967 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990867 4442 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:58:23.007967 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990869 4442 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:58:23.007967 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990872 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:58:23.007967 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990875 4442 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:58:23.007967 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990879 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:58:23.007967 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990881 4442 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:58:23.007967 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990884 4442 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:58:23.007967 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990886 4442 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:58:23.008517 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990889 4442 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:58:23.008517 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990892 4442 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:58:23.008517 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990896 4442 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:58:23.008517 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990899 4442 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:58:23.008517 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990901 4442 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:58:23.008517 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:22.990904 4442 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:58:23.008517 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.990962 4442 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 17:58:23.008517 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.992042 4442 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 17:58:23.008517 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.992112 4442 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 17:58:23.008517 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.992338 4442 server.go:1014] "Starting client certificate rotation" Oct 30 17:58:23.008517 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.992480 4442 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 17:58:23.008517 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.992538 4442 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 17:52:43 +0000 UTC" deadline="2028-04-08 22:48:17.298793798 +0000 UTC" Oct 30 17:58:23.008517 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.992579 4442 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12628h49m54.306217441s" Oct 30 17:58:23.008517 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.992806 4442 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 17:58:23.008517 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.993917 4442 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 17:58:23.008517 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.995404 4442 log.go:25] "Validated CRI v1 runtime API" Oct 30 17:58:23.008928 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.997123 4442 log.go:25] "Validated CRI v1 image API" Oct 30 17:58:23.008928 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.997969 4442 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 17:58:23.008928 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.999979 4442 fs.go:135] Filesystem UUIDs: map[6ed5bd6d-db36-4bb3-a60a-f4a4624e9220:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8d22543c-06b7-4209-b7be-433a475b3533:/dev/nvme0n1p4] Oct 30 17:58:23.009045 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:22.999999 4442 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/368fe8a43f1f9d87c0fd555c1f310648af3f9c8792bd44010dde89dd3412a7de/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/368fe8a43f1f9d87c0fd555c1f310648af3f9c8792bd44010dde89dd3412a7de/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3bc6aeac772dd708ce9046b0304b23ae422455f1543d097553d14f687d51d86f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3bc6aeac772dd708ce9046b0304b23ae422455f1543d097553d14f687d51d86f/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/507d24b53aa3c10ca3e0817caae8defea37f8597936fa8f0a8633359eaa7a42d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/507d24b53aa3c10ca3e0817caae8defea37f8597936fa8f0a8633359eaa7a42d/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/561db2dcb46df96c6309155b3811410b63e8b552e88a200ed32e8949742b7aa8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/561db2dcb46df96c6309155b3811410b63e8b552e88a200ed32e8949742b7aa8/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5b19ec34f9443e71dd23a1c4adebb7a8fdcfc9a9c2790cb54ca232ad88ccfebd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5b19ec34f9443e71dd23a1c4adebb7a8fdcfc9a9c2790cb54ca232ad88ccfebd/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6639644759f355b3688140ef37b276a99cc9aaae3564b1358f929987a4e0af0e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6639644759f355b3688140ef37b276a99cc9aaae3564b1358f929987a4e0af0e/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7541c221aa4e62cb01d078764e9f799e80eb222aa78fabb01004b143f09a8534/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7541c221aa4e62cb01d078764e9f799e80eb222aa78fabb01004b143f09a8534/userdata/shm major:0 minor:222 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/90c5a4d78f5e69d3d3d7d2e7a3303f8ed64b8a08cdcc180430b36f637af24dee/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/90c5a4d78f5e69d3d3d7d2e7a3303f8ed64b8a08cdcc180430b36f637af24dee/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/91820cfa6dc81f9c6c366883a9c69fbc51bd5b4d7cf837fb6a1d97a111addfdf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/91820cfa6dc81f9c6c366883a9c69fbc51bd5b4d7cf837fb6a1d97a111addfdf/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/af185b4ea7b5d4e8cc75783cd205502cd36711430d5772eced1ea92c6bec8bf1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/af185b4ea7b5d4e8cc75783cd205502cd36711430d5772eced1ea92c6bec8bf1/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c41265ebc3199e6664c64099e6b1826ef30d8ab567d887f01bc7fd8e37cfde4d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c41265ebc3199e6664c64099e6b1826ef30d8ab567d887f01bc7fd8e37cfde4d/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cdd236e6bad8950bc60a0cf0d2be119eff0267273b1421a7a22fed2b7c70001e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cdd236e6bad8950bc60a0cf0d2be119eff0267273b1421a7a22fed2b7c70001e/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eeeb661dbbd8a1ec768602d833f2feab2fa496aa751dbe4ae51816fb5ffc8de4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eeeb661dbbd8a1ec768602d833f2feab2fa496aa751dbe4ae51816fb5ffc8de4/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f894d624a353c3f8e2c9cdb15ecfa62e4288061b1f6ce1662e6245642069e557/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f894d624a353c3f8e2c9cdb15ecfa62e4288061b1f6ce1662e6245642069e557/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0772096e-be2c-449d-9bf6-c56cbd6c203a/volumes/kubernetes.io~projected/kube-api-access-cr4zd:{mountpoint:/var/lib/kubelet/pods/0772096e-be2c-449d-9bf6-c56cbd6c203a/volumes/kubernetes.io~projected/kube-api-access-cr4zd major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0af98be4-3c2f-413e-838e-f842854411f2/volumes/kubernetes.io~projected/kube-api-access-zvntm:{mountpoint:/var/lib/kubelet/pods/0af98be4-3c2f-413e-838e-f842854411f2/volumes/kubernetes.io~projected/kube-api-access-zvntm major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/245afcc3-9c86-4c4e-ac8f-9f7d427c5920/volumes/kubernetes.io~projected/kube-api-access-99kgv:{mountpoint:/var/lib/kubelet/pods/245afcc3-9c86-4c4e-ac8f-9f7d427c5920/volumes/kubernetes.io~projected/kube-api-access-99kgv major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44e585a6-cc1a-4ebc-a592-907ccee3729f/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/44e585a6-cc1a-4ebc-a592-907ccee3729f/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44e585a6-cc1a-4ebc-a592-907ccee3729f/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/44e585a6-cc1a-4ebc-a592-907ccee3729f/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44e585a6-cc1a-4ebc-a592-907ccee3729f/volumes/kubernetes.io~projected/kube-api-access-9n82w:{mountpoint:/var/lib/kubelet/pods/44e585a6-cc1a-4ebc-a592-907ccee3729f/volumes/kubernetes.io~projected/kube-api-access-9n82w major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/516c6b6c-fc40-49cf-93b5-1c4471b02cf0/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/516c6b6c-fc40-49cf-93b5-1c4471b02cf0/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/516c6b6c-fc40-49cf-93b5-1c4471b02cf0/volumes/kubernetes.io~projected/kube-api-access-65887:{mountpoint:/var/lib/kubelet/pods/516c6b6c-fc40-49cf-93b5-1c4471b02cf0/volumes/kubernetes.io~projected/kube-api-access-65887 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/516c6b6c-fc40-49cf-93b5-1c4471b02cf0/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/516c6b6c-fc40-49cf-93b5-1c4471b02cf0/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/568da9c4-2c4f-4020-95ca-9f7ac7db7c00/volumes/kubernetes.io~projected/kube-api-access-7h9kb:{mountpoint:/var/lib/kubelet/pods/568da9c4-2c4f-4020-95ca-9f7ac7db7c00/volumes/kubernetes.io~projected/kube-api-access-7h9kb major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59229234-921c-4177-8b85-3e982194f625/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/59229234-921c-4177-8b85-3e982194f625/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8215ee49-dab0-40f1-af73-3d4493e3bc04/volumes/kubernetes.io~projected/kube-api-access-gqmfq:{mountpoint:/var/lib/kubelet/pods/8215ee49-dab0-40f1-af73-3d4493e3bc04/volumes/kubernetes.io~projected/kube-api-access-gqmfq major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ba0b8dc-0459-440d-91dc-ad18f4b279fa/volumes/kubernetes.io~projected/kube-api-access-wt5ws:{mountpoint:/var/lib/kubelet/pods/8ba0b8dc-0459-440d-91dc-ad18f4b279fa/volumes/kubernetes.io~projected/kube-api-access-wt5ws major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a24e24cc-a321-4a62-b023-7b015f155d55/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/a24e24cc-a321-4a62-b023-7b015f155d55/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a28935a6-1847-4f43-9e7a-332c6b3ce49f/volumes/kubernetes.io~projected/kube-api-access-m5sr2:{mountpoint:/var/lib/kubelet/pods/a28935a6-1847-4f43-9e7a-332c6b3ce49f/volumes/kubernetes.io~projected/kube-api-access-m5sr2 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6842e7e-ad53-424d-be89-f6b29f22ea24/volumes/kubernetes.io~projected/kube-api-access-j2wnc:{mountpoint:/var/lib/kubelet/pods/a6842e7e-ad53-424d-be89-f6b29f22ea24/volumes/kubernetes.io~projected/kube-api-access-j2wnc major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf5ca486-37a8-4da0-acbf-d389caf9f3e6/volumes/kubernetes.io~projected/kube-api-access-84lr2:{mountpoint:/var/lib/kubelet/pods/bf5ca486-37a8-4da0-acbf-d389caf9f3e6/volumes/kubernetes.io~projected/kube-api-access-84lr2 major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d37ce412-3bd6-4093-9d21-3b72d396c1bd/volumes/kubernetes.io~projected/kube-api-access-nh2xp:{mountpoint:/var/lib/kubelet/pods/d37ce412-3bd6-4093-9d21-3b72d396c1bd/volumes/kubernetes.io~projected/kube-api-access-nh2xp major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6164286-3eb9-4e3c-82bc-4886de22b3fb/volumes/kubernetes.io~projected/kube-api-access-ckxh9:{mountpoint:/var/lib/kubelet/pods/d6164286-3eb9-4e3c-82bc-4886de22b3fb/volumes/kubernetes.io~projected/kube-api-access-ckxh9 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6164286-3eb9-4e3c-82bc-4886de22b3fb/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/d6164286-3eb9-4e3c-82bc-4886de22b3fb/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0a64047-334d-4df8-83de-3a211200ea62/volumes/kubernetes.io~projected/kube-api-access-wchpl:{mountpoint:/var/lib/kubelet/pods/e0a64047-334d-4df8-83de-3a211200ea62/volumes/kubernetes.io~projected/kube-api-access-wchpl major:0 minor:62 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/8a7a1f81026bb4c382f7c631012f8603741ce19bc83962c5813fd7c9c19aa568/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/163e87e6c981492e4a88ec2667e00668252f3c9e5117cdd8d00539dffc20bb75/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/7ead22083c5fad4f975f8e98d1dde2ac08c225c93a49afa179541f18caee399f/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/da9e19ced7d80c833f5cb7d37aa68a5e7a773bd0ee1855585fd5b81baf26a961/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/bb7392f273c9d6e7a19fd8c87a8267671f25ff613087122cb998fa090327eba7/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/9844e9cbc31b32ed484ea61ec198c02715a3da7751cd13fa32b860b09b0d125c/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/3d115660edd324b0693a742a9edb05911760970c986a162939a36f587139c552/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/050aebd4380f7d69ea0b8761656b8e4348eb00dd85a295e978f2a7cea41cac03/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/10155776fdfff54b238bc2d5a09689d3ad1a0360e12251072bd1959fe98b5527/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/7d2d4d6d4a36789d41ee147813cd3f78bc2b86001fb6fd824d5b62160e676cbc/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/0bd6c491d20919959e39fd980ecb8048c2b980354a02a2b963d7b64b477dd16d/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/38297f9fd48456c37ca0eb03f789c99e2736590e450639e5b9811fa896573f9e/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/8d1d791082e2c536a0bc403ea01f61827fa79cfb5be9b11e89d8f302543362a3/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/806a9a27b5d5ea1e38b6b2755f581587f694fcf475de47ce4139ff6a2284bc9c/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/e9d689f0c61f50941ab6737532d75d6a88ef594e4ad622bd33a6f46b4360c418/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/24eecdeca2fb54a43a0d1ec6bc6826b053e308f73f9b1773c92933920654dc94/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/c2de1334535a93e30c7a3d7221c0e5c4a697d55955eb7ee17694ef73d51a3d12/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/d4076f2cd6f29c3a611a3c82e8b0478a81299b22333f578fd82bb8067be3db93/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/9615cbdeff0b0a9b1172c5504b54c9de7004a3685e6d3dd1abf8aecb338caf6f/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/e566f1a508c3abd85035b4a0e85823f662907aa826bd71ee62aa2baed496a81d/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/cd46a031f53280a8feecfc4d4f3679c2f265370828ec11a9c617f0e7e8c51745/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/958720c31111e070d81ff3f032a025f2b73ac098f4ff928126bf8337fdac227c/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-206:{mountpoint:/var/lib/containers/storage/overlay/7413d3f28fe4d9604f99327d93884dc61f34dc74fac88e1c434766ab09773217/merged major:0 minor:206 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/963ac55c40a575b2e17b521cdbba30dd1eaa9d890554b122fb7a206e0f94aa2d/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-224:{mountpoint:/var/lib/containers/storage/overlay/8fec31275d4fff35e30dbd4f6a79c44ffe62ef3930ca4039d3ccffb6946cb159/merged major:0 minor:224 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/0a0f2c36ed2ad929234df99dc56044f095e79afd70c924e5f167ff2e34f35e40/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/30c7148d59ab40cd5bcc44915a5c13cb83fe35c442ff4be506672b5350c91701/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/a7dd7a708e304ef838587d7c00528d858cbf6574b442bc0fa0b8fb92bc4e5575/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/f87ebb725f49acccad99459fc333e74e6cc364950faf0a4cd7c82e7fdb1b4643/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/fc0a575408179e6b7deeb62f8eb452f2da0100a92fd8d0212c19e3764e2ad8fe/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/7103ebad38219774f77782d55c920a72db1fe9f585a38d3d24c2ae8688a0f127/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/c554d37cc3f03a35abe831cad8dff3ce3392f89a0d906472a25e8beb772be86a/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/1819d6af258218822e2370b5e2921bd400bcab9de3bcfd22592ab0564d6d63eb/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/5095fa27f0b353dba54592568fe7afb16e82135ccd5fcf437db5f82126769dc2/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/c86ac2ddb77ab33e7a55af58a305624a2281c8ccf9a41b37331842bd3247dc25/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/08bc238071982521e67a4009506d593512721406f4097156e39020c63879baa7/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/6a40594b0a62f8c2bfd4c65c8ba75d44041b2b58de41bb41a1da401c8f0ac1ea/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/390eb376f366e8bae17c4dde03e5b09a8cb9406bf35957eb54af0068a81aa687/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/38a293e3a71c65213c9edd854f1ee837c05e3a313758dc87624392c04cec9cb2/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 17:58:23.009404 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.006826 4442 manager.go:217] Machine: {Timestamp:2026-10-30 17:58:23.006349848 +0000 UTC m=+0.096412177 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3092944 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20b0856e4023d926525fe62aced39f SystemUUID:ec20b085-6e40-23d9-2652-5fe62aced39f BootID:e92f1dbc-5cb0-4908-a2bd-5927dbebbac5 Filesystems:[{Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/507d24b53aa3c10ca3e0817caae8defea37f8597936fa8f0a8633359eaa7a42d/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d37ce412-3bd6-4093-9d21-3b72d396c1bd/volumes/kubernetes.io~projected/kube-api-access-nh2xp DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/44e585a6-cc1a-4ebc-a592-907ccee3729f/volumes/kubernetes.io~projected/kube-api-access-9n82w DeviceMajor:0 DeviceMinor:66 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eeeb661dbbd8a1ec768602d833f2feab2fa496aa751dbe4ae51816fb5ffc8de4/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0af98be4-3c2f-413e-838e-f842854411f2/volumes/kubernetes.io~projected/kube-api-access-zvntm DeviceMajor:0 DeviceMinor:64 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c41265ebc3199e6664c64099e6b1826ef30d8ab567d887f01bc7fd8e37cfde4d/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/af185b4ea7b5d4e8cc75783cd205502cd36711430d5772eced1ea92c6bec8bf1/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-206 DeviceMajor:0 DeviceMinor:206 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-224 DeviceMajor:0 DeviceMinor:224 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/44e585a6-cc1a-4ebc-a592-907ccee3729f/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/44e585a6-cc1a-4ebc-a592-907ccee3729f/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/59229234-921c-4177-8b85-3e982194f625/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8ba0b8dc-0459-440d-91dc-ad18f4b279fa/volumes/kubernetes.io~projected/kube-api-access-wt5ws DeviceMajor:0 DeviceMinor:67 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a24e24cc-a321-4a62-b023-7b015f155d55/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:208 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bf5ca486-37a8-4da0-acbf-d389caf9f3e6/volumes/kubernetes.io~projected/kube-api-access-84lr2 DeviceMajor:0 DeviceMinor:213 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a28935a6-1847-4f43-9e7a-332c6b3ce49f/volumes/kubernetes.io~projected/kube-api-access-m5sr2 DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f894d624a353c3f8e2c9cdb15ecfa62e4288061b1f6ce1662e6245642069e557/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cdd236e6bad8950bc60a0cf0d2be119eff0267273b1421a7a22fed2b7c70001e/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/516c6b6c-fc40-49cf-93b5-1c4471b02cf0/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/d6164286-3eb9-4e3c-82bc-4886de22b3fb/volumes/kubernetes.io~projected/kube-api-access-ckxh9 DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7541c221aa4e62cb01d078764e9f799e80eb222aa78fabb01004b143f09a8534/userdata/shm DeviceMajor:0 DeviceMinor:222 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/516c6b6c-fc40-49cf-93b5-1c4471b02cf0/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/561db2dcb46df96c6309155b3811410b63e8b552e88a200ed32e8949742b7aa8/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0772096e-be2c-449d-9bf6-c56cbd6c203a/volumes/kubernetes.io~projected/kube-api-access-cr4zd DeviceMajor:0 DeviceMinor:63 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3bc6aeac772dd708ce9046b0304b23ae422455f1543d097553d14f687d51d86f/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d6164286-3eb9-4e3c-82bc-4886de22b3fb/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/516c6b6c-fc40-49cf-93b5-1c4471b02cf0/volumes/kubernetes.io~projected/kube-api-access-65887 DeviceMajor:0 DeviceMinor:65 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/568da9c4-2c4f-4020-95ca-9f7ac7db7c00/volumes/kubernetes.io~projected/kube-api-access-7h9kb DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8215ee49-dab0-40f1-af73-3d4493e3bc04/volumes/kubernetes.io~projected/kube-api-access-gqmfq DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a6842e7e-ad53-424d-be89-f6b29f22ea24/volumes/kubernetes.io~projected/kube-api-access-j2wnc DeviceMajor:0 DeviceMinor:216 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6639644759f355b3688140ef37b276a99cc9aaae3564b1358f929987a4e0af0e/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/90c5a4d78f5e69d3d3d7d2e7a3303f8ed64b8a08cdcc180430b36f637af24dee/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/91820cfa6dc81f9c6c366883a9c69fbc51bd5b4d7cf837fb6a1d97a111addfdf/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5b19ec34f9443e71dd23a1c4adebb7a8fdcfc9a9c2790cb54ca232ad88ccfebd/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/368fe8a43f1f9d87c0fd555c1f310648af3f9c8792bd44010dde89dd3412a7de/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/245afcc3-9c86-4c4e-ac8f-9f7d427c5920/volumes/kubernetes.io~projected/kube-api-access-99kgv DeviceMajor:0 DeviceMinor:221 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/e0a64047-334d-4df8-83de-3a211200ea62/volumes/kubernetes.io~projected/kube-api-access-wchpl DeviceMajor:0 DeviceMinor:62 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:368fe8a43f1f9d8 MacAddress:c6:97:45:9b:c9:14 Speed:10000 Mtu:8901} {Name:507d24b53aa3c10 MacAddress:96:1f:09:27:90:d7 Speed:10000 Mtu:8901} {Name:7541c221aa4e62c MacAddress:76:d6:42:08:d2:c8 Speed:10000 Mtu:8901} {Name:af185b4ea7b5d4e MacAddress:e2:13:15:04:40:fe Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:fd:19:bc:13 Speed:0 Mtu:9001} {Name:br-int MacAddress:6a:2a:dd:72:81:55 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:fd:19:bc:13 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:8a:fe:af:a7:26:a2 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:72:cf:b0:75:8c:54 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 17:58:23.009404 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.007098 4442 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 17:58:23.009404 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.007166 4442 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 17:58:23.009404 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.007489 4442 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 17:58:23.009404 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.007520 4442 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-176.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 17:58:23.009404 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.007685 4442 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 17:58:23.009404 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.007693 4442 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 17:58:23.009404 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.007704 4442 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 17:58:23.009404 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.007719 4442 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 17:58:23.009404 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.007825 4442 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 17:58:23.009404 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.008012 4442 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 17:58:23.009404 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.008147 4442 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 17:58:23.009404 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.008191 4442 kubelet.go:493] "Attempting to sync node with API server" Oct 30 17:58:23.009404 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.008204 4442 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 17:58:23.009404 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.008216 4442 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 17:58:23.009404 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.008225 4442 kubelet.go:405] "Adding apiserver pod source" Oct 30 17:58:23.009404 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.008234 4442 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 17:58:23.009404 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.008912 4442 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 17:58:23.009404 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.008923 4442 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 17:58:23.010370 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.009588 4442 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 17:58:23.011712 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.011692 4442 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 17:58:23.011712 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.011710 4442 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 17:58:23.011854 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.011838 4442 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 17:58:23.012079 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.012064 4442 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 17:58:23.012157 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.012105 4442 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 17:58:23.012263 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.012252 4442 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 17:58:23.012327 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.012272 4442 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 17:58:23.012327 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.012284 4442 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 17:58:23.012327 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.012294 4442 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 17:58:23.012327 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.012304 4442 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 17:58:23.012327 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.012313 4442 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 17:58:23.012562 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.012331 4442 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 17:58:23.012562 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.012342 4442 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 17:58:23.012562 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.012354 4442 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 17:58:23.012562 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.012364 4442 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 17:58:23.012562 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.012382 4442 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 17:58:23.012562 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.012397 4442 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 17:58:23.012562 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.012439 4442 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 17:58:23.012562 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.012449 4442 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 17:58:23.015910 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.015892 4442 server.go:1263] "Started kubelet" Oct 30 17:58:23.016022 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.015962 4442 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 17:58:23.016162 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.016081 4442 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 17:58:23.016230 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.016174 4442 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 17:58:23.016518 ip-10-0-138-176 systemd[1]: Started Kubernetes Kubelet. Oct 30 17:58:23.016652 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.016572 4442 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 17:58:23.021297 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.020285 4442 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 17:58:23.021297 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.020567 4442 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 17:58:23.021297 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.020716 4442 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 17:58:23.021297 ip-10-0-138-176 kubenswrapper[4442]: E1030 17:58:23.020779 4442 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-176.ec2.internal\" not found" Oct 30 17:58:23.021297 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.020959 4442 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 17:58:23.021297 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.020978 4442 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 17:58:23.021689 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.021563 4442 server.go:317] "Adding debug handlers to kubelet server" Oct 30 17:58:23.023316 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.023262 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a28935a6-1847-4f43-9e7a-332c6b3ce49f" volumeName="kubernetes.io/configmap/a28935a6-1847-4f43-9e7a-332c6b3ce49f-iptables-alerter-script" seLinuxMountContext="" Oct 30 17:58:23.023429 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.023325 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0af98be4-3c2f-413e-838e-f842854411f2" volumeName="kubernetes.io/configmap/0af98be4-3c2f-413e-838e-f842854411f2-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 17:58:23.023429 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.023340 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="516c6b6c-fc40-49cf-93b5-1c4471b02cf0" volumeName="kubernetes.io/configmap/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-ovnkube-script-lib" seLinuxMountContext="" Oct 30 17:58:23.023429 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.023365 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59229234-921c-4177-8b85-3e982194f625" volumeName="kubernetes.io/configmap/59229234-921c-4177-8b85-3e982194f625-konnectivity-ca" seLinuxMountContext="" Oct 30 17:58:23.023429 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.023376 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0a64047-334d-4df8-83de-3a211200ea62" volumeName="kubernetes.io/projected/e0a64047-334d-4df8-83de-3a211200ea62-kube-api-access-wchpl" seLinuxMountContext="" Oct 30 17:58:23.023429 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.023394 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="245afcc3-9c86-4c4e-ac8f-9f7d427c5920" volumeName="kubernetes.io/empty-dir/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-data-volume" seLinuxMountContext="" Oct 30 17:58:23.023429 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.023415 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="568da9c4-2c4f-4020-95ca-9f7ac7db7c00" volumeName="kubernetes.io/projected/568da9c4-2c4f-4020-95ca-9f7ac7db7c00-kube-api-access-7h9kb" seLinuxMountContext="" Oct 30 17:58:23.023656 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.023432 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ba0b8dc-0459-440d-91dc-ad18f4b279fa" volumeName="kubernetes.io/configmap/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-cni-binary-copy" seLinuxMountContext="" Oct 30 17:58:23.023656 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.023452 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ba0b8dc-0459-440d-91dc-ad18f4b279fa" volumeName="kubernetes.io/configmap/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-multus-daemon-config" seLinuxMountContext="" Oct 30 17:58:23.023656 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.023473 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0af98be4-3c2f-413e-838e-f842854411f2" volumeName="kubernetes.io/configmap/0af98be4-3c2f-413e-838e-f842854411f2-cni-binary-copy" seLinuxMountContext="" Oct 30 17:58:23.023656 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.023489 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6842e7e-ad53-424d-be89-f6b29f22ea24" volumeName="kubernetes.io/projected/a6842e7e-ad53-424d-be89-f6b29f22ea24-kube-api-access-j2wnc" seLinuxMountContext="" Oct 30 17:58:23.023656 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.023501 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="245afcc3-9c86-4c4e-ac8f-9f7d427c5920" volumeName="kubernetes.io/projected/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-kube-api-access-99kgv" seLinuxMountContext="" Oct 30 17:58:23.023656 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.023530 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44e585a6-cc1a-4ebc-a592-907ccee3729f" volumeName="kubernetes.io/projected/44e585a6-cc1a-4ebc-a592-907ccee3729f-kube-api-access-9n82w" seLinuxMountContext="" Oct 30 17:58:23.023656 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.023546 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8215ee49-dab0-40f1-af73-3d4493e3bc04" volumeName="kubernetes.io/configmap/8215ee49-dab0-40f1-af73-3d4493e3bc04-config-volume" seLinuxMountContext="" Oct 30 17:58:23.023656 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.023566 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8215ee49-dab0-40f1-af73-3d4493e3bc04" volumeName="kubernetes.io/projected/8215ee49-dab0-40f1-af73-3d4493e3bc04-kube-api-access-gqmfq" seLinuxMountContext="" Oct 30 17:58:23.023656 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.023597 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0a64047-334d-4df8-83de-3a211200ea62" volumeName="kubernetes.io/configmap/e0a64047-334d-4df8-83de-3a211200ea62-serviceca" seLinuxMountContext="" Oct 30 17:58:23.023656 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.023618 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0772096e-be2c-449d-9bf6-c56cbd6c203a" volumeName="kubernetes.io/empty-dir/0772096e-be2c-449d-9bf6-c56cbd6c203a-tmp-dir" seLinuxMountContext="" Oct 30 17:58:23.023656 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.023635 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0af98be4-3c2f-413e-838e-f842854411f2" volumeName="kubernetes.io/projected/0af98be4-3c2f-413e-838e-f842854411f2-kube-api-access-zvntm" seLinuxMountContext="" Oct 30 17:58:23.023656 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.023652 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="516c6b6c-fc40-49cf-93b5-1c4471b02cf0" volumeName="kubernetes.io/configmap/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-ovnkube-config" seLinuxMountContext="" Oct 30 17:58:23.024015 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.023684 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59229234-921c-4177-8b85-3e982194f625" volumeName="kubernetes.io/secret/59229234-921c-4177-8b85-3e982194f625-agent-certs" seLinuxMountContext="" Oct 30 17:58:23.024015 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.023696 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ba0b8dc-0459-440d-91dc-ad18f4b279fa" volumeName="kubernetes.io/projected/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-kube-api-access-wt5ws" seLinuxMountContext="" Oct 30 17:58:23.024015 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.023732 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6164286-3eb9-4e3c-82bc-4886de22b3fb" volumeName="kubernetes.io/projected/d6164286-3eb9-4e3c-82bc-4886de22b3fb-kube-api-access-ckxh9" seLinuxMountContext="" Oct 30 17:58:23.024015 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.023749 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0772096e-be2c-449d-9bf6-c56cbd6c203a" volumeName="kubernetes.io/projected/0772096e-be2c-449d-9bf6-c56cbd6c203a-kube-api-access-cr4zd" seLinuxMountContext="" Oct 30 17:58:23.024015 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.023767 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="516c6b6c-fc40-49cf-93b5-1c4471b02cf0" volumeName="kubernetes.io/secret/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 17:58:23.024015 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.023788 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8215ee49-dab0-40f1-af73-3d4493e3bc04" volumeName="kubernetes.io/empty-dir/8215ee49-dab0-40f1-af73-3d4493e3bc04-tmp-dir" seLinuxMountContext="" Oct 30 17:58:23.024015 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.023799 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a28935a6-1847-4f43-9e7a-332c6b3ce49f" volumeName="kubernetes.io/projected/a28935a6-1847-4f43-9e7a-332c6b3ce49f-kube-api-access-m5sr2" seLinuxMountContext="" Oct 30 17:58:23.024015 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.023820 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d37ce412-3bd6-4093-9d21-3b72d396c1bd" volumeName="kubernetes.io/projected/d37ce412-3bd6-4093-9d21-3b72d396c1bd-kube-api-access-nh2xp" seLinuxMountContext="" Oct 30 17:58:23.024015 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.023854 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6164286-3eb9-4e3c-82bc-4886de22b3fb" volumeName="kubernetes.io/secret/d6164286-3eb9-4e3c-82bc-4886de22b3fb-cert" seLinuxMountContext="" Oct 30 17:58:23.024015 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.023877 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44e585a6-cc1a-4ebc-a592-907ccee3729f" volumeName="kubernetes.io/empty-dir/44e585a6-cc1a-4ebc-a592-907ccee3729f-etc-tuned" seLinuxMountContext="" Oct 30 17:58:23.024015 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.023894 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a24e24cc-a321-4a62-b023-7b015f155d55" volumeName="kubernetes.io/secret/a24e24cc-a321-4a62-b023-7b015f155d55-original-pull-secret" seLinuxMountContext="" Oct 30 17:58:23.024015 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.023910 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf5ca486-37a8-4da0-acbf-d389caf9f3e6" volumeName="kubernetes.io/projected/bf5ca486-37a8-4da0-acbf-d389caf9f3e6-kube-api-access-84lr2" seLinuxMountContext="" Oct 30 17:58:23.024015 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.023932 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0af98be4-3c2f-413e-838e-f842854411f2" volumeName="kubernetes.io/configmap/0af98be4-3c2f-413e-838e-f842854411f2-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 17:58:23.024015 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.023944 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44e585a6-cc1a-4ebc-a592-907ccee3729f" volumeName="kubernetes.io/empty-dir/44e585a6-cc1a-4ebc-a592-907ccee3729f-tmp" seLinuxMountContext="" Oct 30 17:58:23.024015 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.023970 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="516c6b6c-fc40-49cf-93b5-1c4471b02cf0" volumeName="kubernetes.io/configmap/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-env-overrides" seLinuxMountContext="" Oct 30 17:58:23.024015 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.023994 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="516c6b6c-fc40-49cf-93b5-1c4471b02cf0" volumeName="kubernetes.io/projected/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-kube-api-access-65887" seLinuxMountContext="" Oct 30 17:58:23.024015 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.024006 4442 reconstruct.go:97] "Volume reconstruction finished" Oct 30 17:58:23.024015 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.024013 4442 reconciler.go:29] "Reconciler: start to sync state" Oct 30 17:58:23.024727 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.020620 4442 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 17:52:45 +0000 UTC" deadline="2028-08-11 09:03:26.452247096 +0000 UTC" Oct 30 17:58:23.024727 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.024418 4442 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15615h5m3.42783623s" Oct 30 17:58:23.026677 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.026651 4442 factory.go:55] Registering systemd factory Oct 30 17:58:23.026677 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.026680 4442 factory.go:223] Registration of the systemd container factory successfully Oct 30 17:58:23.030171 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.026941 4442 factory.go:153] Registering CRI-O factory Oct 30 17:58:23.030171 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.026954 4442 factory.go:223] Registration of the crio container factory successfully Oct 30 17:58:23.030171 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.026999 4442 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 17:58:23.030171 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.027020 4442 factory.go:103] Registering Raw factory Oct 30 17:58:23.030171 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.027032 4442 manager.go:1199] Started watching for new ooms in manager Oct 30 17:58:23.030171 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.027425 4442 manager.go:319] Starting recovery of all containers Oct 30 17:58:23.041724 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.041635 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52rn6_516c6b6c-fc40-49cf-93b5-1c4471b02cf0/ovn-acl-logging/0.log" Oct 30 17:58:23.044110 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.042818 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52rn6_516c6b6c-fc40-49cf-93b5-1c4471b02cf0/ovn-acl-logging/0.log" Oct 30 17:58:23.046780 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.046446 4442 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 17:58:23.056403 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.056372 4442 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 17:58:23.056403 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.056401 4442 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 17:58:23.056599 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.056424 4442 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 17:58:23.056599 ip-10-0-138-176 kubenswrapper[4442]: E1030 17:58:23.056465 4442 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 17:58:23.067291 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.067266 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52rn6_516c6b6c-fc40-49cf-93b5-1c4471b02cf0/ovn-acl-logging/0.log" Oct 30 17:58:23.067651 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.067625 4442 generic.go:356] "Generic (PLEG): container finished" podID="516c6b6c-fc40-49cf-93b5-1c4471b02cf0" containerID="d07de1ff4897c756613da5d24473653c246a3684d6e33058b75bc48eab16cc08" exitCode=1 Oct 30 17:58:23.070247 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.070221 4442 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:58:23.070867 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.070839 4442 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 17:58:23.073969 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.073944 4442 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:58:23.079803 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.079778 4442 generic.go:356] "Generic (PLEG): container finished" podID="0af98be4-3c2f-413e-838e-f842854411f2" containerID="28de92773fb6953ef372ce1bcd8a6793e6b0bfdf9b22287c4530b13caedd3e54" exitCode=0 Oct 30 17:58:23.079803 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.079803 4442 generic.go:356] "Generic (PLEG): container finished" podID="0af98be4-3c2f-413e-838e-f842854411f2" containerID="485e737be18e64e9db94e7449f813ef1dd328082a95375b286ae2fac48b6337b" exitCode=0 Oct 30 17:58:23.080020 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.079812 4442 generic.go:356] "Generic (PLEG): container finished" podID="0af98be4-3c2f-413e-838e-f842854411f2" containerID="22df64c713e5a45f72b39a6b008db1e3894fbcb6737678ed96cb0793b046a9cd" exitCode=0 Oct 30 17:58:23.080020 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.079820 4442 generic.go:356] "Generic (PLEG): container finished" podID="0af98be4-3c2f-413e-838e-f842854411f2" containerID="f2ecd408dbe1af1877160ffe7b9865b538c56c7bdd2a760aaca8ce8e38036d59" exitCode=0 Oct 30 17:58:23.080020 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.079829 4442 generic.go:356] "Generic (PLEG): container finished" podID="0af98be4-3c2f-413e-838e-f842854411f2" containerID="57903c2e169bad6ae7ed053a689ab29d81c12c4faa1aeed18f156f9adb8b8d13" exitCode=0 Oct 30 17:58:23.080020 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.079836 4442 generic.go:356] "Generic (PLEG): container finished" podID="0af98be4-3c2f-413e-838e-f842854411f2" containerID="224a70994571cf40b7ac77cd6dc844d93099de98517fb65cad6f113641d9a662" exitCode=0 Oct 30 17:58:23.083376 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.083348 4442 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:58:23.083376 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.083349 4442 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:58:23.092210 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.092187 4442 generic.go:356] "Generic (PLEG): container finished" podID="bfd89a876f67f9d878ac0d1431582688" containerID="fa6b0527e674aaca86c5e6f2aa27874e46e1076527a3cb31dd7a4f70c1b4d580" exitCode=0 Oct 30 17:58:23.093633 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.093608 4442 manager.go:324] Recovery completed Oct 30 17:58:23.108437 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.108417 4442 cpu_manager.go:226] "Starting" policy="none" Oct 30 17:58:23.108437 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.108433 4442 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 17:58:23.108560 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.108452 4442 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 17:58:23.108622 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.108606 4442 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 17:58:23.109059 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.108625 4442 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 17:58:23.109092 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.109063 4442 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 17:58:23.109092 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.109070 4442 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 17:58:23.109092 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.109076 4442 policy_none.go:50] "Start" Oct 30 17:58:23.109092 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.109085 4442 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 17:58:23.109225 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.109097 4442 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 17:58:23.109266 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.109256 4442 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 17:58:23.109296 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.109266 4442 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 17:58:23.109296 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.109272 4442 policy_none.go:44] "Start" Oct 30 17:58:23.113432 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.113403 4442 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 17:58:23.118714 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.118696 4442 manager.go:341] "Starting Device Plugin manager" Oct 30 17:58:23.118793 ip-10-0-138-176 kubenswrapper[4442]: E1030 17:58:23.118736 4442 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 17:58:23.118793 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.118746 4442 server.go:87] "Starting device plugin registration server" Oct 30 17:58:23.119070 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.119055 4442 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 17:58:23.119171 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.119069 4442 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 17:58:23.119229 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.119192 4442 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 17:58:23.119378 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.119280 4442 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 17:58:23.119378 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.119316 4442 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 17:58:23.119378 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.119323 4442 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 17:58:23.119587 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.119364 4442 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T17:58:23.119304952Z","UUID":"f1633245-5a5b-4426-aa05-2b5a7f77d4a4","Handler":null,"Name":"","Endpoint":""} Oct 30 17:58:23.120610 ip-10-0-138-176 kubenswrapper[4442]: E1030 17:58:23.120588 4442 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 17:58:23.120809 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.120787 4442 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 17:58:23.120898 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.120818 4442 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 17:58:23.157368 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.157310 4442 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-176.ec2.internal"] Oct 30 17:58:23.157867 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.157831 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-176.ec2.internal" event={"ID":"9e81bd78300430e51b01d30caa1123e7","Type":"ContainerStarted","Data":"7b7254c145c86f7687ed674a24901d52ef6550fc1901906d81acdb64bfe0c264"} Oct 30 17:58:23.157867 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.157868 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-176.ec2.internal" event={"ID":"9e81bd78300430e51b01d30caa1123e7","Type":"ContainerStarted","Data":"6639644759f355b3688140ef37b276a99cc9aaae3564b1358f929987a4e0af0e"} Oct 30 17:58:23.158061 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.157888 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal" event={"ID":"bfd89a876f67f9d878ac0d1431582688","Type":"ContainerStarted","Data":"40da92af59281f382128981d7f620727c98b6e5e7fe95406343d7dba1de6a3cf"} Oct 30 17:58:23.158061 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.157899 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal" event={"ID":"bfd89a876f67f9d878ac0d1431582688","Type":"ContainerDied","Data":"fa6b0527e674aaca86c5e6f2aa27874e46e1076527a3cb31dd7a4f70c1b4d580"} Oct 30 17:58:23.158061 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.157901 4442 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal" Oct 30 17:58:23.158061 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.157911 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal" event={"ID":"bfd89a876f67f9d878ac0d1431582688","Type":"ContainerStarted","Data":"561db2dcb46df96c6309155b3811410b63e8b552e88a200ed32e8949742b7aa8"} Oct 30 17:58:23.158061 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.158016 4442 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-176.ec2.internal" Oct 30 17:58:23.178674 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.178646 4442 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:58:23.178819 ip-10-0-138-176 kubenswrapper[4442]: E1030 17:58:23.178704 4442 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-176.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-176.ec2.internal" Oct 30 17:58:23.179151 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.179121 4442 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:58:23.179189 ip-10-0-138-176 kubenswrapper[4442]: E1030 17:58:23.179171 4442 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal" Oct 30 17:58:23.220320 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.220292 4442 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:58:23.221750 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.221726 4442 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:58:23.221850 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.221758 4442 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:58:23.221850 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.221770 4442 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:58:23.221850 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.221801 4442 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-176.ec2.internal" Oct 30 17:58:23.255914 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.255887 4442 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-138-176.ec2.internal" Oct 30 17:58:23.256033 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.255961 4442 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-176.ec2.internal" Oct 30 17:58:23.272374 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.272337 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bfd89a876f67f9d878ac0d1431582688-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal\" (UID: \"bfd89a876f67f9d878ac0d1431582688\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal" Oct 30 17:58:23.272374 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.272371 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfd89a876f67f9d878ac0d1431582688-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal\" (UID: \"bfd89a876f67f9d878ac0d1431582688\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal" Oct 30 17:58:23.272600 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.272394 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9e81bd78300430e51b01d30caa1123e7-config\") pod \"kube-apiserver-proxy-ip-10-0-138-176.ec2.internal\" (UID: \"9e81bd78300430e51b01d30caa1123e7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-176.ec2.internal" Oct 30 17:58:23.373178 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.373079 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9e81bd78300430e51b01d30caa1123e7-config\") pod \"kube-apiserver-proxy-ip-10-0-138-176.ec2.internal\" (UID: \"9e81bd78300430e51b01d30caa1123e7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-176.ec2.internal" Oct 30 17:58:23.373178 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.373110 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bfd89a876f67f9d878ac0d1431582688-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal\" (UID: \"bfd89a876f67f9d878ac0d1431582688\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal" Oct 30 17:58:23.373178 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.373128 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfd89a876f67f9d878ac0d1431582688-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal\" (UID: \"bfd89a876f67f9d878ac0d1431582688\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal" Oct 30 17:58:23.373383 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.373187 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfd89a876f67f9d878ac0d1431582688-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal\" (UID: \"bfd89a876f67f9d878ac0d1431582688\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal" Oct 30 17:58:23.373383 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.373198 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9e81bd78300430e51b01d30caa1123e7-config\") pod \"kube-apiserver-proxy-ip-10-0-138-176.ec2.internal\" (UID: \"9e81bd78300430e51b01d30caa1123e7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-176.ec2.internal" Oct 30 17:58:23.373383 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:23.373201 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bfd89a876f67f9d878ac0d1431582688-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal\" (UID: \"bfd89a876f67f9d878ac0d1431582688\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal" Oct 30 17:58:24.008823 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.008792 4442 apiserver.go:52] "Watching apiserver" Oct 30 17:58:24.047787 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.047755 4442 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 17:58:24.048286 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.048262 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-c84zp","openshift-ingress-canary/ingress-canary-2vkgd","openshift-dns/node-resolver-vp6gj","openshift-dns/dns-default-4qz5q","openshift-multus/network-metrics-daemon-9hrlx","openshift-network-diagnostics/network-check-target-zpsg9","openshift-insights/insights-runtime-extractor-vlnv2","openshift-multus/multus-additional-cni-plugins-78h8z","openshift-multus/multus-wvg8l","openshift-cluster-node-tuning-operator/tuned-6qd9s","openshift-ovn-kubernetes/ovnkube-node-52rn6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh","kube-system/global-pull-secret-syncer-srgjm","openshift-image-registry/node-ca-tsfch","openshift-kube-storage-version-migrator/migrator-66df5d496b-97w68","openshift-network-operator/iptables-alerter-rxpmp","kube-system/kube-apiserver-proxy-ip-10-0-138-176.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal"] Oct 30 17:58:24.050302 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.050276 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9hrlx" Oct 30 17:58:24.055241 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.055221 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4qz5q" Oct 30 17:58:24.055733 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.055712 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vlnv2" Oct 30 17:58:24.067775 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.067751 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-jcq9r\"" Oct 30 17:58:24.067953 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.067933 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fh48h\"" Oct 30 17:58:24.068103 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.067995 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 17:58:24.068103 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.068016 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 17:58:24.068103 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.068070 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 17:58:24.068312 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.068159 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 17:58:24.068312 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.068305 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 17:58:24.068389 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.068324 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 17:58:24.068389 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.068351 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 17:58:24.068389 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.068378 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 17:58:24.073305 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.073282 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 17:58:24.073394 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.073310 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 17:58:24.073394 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.073341 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 17:58:24.073394 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.073384 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 17:58:24.073521 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.073505 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 17:58:24.073549 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.073542 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xl6hf\"" Oct 30 17:58:24.073832 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.073821 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 17:58:24.074574 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.074543 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 17:58:24.074777 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.074762 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-c6627\"" Oct 30 17:58:24.074831 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.074799 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vzq26\"" Oct 30 17:58:24.074831 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.074802 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6fpb9\"" Oct 30 17:58:24.074831 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.074804 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-nwsqb\"" Oct 30 17:58:24.078834 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.078816 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 17:58:24.078979 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.078965 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 17:58:24.079064 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.079049 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 17:58:24.079103 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.079069 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:58:24.079103 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.079073 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 17:58:24.079246 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.079128 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 17:58:24.079246 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.079172 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 17:58:24.079246 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.079218 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 17:58:24.079378 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.079263 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 17:58:24.079378 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.079285 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 17:58:24.079630 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.079614 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 17:58:24.079699 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.079614 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:58:24.079699 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.079678 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 17:58:24.079794 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.079698 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-8mdkh\"" Oct 30 17:58:24.079794 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.079715 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 17:58:24.079881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.079805 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 17:58:24.079881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.079831 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mngtd\"" Oct 30 17:58:24.079881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.079857 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 17:58:24.079881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.079859 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 17:58:24.079881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.079861 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 17:58:24.080094 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.079888 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 17:58:24.080094 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.079933 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-p48xz\"" Oct 30 17:58:24.080094 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.080032 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 17:58:24.080094 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.080043 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pt2qr\"" Oct 30 17:58:24.080094 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.080034 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 17:58:24.080094 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.080070 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 17:58:24.080094 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.080088 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rdnw2\"" Oct 30 17:58:24.080343 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.080106 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-2xlsx\"" Oct 30 17:58:24.080343 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.080106 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 17:58:24.080343 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.080114 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 17:58:24.080343 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.080143 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wj5nd\"" Oct 30 17:58:24.080343 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.080155 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 17:58:24.080343 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.080187 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 17:58:24.080343 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.080195 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9m88r\"" Oct 30 17:58:24.080343 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.080268 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 17:58:24.080343 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.080308 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-t4kbs\"" Oct 30 17:58:24.094458 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.094436 4442 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal" Oct 30 17:58:24.094573 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.094561 4442 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-176.ec2.internal" Oct 30 17:58:24.115673 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.115649 4442 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:58:24.115776 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.115681 4442 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:58:24.115776 ip-10-0-138-176 kubenswrapper[4442]: E1030 17:58:24.115703 4442 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-176.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-176.ec2.internal" Oct 30 17:58:24.115776 ip-10-0-138-176 kubenswrapper[4442]: E1030 17:58:24.115723 4442 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal" Oct 30 17:58:24.121835 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.121810 4442 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 17:58:24.177048 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.177019 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e0a64047-334d-4df8-83de-3a211200ea62-host\") pod \"node-ca-tsfch\" (UID: \"e0a64047-334d-4df8-83de-3a211200ea62\") " pod="openshift-image-registry/node-ca-tsfch" Oct 30 17:58:24.177262 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.177056 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt5ws\" (UniqueName: \"kubernetes.io/projected/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-kube-api-access-wt5ws\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:58:24.177262 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.177081 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqmfq\" (UniqueName: \"kubernetes.io/projected/8215ee49-dab0-40f1-af73-3d4493e3bc04-kube-api-access-gqmfq\") pod \"dns-default-4qz5q\" (UID: \"8215ee49-dab0-40f1-af73-3d4493e3bc04\") " pod="openshift-dns/dns-default-4qz5q" Oct 30 17:58:24.177262 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.177102 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vlnv2\" (UID: \"245afcc3-9c86-4c4e-ac8f-9f7d427c5920\") " pod="openshift-insights/insights-runtime-extractor-vlnv2" Oct 30 17:58:24.177262 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.177168 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-run-systemd\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.177262 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.177198 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-host\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:58:24.177262 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.177218 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-multus-cni-dir\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:58:24.177262 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.177250 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2wnc\" (UniqueName: \"kubernetes.io/projected/a6842e7e-ad53-424d-be89-f6b29f22ea24-kube-api-access-j2wnc\") pod \"migrator-66df5d496b-97w68\" (UID: \"a6842e7e-ad53-424d-be89-f6b29f22ea24\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-97w68" Oct 30 17:58:24.177533 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.177268 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vlnv2\" (UID: \"245afcc3-9c86-4c4e-ac8f-9f7d427c5920\") " pod="openshift-insights/insights-runtime-extractor-vlnv2" Oct 30 17:58:24.177533 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.177287 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99kgv\" (UniqueName: \"kubernetes.io/projected/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-kube-api-access-99kgv\") pod \"insights-runtime-extractor-vlnv2\" (UID: \"245afcc3-9c86-4c4e-ac8f-9f7d427c5920\") " pod="openshift-insights/insights-runtime-extractor-vlnv2" Oct 30 17:58:24.177533 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.177381 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-var-lib-openvswitch\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.177533 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.177431 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0af98be4-3c2f-413e-838e-f842854411f2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-78h8z\" (UID: \"0af98be4-3c2f-413e-838e-f842854411f2\") " pod="openshift-multus/multus-additional-cni-plugins-78h8z" Oct 30 17:58:24.177533 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.177477 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-etc-sysctl-conf\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:58:24.177533 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.177510 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-multus-daemon-config\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:58:24.177533 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.177535 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a24e24cc-a321-4a62-b023-7b015f155d55-dbus\") pod \"global-pull-secret-syncer-srgjm\" (UID: \"a24e24cc-a321-4a62-b023-7b015f155d55\") " pod="kube-system/global-pull-secret-syncer-srgjm" Oct 30 17:58:24.177985 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.177552 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-data-volume\") pod \"insights-runtime-extractor-vlnv2\" (UID: \"245afcc3-9c86-4c4e-ac8f-9f7d427c5920\") " pod="openshift-insights/insights-runtime-extractor-vlnv2" Oct 30 17:58:24.177985 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.177569 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-host-kubelet\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.177985 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.177584 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-host-slash\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.177985 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.177601 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-ovnkube-script-lib\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.177985 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.177643 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-lib-modules\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:58:24.177985 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.177660 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-data-volume\") pod \"insights-runtime-extractor-vlnv2\" (UID: \"245afcc3-9c86-4c4e-ac8f-9f7d427c5920\") " pod="openshift-insights/insights-runtime-extractor-vlnv2" Oct 30 17:58:24.177985 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.177699 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/59229234-921c-4177-8b85-3e982194f625-konnectivity-ca\") pod \"konnectivity-agent-c84zp\" (UID: \"59229234-921c-4177-8b85-3e982194f625\") " pod="kube-system/konnectivity-agent-c84zp" Oct 30 17:58:24.177985 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.177815 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0772096e-be2c-449d-9bf6-c56cbd6c203a-tmp-dir\") pod \"node-resolver-vp6gj\" (UID: \"0772096e-be2c-449d-9bf6-c56cbd6c203a\") " pod="openshift-dns/node-resolver-vp6gj" Oct 30 17:58:24.177985 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.177818 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-ovnkube-script-lib\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.177985 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.177822 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-multus-daemon-config\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:58:24.177985 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.177822 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0af98be4-3c2f-413e-838e-f842854411f2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-78h8z\" (UID: \"0af98be4-3c2f-413e-838e-f842854411f2\") " pod="openshift-multus/multus-additional-cni-plugins-78h8z" Oct 30 17:58:24.177985 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.177857 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-host-cni-bin\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.177985 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.177885 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-host-cni-netd\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.177985 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.177884 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0772096e-be2c-449d-9bf6-c56cbd6c203a-tmp-dir\") pod \"node-resolver-vp6gj\" (UID: \"0772096e-be2c-449d-9bf6-c56cbd6c203a\") " pod="openshift-dns/node-resolver-vp6gj" Oct 30 17:58:24.177985 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.177910 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-etc-sysctl-d\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:58:24.177985 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.177938 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-run-openvswitch\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.177985 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.177970 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-host-run-k8s-cni-cncf-io\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:58:24.177985 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.177990 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-host-run-multus-certs\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:58:24.177985 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.177992 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/59229234-921c-4177-8b85-3e982194f625-konnectivity-ca\") pod \"konnectivity-agent-c84zp\" (UID: \"59229234-921c-4177-8b85-3e982194f625\") " pod="kube-system/konnectivity-agent-c84zp" Oct 30 17:58:24.178607 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.178010 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h9kb\" (UniqueName: \"kubernetes.io/projected/568da9c4-2c4f-4020-95ca-9f7ac7db7c00-kube-api-access-7h9kb\") pod \"network-metrics-daemon-9hrlx\" (UID: \"568da9c4-2c4f-4020-95ca-9f7ac7db7c00\") " pod="openshift-multus/network-metrics-daemon-9hrlx" Oct 30 17:58:24.178607 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.178053 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0af98be4-3c2f-413e-838e-f842854411f2-cnibin\") pod \"multus-additional-cni-plugins-78h8z\" (UID: \"0af98be4-3c2f-413e-838e-f842854411f2\") " pod="openshift-multus/multus-additional-cni-plugins-78h8z" Oct 30 17:58:24.178607 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.178502 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/59229234-921c-4177-8b85-3e982194f625-agent-certs\") pod \"konnectivity-agent-c84zp\" (UID: \"59229234-921c-4177-8b85-3e982194f625\") " pod="kube-system/konnectivity-agent-c84zp" Oct 30 17:58:24.178607 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.178521 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d37ce412-3bd6-4093-9d21-3b72d396c1bd-etc-selinux\") pod \"aws-ebs-csi-driver-node-8wmlh\" (UID: \"d37ce412-3bd6-4093-9d21-3b72d396c1bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" Oct 30 17:58:24.178607 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.178536 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8215ee49-dab0-40f1-af73-3d4493e3bc04-config-volume\") pod \"dns-default-4qz5q\" (UID: \"8215ee49-dab0-40f1-af73-3d4493e3bc04\") " pod="openshift-dns/dns-default-4qz5q" Oct 30 17:58:24.178607 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.178549 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-run-ovn\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.178607 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.178568 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-ovn-node-metrics-cert\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.178920 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.178608 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-run\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:58:24.178920 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.178681 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-sys\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:58:24.178920 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.178706 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-var-lib-kubelet\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:58:24.178920 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.178733 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/44e585a6-cc1a-4ebc-a592-907ccee3729f-etc-tuned\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:58:24.178920 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.178782 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-multus-conf-dir\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:58:24.178920 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.178797 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-ovn-node-metrics-cert\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.178920 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.178801 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d37ce412-3bd6-4093-9d21-3b72d396c1bd-sys-fs\") pod \"aws-ebs-csi-driver-node-8wmlh\" (UID: \"d37ce412-3bd6-4093-9d21-3b72d396c1bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" Oct 30 17:58:24.178920 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.178808 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8215ee49-dab0-40f1-af73-3d4493e3bc04-config-volume\") pod \"dns-default-4qz5q\" (UID: \"8215ee49-dab0-40f1-af73-3d4493e3bc04\") " pod="openshift-dns/dns-default-4qz5q" Oct 30 17:58:24.178920 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.178837 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/44e585a6-cc1a-4ebc-a592-907ccee3729f-etc-tuned\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:58:24.178920 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.178855 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8215ee49-dab0-40f1-af73-3d4493e3bc04-tmp-dir\") pod \"dns-default-4qz5q\" (UID: \"8215ee49-dab0-40f1-af73-3d4493e3bc04\") " pod="openshift-dns/dns-default-4qz5q" Oct 30 17:58:24.178920 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.178859 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/59229234-921c-4177-8b85-3e982194f625-agent-certs\") pod \"konnectivity-agent-c84zp\" (UID: \"59229234-921c-4177-8b85-3e982194f625\") " pod="kube-system/konnectivity-agent-c84zp" Oct 30 17:58:24.178920 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.178878 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.178920 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.178906 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65887\" (UniqueName: \"kubernetes.io/projected/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-kube-api-access-65887\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.179340 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.178935 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-etc-kubernetes\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:58:24.179340 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.178974 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8215ee49-dab0-40f1-af73-3d4493e3bc04-tmp-dir\") pod \"dns-default-4qz5q\" (UID: \"8215ee49-dab0-40f1-af73-3d4493e3bc04\") " pod="openshift-dns/dns-default-4qz5q" Oct 30 17:58:24.179340 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.178998 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-etc-systemd\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:58:24.179340 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.179018 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44e585a6-cc1a-4ebc-a592-907ccee3729f-tmp\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:58:24.179340 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.179033 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-host-run-netns\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:58:24.179550 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.179423 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-host-var-lib-cni-bin\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:58:24.179550 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.179457 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44e585a6-cc1a-4ebc-a592-907ccee3729f-tmp\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:58:24.179550 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.179461 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-hostroot\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:58:24.179550 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.179516 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a24e24cc-a321-4a62-b023-7b015f155d55-kubelet-config\") pod \"global-pull-secret-syncer-srgjm\" (UID: \"a24e24cc-a321-4a62-b023-7b015f155d55\") " pod="kube-system/global-pull-secret-syncer-srgjm" Oct 30 17:58:24.179728 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.179551 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d37ce412-3bd6-4093-9d21-3b72d396c1bd-socket-dir\") pod \"aws-ebs-csi-driver-node-8wmlh\" (UID: \"d37ce412-3bd6-4093-9d21-3b72d396c1bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" Oct 30 17:58:24.179728 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.179642 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh2xp\" (UniqueName: \"kubernetes.io/projected/d37ce412-3bd6-4093-9d21-3b72d396c1bd-kube-api-access-nh2xp\") pod \"aws-ebs-csi-driver-node-8wmlh\" (UID: \"d37ce412-3bd6-4093-9d21-3b72d396c1bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" Oct 30 17:58:24.179983 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.179960 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8215ee49-dab0-40f1-af73-3d4493e3bc04-metrics-tls\") pod \"dns-default-4qz5q\" (UID: \"8215ee49-dab0-40f1-af73-3d4493e3bc04\") " pod="openshift-dns/dns-default-4qz5q" Oct 30 17:58:24.180055 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.180018 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvntm\" (UniqueName: \"kubernetes.io/projected/0af98be4-3c2f-413e-838e-f842854411f2-kube-api-access-zvntm\") pod \"multus-additional-cni-plugins-78h8z\" (UID: \"0af98be4-3c2f-413e-838e-f842854411f2\") " pod="openshift-multus/multus-additional-cni-plugins-78h8z" Oct 30 17:58:24.180224 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.180190 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a24e24cc-a321-4a62-b023-7b015f155d55-original-pull-secret\") pod \"global-pull-secret-syncer-srgjm\" (UID: \"a24e24cc-a321-4a62-b023-7b015f155d55\") " pod="kube-system/global-pull-secret-syncer-srgjm" Oct 30 17:58:24.180339 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.180324 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-crio-socket\") pod \"insights-runtime-extractor-vlnv2\" (UID: \"245afcc3-9c86-4c4e-ac8f-9f7d427c5920\") " pod="openshift-insights/insights-runtime-extractor-vlnv2" Oct 30 17:58:24.180459 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.180445 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-host-run-netns\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.180568 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.180540 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a24e24cc-a321-4a62-b023-7b015f155d55-original-pull-secret\") pod \"global-pull-secret-syncer-srgjm\" (UID: \"a24e24cc-a321-4a62-b023-7b015f155d55\") " pod="kube-system/global-pull-secret-syncer-srgjm" Oct 30 17:58:24.180667 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.180591 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-etc-openvswitch\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.180775 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.180760 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-ovnkube-config\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.180887 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.180874 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0af98be4-3c2f-413e-838e-f842854411f2-cni-binary-copy\") pod \"multus-additional-cni-plugins-78h8z\" (UID: \"0af98be4-3c2f-413e-838e-f842854411f2\") " pod="openshift-multus/multus-additional-cni-plugins-78h8z" Oct 30 17:58:24.180993 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.180980 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0af98be4-3c2f-413e-838e-f842854411f2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-78h8z\" (UID: \"0af98be4-3c2f-413e-838e-f842854411f2\") " pod="openshift-multus/multus-additional-cni-plugins-78h8z" Oct 30 17:58:24.181088 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.181076 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-etc-modprobe-d\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:58:24.181219 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.181206 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d37ce412-3bd6-4093-9d21-3b72d396c1bd-registration-dir\") pod \"aws-ebs-csi-driver-node-8wmlh\" (UID: \"d37ce412-3bd6-4093-9d21-3b72d396c1bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" Oct 30 17:58:24.181333 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.181320 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84lr2\" (UniqueName: \"kubernetes.io/projected/bf5ca486-37a8-4da0-acbf-d389caf9f3e6-kube-api-access-84lr2\") pod \"network-check-target-zpsg9\" (UID: \"bf5ca486-37a8-4da0-acbf-d389caf9f3e6\") " pod="openshift-network-diagnostics/network-check-target-zpsg9" Oct 30 17:58:24.181455 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.181441 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0af98be4-3c2f-413e-838e-f842854411f2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-78h8z\" (UID: \"0af98be4-3c2f-413e-838e-f842854411f2\") " pod="openshift-multus/multus-additional-cni-plugins-78h8z" Oct 30 17:58:24.181568 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.181555 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr4zd\" (UniqueName: \"kubernetes.io/projected/0772096e-be2c-449d-9bf6-c56cbd6c203a-kube-api-access-cr4zd\") pod \"node-resolver-vp6gj\" (UID: \"0772096e-be2c-449d-9bf6-c56cbd6c203a\") " pod="openshift-dns/node-resolver-vp6gj" Oct 30 17:58:24.181678 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.181665 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-system-cni-dir\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:58:24.181785 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.181773 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a28935a6-1847-4f43-9e7a-332c6b3ce49f-iptables-alerter-script\") pod \"iptables-alerter-rxpmp\" (UID: \"a28935a6-1847-4f43-9e7a-332c6b3ce49f\") " pod="openshift-network-operator/iptables-alerter-rxpmp" Oct 30 17:58:24.181891 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.181879 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-systemd-units\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.181998 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.181985 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n82w\" (UniqueName: \"kubernetes.io/projected/44e585a6-cc1a-4ebc-a592-907ccee3729f-kube-api-access-9n82w\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:58:24.182101 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.182087 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckxh9\" (UniqueName: \"kubernetes.io/projected/d6164286-3eb9-4e3c-82bc-4886de22b3fb-kube-api-access-ckxh9\") pod \"ingress-canary-2vkgd\" (UID: \"d6164286-3eb9-4e3c-82bc-4886de22b3fb\") " pod="openshift-ingress-canary/ingress-canary-2vkgd" Oct 30 17:58:24.182489 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.182463 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0af98be4-3c2f-413e-838e-f842854411f2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-78h8z\" (UID: \"0af98be4-3c2f-413e-838e-f842854411f2\") " pod="openshift-multus/multus-additional-cni-plugins-78h8z" Oct 30 17:58:24.182567 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.182464 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0af98be4-3c2f-413e-838e-f842854411f2-cni-binary-copy\") pod \"multus-additional-cni-plugins-78h8z\" (UID: \"0af98be4-3c2f-413e-838e-f842854411f2\") " pod="openshift-multus/multus-additional-cni-plugins-78h8z" Oct 30 17:58:24.182567 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.182530 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-ovnkube-config\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.182678 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.182655 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a28935a6-1847-4f43-9e7a-332c6b3ce49f-iptables-alerter-script\") pod \"iptables-alerter-rxpmp\" (UID: \"a28935a6-1847-4f43-9e7a-332c6b3ce49f\") " pod="openshift-network-operator/iptables-alerter-rxpmp" Oct 30 17:58:24.182678 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.182671 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wchpl\" (UniqueName: \"kubernetes.io/projected/e0a64047-334d-4df8-83de-3a211200ea62-kube-api-access-wchpl\") pod \"node-ca-tsfch\" (UID: \"e0a64047-334d-4df8-83de-3a211200ea62\") " pod="openshift-image-registry/node-ca-tsfch" Oct 30 17:58:24.182863 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.182700 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e0a64047-334d-4df8-83de-3a211200ea62-serviceca\") pod \"node-ca-tsfch\" (UID: \"e0a64047-334d-4df8-83de-3a211200ea62\") " pod="openshift-image-registry/node-ca-tsfch" Oct 30 17:58:24.182863 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.182757 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-cni-binary-copy\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:58:24.182863 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.182790 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-multus-socket-dir-parent\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:58:24.182863 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.182814 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5sr2\" (UniqueName: \"kubernetes.io/projected/a28935a6-1847-4f43-9e7a-332c6b3ce49f-kube-api-access-m5sr2\") pod \"iptables-alerter-rxpmp\" (UID: \"a28935a6-1847-4f43-9e7a-332c6b3ce49f\") " pod="openshift-network-operator/iptables-alerter-rxpmp" Oct 30 17:58:24.182863 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.182837 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e0a64047-334d-4df8-83de-3a211200ea62-serviceca\") pod \"node-ca-tsfch\" (UID: \"e0a64047-334d-4df8-83de-3a211200ea62\") " pod="openshift-image-registry/node-ca-tsfch" Oct 30 17:58:24.182863 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.182842 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0772096e-be2c-449d-9bf6-c56cbd6c203a-hosts-file\") pod \"node-resolver-vp6gj\" (UID: \"0772096e-be2c-449d-9bf6-c56cbd6c203a\") " pod="openshift-dns/node-resolver-vp6gj" Oct 30 17:58:24.183147 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.182960 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-log-socket\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.183147 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.182985 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-cni-binary-copy\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:58:24.183147 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.182995 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-host-run-ovn-kubernetes\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.183399 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.183380 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a28935a6-1847-4f43-9e7a-332c6b3ce49f-host-slash\") pod \"iptables-alerter-rxpmp\" (UID: \"a28935a6-1847-4f43-9e7a-332c6b3ce49f\") " pod="openshift-network-operator/iptables-alerter-rxpmp" Oct 30 17:58:24.183474 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.183434 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d37ce412-3bd6-4093-9d21-3b72d396c1bd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8wmlh\" (UID: \"d37ce412-3bd6-4093-9d21-3b72d396c1bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" Oct 30 17:58:24.183474 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.183470 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/568da9c4-2c4f-4020-95ca-9f7ac7db7c00-metrics-certs\") pod \"network-metrics-daemon-9hrlx\" (UID: \"568da9c4-2c4f-4020-95ca-9f7ac7db7c00\") " pod="openshift-multus/network-metrics-daemon-9hrlx" Oct 30 17:58:24.183610 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.183531 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0af98be4-3c2f-413e-838e-f842854411f2-system-cni-dir\") pod \"multus-additional-cni-plugins-78h8z\" (UID: \"0af98be4-3c2f-413e-838e-f842854411f2\") " pod="openshift-multus/multus-additional-cni-plugins-78h8z" Oct 30 17:58:24.183610 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.183582 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-etc-sysconfig\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:58:24.183725 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.183617 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-etc-kubernetes\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:58:24.183725 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.183678 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d37ce412-3bd6-4093-9d21-3b72d396c1bd-device-dir\") pod \"aws-ebs-csi-driver-node-8wmlh\" (UID: \"d37ce412-3bd6-4093-9d21-3b72d396c1bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" Oct 30 17:58:24.183725 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.183711 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-node-log\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.183834 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.183738 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-cnibin\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:58:24.183834 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.183773 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-host-var-lib-cni-multus\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:58:24.183834 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.183807 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-host-var-lib-kubelet\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:58:24.183973 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.183855 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-env-overrides\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.183973 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.183883 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6164286-3eb9-4e3c-82bc-4886de22b3fb-cert\") pod \"ingress-canary-2vkgd\" (UID: \"d6164286-3eb9-4e3c-82bc-4886de22b3fb\") " pod="openshift-ingress-canary/ingress-canary-2vkgd" Oct 30 17:58:24.184066 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.184037 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-env-overrides\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.184162 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.184128 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6164286-3eb9-4e3c-82bc-4886de22b3fb-cert\") pod \"ingress-canary-2vkgd\" (UID: \"d6164286-3eb9-4e3c-82bc-4886de22b3fb\") " pod="openshift-ingress-canary/ingress-canary-2vkgd" Oct 30 17:58:24.195100 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.195081 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqmfq\" (UniqueName: \"kubernetes.io/projected/8215ee49-dab0-40f1-af73-3d4493e3bc04-kube-api-access-gqmfq\") pod \"dns-default-4qz5q\" (UID: \"8215ee49-dab0-40f1-af73-3d4493e3bc04\") " pod="openshift-dns/dns-default-4qz5q" Oct 30 17:58:24.196935 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.196914 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2wnc\" (UniqueName: \"kubernetes.io/projected/a6842e7e-ad53-424d-be89-f6b29f22ea24-kube-api-access-j2wnc\") pod \"migrator-66df5d496b-97w68\" (UID: \"a6842e7e-ad53-424d-be89-f6b29f22ea24\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-97w68" Oct 30 17:58:24.197011 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.196965 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt5ws\" (UniqueName: \"kubernetes.io/projected/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-kube-api-access-wt5ws\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:58:24.197112 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.197095 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvntm\" (UniqueName: \"kubernetes.io/projected/0af98be4-3c2f-413e-838e-f842854411f2-kube-api-access-zvntm\") pod \"multus-additional-cni-plugins-78h8z\" (UID: \"0af98be4-3c2f-413e-838e-f842854411f2\") " pod="openshift-multus/multus-additional-cni-plugins-78h8z" Oct 30 17:58:24.197216 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.197197 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99kgv\" (UniqueName: \"kubernetes.io/projected/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-kube-api-access-99kgv\") pod \"insights-runtime-extractor-vlnv2\" (UID: \"245afcc3-9c86-4c4e-ac8f-9f7d427c5920\") " pod="openshift-insights/insights-runtime-extractor-vlnv2" Oct 30 17:58:24.197260 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.197241 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h9kb\" (UniqueName: \"kubernetes.io/projected/568da9c4-2c4f-4020-95ca-9f7ac7db7c00-kube-api-access-7h9kb\") pod \"network-metrics-daemon-9hrlx\" (UID: \"568da9c4-2c4f-4020-95ca-9f7ac7db7c00\") " pod="openshift-multus/network-metrics-daemon-9hrlx" Oct 30 17:58:24.197482 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.197468 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65887\" (UniqueName: \"kubernetes.io/projected/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-kube-api-access-65887\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.198191 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.198176 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh2xp\" (UniqueName: \"kubernetes.io/projected/d37ce412-3bd6-4093-9d21-3b72d396c1bd-kube-api-access-nh2xp\") pod \"aws-ebs-csi-driver-node-8wmlh\" (UID: \"d37ce412-3bd6-4093-9d21-3b72d396c1bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" Oct 30 17:58:24.200731 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.200714 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n82w\" (UniqueName: \"kubernetes.io/projected/44e585a6-cc1a-4ebc-a592-907ccee3729f-kube-api-access-9n82w\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:58:24.202123 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.202108 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr4zd\" (UniqueName: \"kubernetes.io/projected/0772096e-be2c-449d-9bf6-c56cbd6c203a-kube-api-access-cr4zd\") pod \"node-resolver-vp6gj\" (UID: \"0772096e-be2c-449d-9bf6-c56cbd6c203a\") " pod="openshift-dns/node-resolver-vp6gj" Oct 30 17:58:24.202282 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.202267 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckxh9\" (UniqueName: \"kubernetes.io/projected/d6164286-3eb9-4e3c-82bc-4886de22b3fb-kube-api-access-ckxh9\") pod \"ingress-canary-2vkgd\" (UID: \"d6164286-3eb9-4e3c-82bc-4886de22b3fb\") " pod="openshift-ingress-canary/ingress-canary-2vkgd" Oct 30 17:58:24.203323 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.203304 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5sr2\" (UniqueName: \"kubernetes.io/projected/a28935a6-1847-4f43-9e7a-332c6b3ce49f-kube-api-access-m5sr2\") pod \"iptables-alerter-rxpmp\" (UID: \"a28935a6-1847-4f43-9e7a-332c6b3ce49f\") " pod="openshift-network-operator/iptables-alerter-rxpmp" Oct 30 17:58:24.203417 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.203403 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wchpl\" (UniqueName: \"kubernetes.io/projected/e0a64047-334d-4df8-83de-3a211200ea62-kube-api-access-wchpl\") pod \"node-ca-tsfch\" (UID: \"e0a64047-334d-4df8-83de-3a211200ea62\") " pod="openshift-image-registry/node-ca-tsfch" Oct 30 17:58:24.203940 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.203926 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84lr2\" (UniqueName: \"kubernetes.io/projected/bf5ca486-37a8-4da0-acbf-d389caf9f3e6-kube-api-access-84lr2\") pod \"network-check-target-zpsg9\" (UID: \"bf5ca486-37a8-4da0-acbf-d389caf9f3e6\") " pod="openshift-network-diagnostics/network-check-target-zpsg9" Oct 30 17:58:24.284554 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.284463 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d37ce412-3bd6-4093-9d21-3b72d396c1bd-socket-dir\") pod \"aws-ebs-csi-driver-node-8wmlh\" (UID: \"d37ce412-3bd6-4093-9d21-3b72d396c1bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" Oct 30 17:58:24.284554 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.284495 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8215ee49-dab0-40f1-af73-3d4493e3bc04-metrics-tls\") pod \"dns-default-4qz5q\" (UID: \"8215ee49-dab0-40f1-af73-3d4493e3bc04\") " pod="openshift-dns/dns-default-4qz5q" Oct 30 17:58:24.284554 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.284510 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-crio-socket\") pod \"insights-runtime-extractor-vlnv2\" (UID: \"245afcc3-9c86-4c4e-ac8f-9f7d427c5920\") " pod="openshift-insights/insights-runtime-extractor-vlnv2" Oct 30 17:58:24.284554 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.284527 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-host-run-netns\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.284554 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.284549 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-etc-openvswitch\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.284905 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.284597 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-host-run-netns\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.284905 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.284597 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d37ce412-3bd6-4093-9d21-3b72d396c1bd-socket-dir\") pod \"aws-ebs-csi-driver-node-8wmlh\" (UID: \"d37ce412-3bd6-4093-9d21-3b72d396c1bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" Oct 30 17:58:24.284905 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.284604 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-crio-socket\") pod \"insights-runtime-extractor-vlnv2\" (UID: \"245afcc3-9c86-4c4e-ac8f-9f7d427c5920\") " pod="openshift-insights/insights-runtime-extractor-vlnv2" Oct 30 17:58:24.284905 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.284632 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0af98be4-3c2f-413e-838e-f842854411f2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-78h8z\" (UID: \"0af98be4-3c2f-413e-838e-f842854411f2\") " pod="openshift-multus/multus-additional-cni-plugins-78h8z" Oct 30 17:58:24.284905 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.284641 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-etc-openvswitch\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.284905 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.284660 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-etc-modprobe-d\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:58:24.284905 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.284678 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d37ce412-3bd6-4093-9d21-3b72d396c1bd-registration-dir\") pod \"aws-ebs-csi-driver-node-8wmlh\" (UID: \"d37ce412-3bd6-4093-9d21-3b72d396c1bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" Oct 30 17:58:24.284905 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.284706 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-system-cni-dir\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:58:24.284905 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.284713 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0af98be4-3c2f-413e-838e-f842854411f2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-78h8z\" (UID: \"0af98be4-3c2f-413e-838e-f842854411f2\") " pod="openshift-multus/multus-additional-cni-plugins-78h8z" Oct 30 17:58:24.284905 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.284728 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-systemd-units\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.284905 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.284759 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-multus-socket-dir-parent\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:58:24.284905 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.284764 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d37ce412-3bd6-4093-9d21-3b72d396c1bd-registration-dir\") pod \"aws-ebs-csi-driver-node-8wmlh\" (UID: \"d37ce412-3bd6-4093-9d21-3b72d396c1bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" Oct 30 17:58:24.284905 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.284763 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-system-cni-dir\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:58:24.284905 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.284780 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0772096e-be2c-449d-9bf6-c56cbd6c203a-hosts-file\") pod \"node-resolver-vp6gj\" (UID: \"0772096e-be2c-449d-9bf6-c56cbd6c203a\") " pod="openshift-dns/node-resolver-vp6gj" Oct 30 17:58:24.284905 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.284803 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-etc-modprobe-d\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:58:24.284905 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.284806 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-systemd-units\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.284905 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.284827 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-multus-socket-dir-parent\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:58:24.284905 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.284838 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-log-socket\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.284905 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.284806 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-log-socket\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.284905 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.284871 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-host-run-ovn-kubernetes\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.284905 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.284878 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0772096e-be2c-449d-9bf6-c56cbd6c203a-hosts-file\") pod \"node-resolver-vp6gj\" (UID: \"0772096e-be2c-449d-9bf6-c56cbd6c203a\") " pod="openshift-dns/node-resolver-vp6gj" Oct 30 17:58:24.284905 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.284897 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a28935a6-1847-4f43-9e7a-332c6b3ce49f-host-slash\") pod \"iptables-alerter-rxpmp\" (UID: \"a28935a6-1847-4f43-9e7a-332c6b3ce49f\") " pod="openshift-network-operator/iptables-alerter-rxpmp" Oct 30 17:58:24.284905 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.284914 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-host-run-ovn-kubernetes\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.284926 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a28935a6-1847-4f43-9e7a-332c6b3ce49f-host-slash\") pod \"iptables-alerter-rxpmp\" (UID: \"a28935a6-1847-4f43-9e7a-332c6b3ce49f\") " pod="openshift-network-operator/iptables-alerter-rxpmp" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.284949 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d37ce412-3bd6-4093-9d21-3b72d396c1bd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8wmlh\" (UID: \"d37ce412-3bd6-4093-9d21-3b72d396c1bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.284949 4442 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.284970 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/568da9c4-2c4f-4020-95ca-9f7ac7db7c00-metrics-certs\") pod \"network-metrics-daemon-9hrlx\" (UID: \"568da9c4-2c4f-4020-95ca-9f7ac7db7c00\") " pod="openshift-multus/network-metrics-daemon-9hrlx" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.284985 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0af98be4-3c2f-413e-838e-f842854411f2-system-cni-dir\") pod \"multus-additional-cni-plugins-78h8z\" (UID: \"0af98be4-3c2f-413e-838e-f842854411f2\") " pod="openshift-multus/multus-additional-cni-plugins-78h8z" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.284999 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-etc-sysconfig\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285025 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d37ce412-3bd6-4093-9d21-3b72d396c1bd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8wmlh\" (UID: \"d37ce412-3bd6-4093-9d21-3b72d396c1bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285032 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-etc-kubernetes\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285049 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-etc-sysconfig\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285050 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0af98be4-3c2f-413e-838e-f842854411f2-system-cni-dir\") pod \"multus-additional-cni-plugins-78h8z\" (UID: \"0af98be4-3c2f-413e-838e-f842854411f2\") " pod="openshift-multus/multus-additional-cni-plugins-78h8z" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285061 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d37ce412-3bd6-4093-9d21-3b72d396c1bd-device-dir\") pod \"aws-ebs-csi-driver-node-8wmlh\" (UID: \"d37ce412-3bd6-4093-9d21-3b72d396c1bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285078 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-etc-kubernetes\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285100 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d37ce412-3bd6-4093-9d21-3b72d396c1bd-device-dir\") pod \"aws-ebs-csi-driver-node-8wmlh\" (UID: \"d37ce412-3bd6-4093-9d21-3b72d396c1bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: E1030 17:58:24.285111 4442 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285159 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-node-log\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285100 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-node-log\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: E1030 17:58:24.285200 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/568da9c4-2c4f-4020-95ca-9f7ac7db7c00-metrics-certs podName:568da9c4-2c4f-4020-95ca-9f7ac7db7c00 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:24.785185595 +0000 UTC m=+1.875247932 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/568da9c4-2c4f-4020-95ca-9f7ac7db7c00-metrics-certs") pod "network-metrics-daemon-9hrlx" (UID: "568da9c4-2c4f-4020-95ca-9f7ac7db7c00") : secret "metrics-daemon-secret" not found Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285215 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-cnibin\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285234 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-host-var-lib-cni-multus\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285248 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-host-var-lib-kubelet\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285265 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e0a64047-334d-4df8-83de-3a211200ea62-host\") pod \"node-ca-tsfch\" (UID: \"e0a64047-334d-4df8-83de-3a211200ea62\") " pod="openshift-image-registry/node-ca-tsfch" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285289 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vlnv2\" (UID: \"245afcc3-9c86-4c4e-ac8f-9f7d427c5920\") " pod="openshift-insights/insights-runtime-extractor-vlnv2" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285300 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-host-var-lib-cni-multus\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285312 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-host-var-lib-kubelet\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285339 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-run-systemd\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285345 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e0a64047-334d-4df8-83de-3a211200ea62-host\") pod \"node-ca-tsfch\" (UID: \"e0a64047-334d-4df8-83de-3a211200ea62\") " pod="openshift-image-registry/node-ca-tsfch" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285385 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-host\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285410 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-multus-cni-dir\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285416 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-run-systemd\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285437 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vlnv2\" (UID: \"245afcc3-9c86-4c4e-ac8f-9f7d427c5920\") " pod="openshift-insights/insights-runtime-extractor-vlnv2" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285460 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-host\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285464 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-var-lib-openvswitch\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285495 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-multus-cni-dir\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285502 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-var-lib-openvswitch\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: E1030 17:58:24.285516 4442 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: E1030 17:58:24.285559 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-insights-runtime-extractor-tls podName:245afcc3-9c86-4c4e-ac8f-9f7d427c5920 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:24.78555022 +0000 UTC m=+1.875612582 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vlnv2" (UID: "245afcc3-9c86-4c4e-ac8f-9f7d427c5920") : secret "insights-runtime-extractor-tls" not found Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285499 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-etc-sysctl-conf\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285592 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-etc-sysctl-conf\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285591 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a24e24cc-a321-4a62-b023-7b015f155d55-dbus\") pod \"global-pull-secret-syncer-srgjm\" (UID: \"a24e24cc-a321-4a62-b023-7b015f155d55\") " pod="kube-system/global-pull-secret-syncer-srgjm" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285615 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-host-kubelet\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285627 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a24e24cc-a321-4a62-b023-7b015f155d55-dbus\") pod \"global-pull-secret-syncer-srgjm\" (UID: \"a24e24cc-a321-4a62-b023-7b015f155d55\") " pod="kube-system/global-pull-secret-syncer-srgjm" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285631 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-host-slash\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285651 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-host-slash\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285656 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-lib-modules\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285686 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-host-cni-bin\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285691 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-host-kubelet\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285709 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-host-cni-netd\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285731 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-host-cni-bin\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285739 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-lib-modules\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285755 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-etc-sysctl-d\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285772 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-run-openvswitch\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285784 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-host-cni-netd\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285788 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-host-run-k8s-cni-cncf-io\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285814 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-host-run-k8s-cni-cncf-io\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285830 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-run-openvswitch\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285851 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-host-run-multus-certs\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285866 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-etc-sysctl-d\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285887 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-host-run-multus-certs\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285885 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0af98be4-3c2f-413e-838e-f842854411f2-cnibin\") pod \"multus-additional-cni-plugins-78h8z\" (UID: \"0af98be4-3c2f-413e-838e-f842854411f2\") " pod="openshift-multus/multus-additional-cni-plugins-78h8z" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285887 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-cnibin\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285921 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d37ce412-3bd6-4093-9d21-3b72d396c1bd-etc-selinux\") pod \"aws-ebs-csi-driver-node-8wmlh\" (UID: \"d37ce412-3bd6-4093-9d21-3b72d396c1bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" Oct 30 17:58:24.285881 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285925 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0af98be4-3c2f-413e-838e-f842854411f2-cnibin\") pod \"multus-additional-cni-plugins-78h8z\" (UID: \"0af98be4-3c2f-413e-838e-f842854411f2\") " pod="openshift-multus/multus-additional-cni-plugins-78h8z" Oct 30 17:58:24.287780 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285949 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d37ce412-3bd6-4093-9d21-3b72d396c1bd-etc-selinux\") pod \"aws-ebs-csi-driver-node-8wmlh\" (UID: \"d37ce412-3bd6-4093-9d21-3b72d396c1bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" Oct 30 17:58:24.287780 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285954 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-run-ovn\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.287780 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285972 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-run-ovn\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.287780 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.285990 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-run\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:58:24.287780 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.286006 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-sys\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:58:24.287780 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.286019 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-var-lib-kubelet\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:58:24.287780 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.286035 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-multus-conf-dir\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:58:24.287780 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.286048 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d37ce412-3bd6-4093-9d21-3b72d396c1bd-sys-fs\") pod \"aws-ebs-csi-driver-node-8wmlh\" (UID: \"d37ce412-3bd6-4093-9d21-3b72d396c1bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" Oct 30 17:58:24.287780 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.286035 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-run\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:58:24.287780 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.286058 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vlnv2\" (UID: \"245afcc3-9c86-4c4e-ac8f-9f7d427c5920\") " pod="openshift-insights/insights-runtime-extractor-vlnv2" Oct 30 17:58:24.287780 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.286090 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d37ce412-3bd6-4093-9d21-3b72d396c1bd-sys-fs\") pod \"aws-ebs-csi-driver-node-8wmlh\" (UID: \"d37ce412-3bd6-4093-9d21-3b72d396c1bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wmlh" Oct 30 17:58:24.287780 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.286099 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-multus-conf-dir\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:58:24.287780 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.286108 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.287780 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.286114 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-sys\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:58:24.287780 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.286091 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-var-lib-kubelet\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:58:24.287780 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.286156 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-etc-kubernetes\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:58:24.287780 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.286176 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-etc-systemd\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:58:24.287780 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.286182 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/516c6b6c-fc40-49cf-93b5-1c4471b02cf0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-52rn6\" (UID: \"516c6b6c-fc40-49cf-93b5-1c4471b02cf0\") " pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.287780 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.286200 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-etc-kubernetes\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:58:24.287780 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.286191 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-host-run-netns\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:58:24.287780 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.286220 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/44e585a6-cc1a-4ebc-a592-907ccee3729f-etc-systemd\") pod \"tuned-6qd9s\" (UID: \"44e585a6-cc1a-4ebc-a592-907ccee3729f\") " pod="openshift-cluster-node-tuning-operator/tuned-6qd9s" Oct 30 17:58:24.287780 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.286226 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-host-run-netns\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:58:24.287780 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.286238 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-host-var-lib-cni-bin\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:58:24.287780 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.286258 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-hostroot\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:58:24.287780 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.286272 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a24e24cc-a321-4a62-b023-7b015f155d55-kubelet-config\") pod \"global-pull-secret-syncer-srgjm\" (UID: \"a24e24cc-a321-4a62-b023-7b015f155d55\") " pod="kube-system/global-pull-secret-syncer-srgjm" Oct 30 17:58:24.287780 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.286317 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-host-var-lib-cni-bin\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:58:24.287780 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.286320 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8ba0b8dc-0459-440d-91dc-ad18f4b279fa-hostroot\") pod \"multus-wvg8l\" (UID: \"8ba0b8dc-0459-440d-91dc-ad18f4b279fa\") " pod="openshift-multus/multus-wvg8l" Oct 30 17:58:24.287780 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.286346 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a24e24cc-a321-4a62-b023-7b015f155d55-kubelet-config\") pod \"global-pull-secret-syncer-srgjm\" (UID: \"a24e24cc-a321-4a62-b023-7b015f155d55\") " pod="kube-system/global-pull-secret-syncer-srgjm" Oct 30 17:58:24.287780 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.287335 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8215ee49-dab0-40f1-af73-3d4493e3bc04-metrics-tls\") pod \"dns-default-4qz5q\" (UID: \"8215ee49-dab0-40f1-af73-3d4493e3bc04\") " pod="openshift-dns/dns-default-4qz5q" Oct 30 17:58:24.356226 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.356195 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4qz5q" Oct 30 17:58:24.648815 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.648671 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vp6gj_0772096e-be2c-449d-9bf6-c56cbd6c203a/dns-node-resolver/0.log" Oct 30 17:58:24.672608 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.672580 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:24.686827 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.686801 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4qz5q"] Oct 30 17:58:24.692410 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:24.692384 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8215ee49_dab0_40f1_af73_3d4493e3bc04.slice/crio-92d72bacf632151a9d4f2d50587650b75e30c6345561313f906141e99e4142be WatchSource:0}: Error finding container 92d72bacf632151a9d4f2d50587650b75e30c6345561313f906141e99e4142be: Status 404 returned error can't find the container with id 92d72bacf632151a9d4f2d50587650b75e30c6345561313f906141e99e4142be Oct 30 17:58:24.694613 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.694595 4442 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:58:24.789404 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.789368 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vlnv2\" (UID: \"245afcc3-9c86-4c4e-ac8f-9f7d427c5920\") " pod="openshift-insights/insights-runtime-extractor-vlnv2" Oct 30 17:58:24.789558 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:24.789432 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/568da9c4-2c4f-4020-95ca-9f7ac7db7c00-metrics-certs\") pod \"network-metrics-daemon-9hrlx\" (UID: \"568da9c4-2c4f-4020-95ca-9f7ac7db7c00\") " pod="openshift-multus/network-metrics-daemon-9hrlx" Oct 30 17:58:24.789558 ip-10-0-138-176 kubenswrapper[4442]: E1030 17:58:24.789516 4442 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 17:58:24.789558 ip-10-0-138-176 kubenswrapper[4442]: E1030 17:58:24.789519 4442 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:58:24.789666 ip-10-0-138-176 kubenswrapper[4442]: E1030 17:58:24.789566 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/568da9c4-2c4f-4020-95ca-9f7ac7db7c00-metrics-certs podName:568da9c4-2c4f-4020-95ca-9f7ac7db7c00 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:25.789554241 +0000 UTC m=+2.879616561 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/568da9c4-2c4f-4020-95ca-9f7ac7db7c00-metrics-certs") pod "network-metrics-daemon-9hrlx" (UID: "568da9c4-2c4f-4020-95ca-9f7ac7db7c00") : secret "metrics-daemon-secret" not found Oct 30 17:58:24.789666 ip-10-0-138-176 kubenswrapper[4442]: E1030 17:58:24.789578 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-insights-runtime-extractor-tls podName:245afcc3-9c86-4c4e-ac8f-9f7d427c5920 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:25.789572297 +0000 UTC m=+2.879634613 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vlnv2" (UID: "245afcc3-9c86-4c4e-ac8f-9f7d427c5920") : secret "insights-runtime-extractor-tls" not found Oct 30 17:58:25.098210 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:25.098113 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4qz5q" event={"ID":"8215ee49-dab0-40f1-af73-3d4493e3bc04","Type":"ContainerStarted","Data":"92d72bacf632151a9d4f2d50587650b75e30c6345561313f906141e99e4142be"} Oct 30 17:58:25.649831 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:25.649799 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tsfch_e0a64047-334d-4df8-83de-3a211200ea62/node-ca/0.log" Oct 30 17:58:25.798556 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:25.798516 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vlnv2\" (UID: \"245afcc3-9c86-4c4e-ac8f-9f7d427c5920\") " pod="openshift-insights/insights-runtime-extractor-vlnv2" Oct 30 17:58:25.798728 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:25.798576 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/568da9c4-2c4f-4020-95ca-9f7ac7db7c00-metrics-certs\") pod \"network-metrics-daemon-9hrlx\" (UID: \"568da9c4-2c4f-4020-95ca-9f7ac7db7c00\") " pod="openshift-multus/network-metrics-daemon-9hrlx" Oct 30 17:58:25.798728 ip-10-0-138-176 kubenswrapper[4442]: E1030 17:58:25.798675 4442 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 17:58:25.798728 ip-10-0-138-176 kubenswrapper[4442]: E1030 17:58:25.798674 4442 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:58:25.798821 ip-10-0-138-176 kubenswrapper[4442]: E1030 17:58:25.798736 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/568da9c4-2c4f-4020-95ca-9f7ac7db7c00-metrics-certs podName:568da9c4-2c4f-4020-95ca-9f7ac7db7c00 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:27.798719167 +0000 UTC m=+4.888781493 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/568da9c4-2c4f-4020-95ca-9f7ac7db7c00-metrics-certs") pod "network-metrics-daemon-9hrlx" (UID: "568da9c4-2c4f-4020-95ca-9f7ac7db7c00") : secret "metrics-daemon-secret" not found Oct 30 17:58:25.798821 ip-10-0-138-176 kubenswrapper[4442]: E1030 17:58:25.798755 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-insights-runtime-extractor-tls podName:245afcc3-9c86-4c4e-ac8f-9f7d427c5920 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:27.798742648 +0000 UTC m=+4.888804965 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vlnv2" (UID: "245afcc3-9c86-4c4e-ac8f-9f7d427c5920") : secret "insights-runtime-extractor-tls" not found Oct 30 17:58:26.049429 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:26.049352 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2vkgd_d6164286-3eb9-4e3c-82bc-4886de22b3fb/serve-healthcheck-canary/0.log" Oct 30 17:58:26.649899 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:26.649877 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-97w68_a6842e7e-ad53-424d-be89-f6b29f22ea24/migrator/0.log" Oct 30 17:58:27.105850 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:27.105816 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4qz5q" event={"ID":"8215ee49-dab0-40f1-af73-3d4493e3bc04","Type":"ContainerStarted","Data":"1b26c751e3206f71f948fcdbc0354a2f3ff6603622f69e59ac12c19620edb1f1"} Oct 30 17:58:27.106177 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:27.105864 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4qz5q" event={"ID":"8215ee49-dab0-40f1-af73-3d4493e3bc04","Type":"ContainerStarted","Data":"ea9bd8ce398f1e68bd649de6ab2fe1d947611a7e91343ee2f2d0ec915f397009"} Oct 30 17:58:27.106177 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:27.105952 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4qz5q" Oct 30 17:58:27.312781 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:27.312746 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zpsg9" Oct 30 17:58:27.812034 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:27.811992 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/568da9c4-2c4f-4020-95ca-9f7ac7db7c00-metrics-certs\") pod \"network-metrics-daemon-9hrlx\" (UID: \"568da9c4-2c4f-4020-95ca-9f7ac7db7c00\") " pod="openshift-multus/network-metrics-daemon-9hrlx" Oct 30 17:58:27.812520 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:27.812047 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vlnv2\" (UID: \"245afcc3-9c86-4c4e-ac8f-9f7d427c5920\") " pod="openshift-insights/insights-runtime-extractor-vlnv2" Oct 30 17:58:27.812520 ip-10-0-138-176 kubenswrapper[4442]: E1030 17:58:27.812183 4442 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 17:58:27.812520 ip-10-0-138-176 kubenswrapper[4442]: E1030 17:58:27.812260 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/568da9c4-2c4f-4020-95ca-9f7ac7db7c00-metrics-certs podName:568da9c4-2c4f-4020-95ca-9f7ac7db7c00 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:31.812240101 +0000 UTC m=+8.902302444 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/568da9c4-2c4f-4020-95ca-9f7ac7db7c00-metrics-certs") pod "network-metrics-daemon-9hrlx" (UID: "568da9c4-2c4f-4020-95ca-9f7ac7db7c00") : secret "metrics-daemon-secret" not found Oct 30 17:58:27.812520 ip-10-0-138-176 kubenswrapper[4442]: E1030 17:58:27.812187 4442 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:58:27.812520 ip-10-0-138-176 kubenswrapper[4442]: E1030 17:58:27.812355 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-insights-runtime-extractor-tls podName:245afcc3-9c86-4c4e-ac8f-9f7d427c5920 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:31.812336747 +0000 UTC m=+8.902399081 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vlnv2" (UID: "245afcc3-9c86-4c4e-ac8f-9f7d427c5920") : secret "insights-runtime-extractor-tls" not found Oct 30 17:58:29.619072 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:29.619027 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:29.619457 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:29.619257 4442 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:58:31.493732 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:31.493700 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-c84zp" Oct 30 17:58:31.836816 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:31.836735 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/568da9c4-2c4f-4020-95ca-9f7ac7db7c00-metrics-certs\") pod \"network-metrics-daemon-9hrlx\" (UID: \"568da9c4-2c4f-4020-95ca-9f7ac7db7c00\") " pod="openshift-multus/network-metrics-daemon-9hrlx" Oct 30 17:58:31.836816 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:31.836775 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vlnv2\" (UID: \"245afcc3-9c86-4c4e-ac8f-9f7d427c5920\") " pod="openshift-insights/insights-runtime-extractor-vlnv2" Oct 30 17:58:31.836990 ip-10-0-138-176 kubenswrapper[4442]: E1030 17:58:31.836870 4442 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:58:31.836990 ip-10-0-138-176 kubenswrapper[4442]: E1030 17:58:31.836926 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-insights-runtime-extractor-tls podName:245afcc3-9c86-4c4e-ac8f-9f7d427c5920 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:39.836912843 +0000 UTC m=+16.926975159 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vlnv2" (UID: "245afcc3-9c86-4c4e-ac8f-9f7d427c5920") : secret "insights-runtime-extractor-tls" not found Oct 30 17:58:31.839148 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:31.839114 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/568da9c4-2c4f-4020-95ca-9f7ac7db7c00-metrics-certs\") pod \"network-metrics-daemon-9hrlx\" (UID: \"568da9c4-2c4f-4020-95ca-9f7ac7db7c00\") " pod="openshift-multus/network-metrics-daemon-9hrlx" Oct 30 17:58:31.851078 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:31.851054 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9hrlx" Oct 30 17:58:32.147075 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:32.147038 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9hrlx"] Oct 30 17:58:32.152076 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:32.152047 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod568da9c4_2c4f_4020_95ca_9f7ac7db7c00.slice/crio-1faf4187d8b069500e2a708665faf6d57f2a86bedc2e98f6a9d6f83e0196b448 WatchSource:0}: Error finding container 1faf4187d8b069500e2a708665faf6d57f2a86bedc2e98f6a9d6f83e0196b448: Status 404 returned error can't find the container with id 1faf4187d8b069500e2a708665faf6d57f2a86bedc2e98f6a9d6f83e0196b448 Oct 30 17:58:33.127113 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:33.123449 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9hrlx" event={"ID":"568da9c4-2c4f-4020-95ca-9f7ac7db7c00","Type":"ContainerStarted","Data":"ff421cade624e3040c2e3a6d95a3a67360d3e7573a262b61b37341aff4736dc0"} Oct 30 17:58:33.127113 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:33.123567 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9hrlx" event={"ID":"568da9c4-2c4f-4020-95ca-9f7ac7db7c00","Type":"ContainerStarted","Data":"cb04f2200d92b748e4083ed4eabdb59d808ef1791fa1494ccb2b5de5348816bd"} Oct 30 17:58:33.127113 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:33.123602 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9hrlx" event={"ID":"568da9c4-2c4f-4020-95ca-9f7ac7db7c00","Type":"ContainerStarted","Data":"1faf4187d8b069500e2a708665faf6d57f2a86bedc2e98f6a9d6f83e0196b448"} Oct 30 17:58:35.297291 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:35.297260 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-52rn6" Oct 30 17:58:37.110244 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:37.110205 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4qz5q" Oct 30 17:58:38.339476 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:38.339444 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-59dcddb9c7-k7vz2"] Oct 30 17:58:38.342414 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:38.342384 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59dcddb9c7-k7vz2" Oct 30 17:58:38.383871 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:38.383841 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 17:58:38.384028 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:38.383887 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 17:58:38.395439 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:38.395417 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 17:58:38.395439 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:38.395436 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 17:58:38.395643 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:38.395486 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zdsf4\"" Oct 30 17:58:38.396263 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:38.396244 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59dcddb9c7-k7vz2"] Oct 30 17:58:38.482631 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:38.482595 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5922c57f-e33c-4af5-a015-3c010e6a661f-registry-certificates\") pod \"image-registry-59dcddb9c7-k7vz2\" (UID: \"5922c57f-e33c-4af5-a015-3c010e6a661f\") " pod="openshift-image-registry/image-registry-59dcddb9c7-k7vz2" Oct 30 17:58:38.482631 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:38.482627 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5922c57f-e33c-4af5-a015-3c010e6a661f-bound-sa-token\") pod \"image-registry-59dcddb9c7-k7vz2\" (UID: \"5922c57f-e33c-4af5-a015-3c010e6a661f\") " pod="openshift-image-registry/image-registry-59dcddb9c7-k7vz2" Oct 30 17:58:38.482832 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:38.482659 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5922c57f-e33c-4af5-a015-3c010e6a661f-installation-pull-secrets\") pod \"image-registry-59dcddb9c7-k7vz2\" (UID: \"5922c57f-e33c-4af5-a015-3c010e6a661f\") " pod="openshift-image-registry/image-registry-59dcddb9c7-k7vz2" Oct 30 17:58:38.482832 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:38.482751 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5922c57f-e33c-4af5-a015-3c010e6a661f-trusted-ca\") pod \"image-registry-59dcddb9c7-k7vz2\" (UID: \"5922c57f-e33c-4af5-a015-3c010e6a661f\") " pod="openshift-image-registry/image-registry-59dcddb9c7-k7vz2" Oct 30 17:58:38.482832 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:38.482782 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxmxn\" (UniqueName: \"kubernetes.io/projected/5922c57f-e33c-4af5-a015-3c010e6a661f-kube-api-access-xxmxn\") pod \"image-registry-59dcddb9c7-k7vz2\" (UID: \"5922c57f-e33c-4af5-a015-3c010e6a661f\") " pod="openshift-image-registry/image-registry-59dcddb9c7-k7vz2" Oct 30 17:58:38.482832 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:38.482801 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5922c57f-e33c-4af5-a015-3c010e6a661f-ca-trust-extracted\") pod \"image-registry-59dcddb9c7-k7vz2\" (UID: \"5922c57f-e33c-4af5-a015-3c010e6a661f\") " pod="openshift-image-registry/image-registry-59dcddb9c7-k7vz2" Oct 30 17:58:38.482959 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:38.482844 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5922c57f-e33c-4af5-a015-3c010e6a661f-registry-tls\") pod \"image-registry-59dcddb9c7-k7vz2\" (UID: \"5922c57f-e33c-4af5-a015-3c010e6a661f\") " pod="openshift-image-registry/image-registry-59dcddb9c7-k7vz2" Oct 30 17:58:38.482991 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:38.482959 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5922c57f-e33c-4af5-a015-3c010e6a661f-image-registry-private-configuration\") pod \"image-registry-59dcddb9c7-k7vz2\" (UID: \"5922c57f-e33c-4af5-a015-3c010e6a661f\") " pod="openshift-image-registry/image-registry-59dcddb9c7-k7vz2" Oct 30 17:58:38.584353 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:38.584312 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5922c57f-e33c-4af5-a015-3c010e6a661f-trusted-ca\") pod \"image-registry-59dcddb9c7-k7vz2\" (UID: \"5922c57f-e33c-4af5-a015-3c010e6a661f\") " pod="openshift-image-registry/image-registry-59dcddb9c7-k7vz2" Oct 30 17:58:38.584353 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:38.584356 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxmxn\" (UniqueName: \"kubernetes.io/projected/5922c57f-e33c-4af5-a015-3c010e6a661f-kube-api-access-xxmxn\") pod \"image-registry-59dcddb9c7-k7vz2\" (UID: \"5922c57f-e33c-4af5-a015-3c010e6a661f\") " pod="openshift-image-registry/image-registry-59dcddb9c7-k7vz2" Oct 30 17:58:38.584594 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:38.584376 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5922c57f-e33c-4af5-a015-3c010e6a661f-ca-trust-extracted\") pod \"image-registry-59dcddb9c7-k7vz2\" (UID: \"5922c57f-e33c-4af5-a015-3c010e6a661f\") " pod="openshift-image-registry/image-registry-59dcddb9c7-k7vz2" Oct 30 17:58:38.584594 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:38.584431 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5922c57f-e33c-4af5-a015-3c010e6a661f-registry-tls\") pod \"image-registry-59dcddb9c7-k7vz2\" (UID: \"5922c57f-e33c-4af5-a015-3c010e6a661f\") " pod="openshift-image-registry/image-registry-59dcddb9c7-k7vz2" Oct 30 17:58:38.584594 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:38.584465 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5922c57f-e33c-4af5-a015-3c010e6a661f-image-registry-private-configuration\") pod \"image-registry-59dcddb9c7-k7vz2\" (UID: \"5922c57f-e33c-4af5-a015-3c010e6a661f\") " pod="openshift-image-registry/image-registry-59dcddb9c7-k7vz2" Oct 30 17:58:38.584594 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:38.584490 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5922c57f-e33c-4af5-a015-3c010e6a661f-registry-certificates\") pod \"image-registry-59dcddb9c7-k7vz2\" (UID: \"5922c57f-e33c-4af5-a015-3c010e6a661f\") " pod="openshift-image-registry/image-registry-59dcddb9c7-k7vz2" Oct 30 17:58:38.584594 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:38.584505 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5922c57f-e33c-4af5-a015-3c010e6a661f-bound-sa-token\") pod \"image-registry-59dcddb9c7-k7vz2\" (UID: \"5922c57f-e33c-4af5-a015-3c010e6a661f\") " pod="openshift-image-registry/image-registry-59dcddb9c7-k7vz2" Oct 30 17:58:38.584594 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:38.584557 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5922c57f-e33c-4af5-a015-3c010e6a661f-installation-pull-secrets\") pod \"image-registry-59dcddb9c7-k7vz2\" (UID: \"5922c57f-e33c-4af5-a015-3c010e6a661f\") " pod="openshift-image-registry/image-registry-59dcddb9c7-k7vz2" Oct 30 17:58:38.584987 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:38.584961 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5922c57f-e33c-4af5-a015-3c010e6a661f-ca-trust-extracted\") pod \"image-registry-59dcddb9c7-k7vz2\" (UID: \"5922c57f-e33c-4af5-a015-3c010e6a661f\") " pod="openshift-image-registry/image-registry-59dcddb9c7-k7vz2" Oct 30 17:58:38.585507 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:38.585478 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5922c57f-e33c-4af5-a015-3c010e6a661f-registry-certificates\") pod \"image-registry-59dcddb9c7-k7vz2\" (UID: \"5922c57f-e33c-4af5-a015-3c010e6a661f\") " pod="openshift-image-registry/image-registry-59dcddb9c7-k7vz2" Oct 30 17:58:38.585611 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:38.585572 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5922c57f-e33c-4af5-a015-3c010e6a661f-trusted-ca\") pod \"image-registry-59dcddb9c7-k7vz2\" (UID: \"5922c57f-e33c-4af5-a015-3c010e6a661f\") " pod="openshift-image-registry/image-registry-59dcddb9c7-k7vz2" Oct 30 17:58:38.587168 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:38.587122 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5922c57f-e33c-4af5-a015-3c010e6a661f-registry-tls\") pod \"image-registry-59dcddb9c7-k7vz2\" (UID: \"5922c57f-e33c-4af5-a015-3c010e6a661f\") " pod="openshift-image-registry/image-registry-59dcddb9c7-k7vz2" Oct 30 17:58:38.587273 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:38.587258 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5922c57f-e33c-4af5-a015-3c010e6a661f-installation-pull-secrets\") pod \"image-registry-59dcddb9c7-k7vz2\" (UID: \"5922c57f-e33c-4af5-a015-3c010e6a661f\") " pod="openshift-image-registry/image-registry-59dcddb9c7-k7vz2" Oct 30 17:58:38.587330 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:38.587272 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5922c57f-e33c-4af5-a015-3c010e6a661f-image-registry-private-configuration\") pod \"image-registry-59dcddb9c7-k7vz2\" (UID: \"5922c57f-e33c-4af5-a015-3c010e6a661f\") " pod="openshift-image-registry/image-registry-59dcddb9c7-k7vz2" Oct 30 17:58:38.603683 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:38.603600 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5922c57f-e33c-4af5-a015-3c010e6a661f-bound-sa-token\") pod \"image-registry-59dcddb9c7-k7vz2\" (UID: \"5922c57f-e33c-4af5-a015-3c010e6a661f\") " pod="openshift-image-registry/image-registry-59dcddb9c7-k7vz2" Oct 30 17:58:38.603814 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:38.603732 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxmxn\" (UniqueName: \"kubernetes.io/projected/5922c57f-e33c-4af5-a015-3c010e6a661f-kube-api-access-xxmxn\") pod \"image-registry-59dcddb9c7-k7vz2\" (UID: \"5922c57f-e33c-4af5-a015-3c010e6a661f\") " pod="openshift-image-registry/image-registry-59dcddb9c7-k7vz2" Oct 30 17:58:38.652747 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:38.652712 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59dcddb9c7-k7vz2" Oct 30 17:58:38.973496 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:38.973458 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59dcddb9c7-k7vz2"] Oct 30 17:58:38.979692 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:38.979667 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5922c57f_e33c_4af5_a015_3c010e6a661f.slice/crio-f2339fe92a2bf0523a7f7b61fb3127e3e36f31fe315421ead7f8890adc0e0a81 WatchSource:0}: Error finding container f2339fe92a2bf0523a7f7b61fb3127e3e36f31fe315421ead7f8890adc0e0a81: Status 404 returned error can't find the container with id f2339fe92a2bf0523a7f7b61fb3127e3e36f31fe315421ead7f8890adc0e0a81 Oct 30 17:58:39.143523 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:39.143483 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59dcddb9c7-k7vz2" event={"ID":"5922c57f-e33c-4af5-a015-3c010e6a661f","Type":"ContainerStarted","Data":"ebc6544a228f823f66d8f78d5c6a7f5c9de0a1ec8cd0d7b7b084c8a272d33636"} Oct 30 17:58:39.143705 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:39.143542 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59dcddb9c7-k7vz2" event={"ID":"5922c57f-e33c-4af5-a015-3c010e6a661f","Type":"ContainerStarted","Data":"f2339fe92a2bf0523a7f7b61fb3127e3e36f31fe315421ead7f8890adc0e0a81"} Oct 30 17:58:39.143705 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:39.143588 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-59dcddb9c7-k7vz2" Oct 30 17:58:39.192120 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:39.192065 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-59dcddb9c7-k7vz2" podStartSLOduration=1.192051805 podStartE2EDuration="1.192051805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:39.191755593 +0000 UTC m=+16.281817933" watchObservedRunningTime="2026-10-30 17:58:39.192051805 +0000 UTC m=+16.282114141" Oct 30 17:58:39.893838 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:39.893796 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vlnv2\" (UID: \"245afcc3-9c86-4c4e-ac8f-9f7d427c5920\") " pod="openshift-insights/insights-runtime-extractor-vlnv2" Oct 30 17:58:39.896288 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:39.896271 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/245afcc3-9c86-4c4e-ac8f-9f7d427c5920-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vlnv2\" (UID: \"245afcc3-9c86-4c4e-ac8f-9f7d427c5920\") " pod="openshift-insights/insights-runtime-extractor-vlnv2" Oct 30 17:58:39.956941 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:39.956911 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vlnv2" Oct 30 17:58:40.247056 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:40.246984 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vlnv2"] Oct 30 17:58:40.252217 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:40.252183 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod245afcc3_9c86_4c4e_ac8f_9f7d427c5920.slice/crio-e20e65626a2a4b71c18e68d6a32ca13df3b35ce742ecae5db593eede3360fa1a WatchSource:0}: Error finding container e20e65626a2a4b71c18e68d6a32ca13df3b35ce742ecae5db593eede3360fa1a: Status 404 returned error can't find the container with id e20e65626a2a4b71c18e68d6a32ca13df3b35ce742ecae5db593eede3360fa1a Oct 30 17:58:40.966883 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:40.966851 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-nvgn2"] Oct 30 17:58:40.969912 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:40.969892 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-nvgn2" Oct 30 17:58:40.995857 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:40.995836 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-wmxnf\"" Oct 30 17:58:40.996120 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:40.996104 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 17:58:40.996206 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:40.996194 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 17:58:40.996246 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:40.996211 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 17:58:40.996399 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:40.996383 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 17:58:40.996463 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:40.996451 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 17:58:40.998538 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:40.998522 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-nvgn2"] Oct 30 17:58:41.102748 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:41.102715 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3ae3331e-2881-427e-8685-2aa0473590bd-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-nvgn2\" (UID: \"3ae3331e-2881-427e-8685-2aa0473590bd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nvgn2" Oct 30 17:58:41.102748 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:41.102747 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ae3331e-2881-427e-8685-2aa0473590bd-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-nvgn2\" (UID: \"3ae3331e-2881-427e-8685-2aa0473590bd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nvgn2" Oct 30 17:58:41.102959 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:41.102770 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkqz8\" (UniqueName: \"kubernetes.io/projected/3ae3331e-2881-427e-8685-2aa0473590bd-kube-api-access-tkqz8\") pod \"prometheus-operator-9cd89b74c-nvgn2\" (UID: \"3ae3331e-2881-427e-8685-2aa0473590bd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nvgn2" Oct 30 17:58:41.102959 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:41.102814 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ae3331e-2881-427e-8685-2aa0473590bd-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-nvgn2\" (UID: \"3ae3331e-2881-427e-8685-2aa0473590bd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nvgn2" Oct 30 17:58:41.154818 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:41.154739 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vlnv2" event={"ID":"245afcc3-9c86-4c4e-ac8f-9f7d427c5920","Type":"ContainerStarted","Data":"0a9438df6ce2549ecc48e2bd43e08ae56efd139c2558e5b792efa51dec85c007"} Oct 30 17:58:41.154818 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:41.154811 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vlnv2" event={"ID":"245afcc3-9c86-4c4e-ac8f-9f7d427c5920","Type":"ContainerStarted","Data":"140678d240c1c589ce6429ad8240391bfa1d93a06052e71714ec43ed8cb65bd4"} Oct 30 17:58:41.154979 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:41.154830 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vlnv2" event={"ID":"245afcc3-9c86-4c4e-ac8f-9f7d427c5920","Type":"ContainerStarted","Data":"e20e65626a2a4b71c18e68d6a32ca13df3b35ce742ecae5db593eede3360fa1a"} Oct 30 17:58:41.203732 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:41.203705 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3ae3331e-2881-427e-8685-2aa0473590bd-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-nvgn2\" (UID: \"3ae3331e-2881-427e-8685-2aa0473590bd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nvgn2" Oct 30 17:58:41.203792 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:41.203740 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ae3331e-2881-427e-8685-2aa0473590bd-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-nvgn2\" (UID: \"3ae3331e-2881-427e-8685-2aa0473590bd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nvgn2" Oct 30 17:58:41.203792 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:41.203770 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkqz8\" (UniqueName: \"kubernetes.io/projected/3ae3331e-2881-427e-8685-2aa0473590bd-kube-api-access-tkqz8\") pod \"prometheus-operator-9cd89b74c-nvgn2\" (UID: \"3ae3331e-2881-427e-8685-2aa0473590bd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nvgn2" Oct 30 17:58:41.203870 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:41.203798 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ae3331e-2881-427e-8685-2aa0473590bd-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-nvgn2\" (UID: \"3ae3331e-2881-427e-8685-2aa0473590bd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nvgn2" Oct 30 17:58:41.204536 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:41.204510 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ae3331e-2881-427e-8685-2aa0473590bd-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-nvgn2\" (UID: \"3ae3331e-2881-427e-8685-2aa0473590bd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nvgn2" Oct 30 17:58:41.206311 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:41.206291 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3ae3331e-2881-427e-8685-2aa0473590bd-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-nvgn2\" (UID: \"3ae3331e-2881-427e-8685-2aa0473590bd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nvgn2" Oct 30 17:58:41.206404 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:41.206320 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ae3331e-2881-427e-8685-2aa0473590bd-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-nvgn2\" (UID: \"3ae3331e-2881-427e-8685-2aa0473590bd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nvgn2" Oct 30 17:58:41.224204 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:41.224168 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkqz8\" (UniqueName: \"kubernetes.io/projected/3ae3331e-2881-427e-8685-2aa0473590bd-kube-api-access-tkqz8\") pod \"prometheus-operator-9cd89b74c-nvgn2\" (UID: \"3ae3331e-2881-427e-8685-2aa0473590bd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nvgn2" Oct 30 17:58:41.282792 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:41.282751 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-nvgn2" Oct 30 17:58:41.573021 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:41.572945 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-nvgn2"] Oct 30 17:58:41.578014 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:41.577980 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ae3331e_2881_427e_8685_2aa0473590bd.slice/crio-d5e749f026f926687766fb574d05ff70cedaf3ad7d5f31b6b09aca6c00e4778e WatchSource:0}: Error finding container d5e749f026f926687766fb574d05ff70cedaf3ad7d5f31b6b09aca6c00e4778e: Status 404 returned error can't find the container with id d5e749f026f926687766fb574d05ff70cedaf3ad7d5f31b6b09aca6c00e4778e Oct 30 17:58:42.161712 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:42.161670 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-nvgn2" event={"ID":"3ae3331e-2881-427e-8685-2aa0473590bd","Type":"ContainerStarted","Data":"d5e749f026f926687766fb574d05ff70cedaf3ad7d5f31b6b09aca6c00e4778e"} Oct 30 17:58:43.166093 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:43.166054 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-nvgn2" event={"ID":"3ae3331e-2881-427e-8685-2aa0473590bd","Type":"ContainerStarted","Data":"7309c905bdd7e4930762bbec33bebcf195fa44beab830c34f8b761894a1462d6"} Oct 30 17:58:43.166585 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:43.166104 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-nvgn2" event={"ID":"3ae3331e-2881-427e-8685-2aa0473590bd","Type":"ContainerStarted","Data":"11e4353b2e8246d0e6a4ceb8962f55d721b415abe1413a3afd458fe5899d5d93"} Oct 30 17:58:43.168338 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:43.168314 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vlnv2" event={"ID":"245afcc3-9c86-4c4e-ac8f-9f7d427c5920","Type":"ContainerStarted","Data":"32796955b34ac77c1e858f1f025524057001e57f8eb244521d2b2cd3faafd9d8"} Oct 30 17:58:43.205443 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:43.205394 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-nvgn2" podStartSLOduration=2.059369869 podStartE2EDuration="3.205382571s" totalImagesPullingTime="1.146012702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:58:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:43.204254266 +0000 UTC m=+20.294316607" watchObservedRunningTime="2026-10-30 17:58:43.205382571 +0000 UTC m=+20.295444910" Oct 30 17:58:45.606334 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.606302 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-cw2zd"] Oct 30 17:58:45.611249 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.611226 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cw2zd" Oct 30 17:58:45.618316 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.618290 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6zxkn"] Oct 30 17:58:45.621855 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.621837 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6zxkn" Oct 30 17:58:45.633105 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.633082 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-cw2zd"] Oct 30 17:58:45.635093 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.634897 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 17:58:45.635288 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.634912 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 17:58:45.635395 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.635079 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-xvzft\"" Oct 30 17:58:45.651006 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.650981 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 17:58:45.651182 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.651024 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 17:58:45.651182 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.651093 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rlt9d\"" Oct 30 17:58:45.674044 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.674018 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 17:58:45.674603 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.674586 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 17:58:45.735371 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.735335 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b6361df3-6ef7-42bc-8c50-c6f317d9d46d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-cw2zd\" (UID: \"b6361df3-6ef7-42bc-8c50-c6f317d9d46d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cw2zd" Oct 30 17:58:45.735371 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.735368 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsb24\" (UniqueName: \"kubernetes.io/projected/b6361df3-6ef7-42bc-8c50-c6f317d9d46d-kube-api-access-lsb24\") pod \"kube-state-metrics-5dc8d7d89b-cw2zd\" (UID: \"b6361df3-6ef7-42bc-8c50-c6f317d9d46d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cw2zd" Oct 30 17:58:45.735598 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.735387 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/29e7a0a9-5a42-47b2-bf89-eccde79a39fd-root\") pod \"node-exporter-6zxkn\" (UID: \"29e7a0a9-5a42-47b2-bf89-eccde79a39fd\") " pod="openshift-monitoring/node-exporter-6zxkn" Oct 30 17:58:45.735598 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.735407 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b6361df3-6ef7-42bc-8c50-c6f317d9d46d-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-cw2zd\" (UID: \"b6361df3-6ef7-42bc-8c50-c6f317d9d46d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cw2zd" Oct 30 17:58:45.735598 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.735485 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29e7a0a9-5a42-47b2-bf89-eccde79a39fd-metrics-client-ca\") pod \"node-exporter-6zxkn\" (UID: \"29e7a0a9-5a42-47b2-bf89-eccde79a39fd\") " pod="openshift-monitoring/node-exporter-6zxkn" Oct 30 17:58:45.735598 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.735580 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b6361df3-6ef7-42bc-8c50-c6f317d9d46d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-cw2zd\" (UID: \"b6361df3-6ef7-42bc-8c50-c6f317d9d46d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cw2zd" Oct 30 17:58:45.735752 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.735627 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/29e7a0a9-5a42-47b2-bf89-eccde79a39fd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6zxkn\" (UID: \"29e7a0a9-5a42-47b2-bf89-eccde79a39fd\") " pod="openshift-monitoring/node-exporter-6zxkn" Oct 30 17:58:45.735752 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.735699 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/29e7a0a9-5a42-47b2-bf89-eccde79a39fd-sys\") pod \"node-exporter-6zxkn\" (UID: \"29e7a0a9-5a42-47b2-bf89-eccde79a39fd\") " pod="openshift-monitoring/node-exporter-6zxkn" Oct 30 17:58:45.735752 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.735719 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6361df3-6ef7-42bc-8c50-c6f317d9d46d-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-cw2zd\" (UID: \"b6361df3-6ef7-42bc-8c50-c6f317d9d46d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cw2zd" Oct 30 17:58:45.735752 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.735743 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/29e7a0a9-5a42-47b2-bf89-eccde79a39fd-node-exporter-accelerators-collector-config\") pod \"node-exporter-6zxkn\" (UID: \"29e7a0a9-5a42-47b2-bf89-eccde79a39fd\") " pod="openshift-monitoring/node-exporter-6zxkn" Oct 30 17:58:45.735891 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.735767 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52dqg\" (UniqueName: \"kubernetes.io/projected/29e7a0a9-5a42-47b2-bf89-eccde79a39fd-kube-api-access-52dqg\") pod \"node-exporter-6zxkn\" (UID: \"29e7a0a9-5a42-47b2-bf89-eccde79a39fd\") " pod="openshift-monitoring/node-exporter-6zxkn" Oct 30 17:58:45.735891 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.735823 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/29e7a0a9-5a42-47b2-bf89-eccde79a39fd-node-exporter-tls\") pod \"node-exporter-6zxkn\" (UID: \"29e7a0a9-5a42-47b2-bf89-eccde79a39fd\") " pod="openshift-monitoring/node-exporter-6zxkn" Oct 30 17:58:45.735891 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.735841 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6361df3-6ef7-42bc-8c50-c6f317d9d46d-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-cw2zd\" (UID: \"b6361df3-6ef7-42bc-8c50-c6f317d9d46d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cw2zd" Oct 30 17:58:45.735994 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.735899 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/29e7a0a9-5a42-47b2-bf89-eccde79a39fd-node-exporter-textfile\") pod \"node-exporter-6zxkn\" (UID: \"29e7a0a9-5a42-47b2-bf89-eccde79a39fd\") " pod="openshift-monitoring/node-exporter-6zxkn" Oct 30 17:58:45.735994 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.735917 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/29e7a0a9-5a42-47b2-bf89-eccde79a39fd-node-exporter-wtmp\") pod \"node-exporter-6zxkn\" (UID: \"29e7a0a9-5a42-47b2-bf89-eccde79a39fd\") " pod="openshift-monitoring/node-exporter-6zxkn" Oct 30 17:58:45.836668 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.836633 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52dqg\" (UniqueName: \"kubernetes.io/projected/29e7a0a9-5a42-47b2-bf89-eccde79a39fd-kube-api-access-52dqg\") pod \"node-exporter-6zxkn\" (UID: \"29e7a0a9-5a42-47b2-bf89-eccde79a39fd\") " pod="openshift-monitoring/node-exporter-6zxkn" Oct 30 17:58:45.836668 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.836667 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/29e7a0a9-5a42-47b2-bf89-eccde79a39fd-node-exporter-tls\") pod \"node-exporter-6zxkn\" (UID: \"29e7a0a9-5a42-47b2-bf89-eccde79a39fd\") " pod="openshift-monitoring/node-exporter-6zxkn" Oct 30 17:58:45.836890 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.836686 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6361df3-6ef7-42bc-8c50-c6f317d9d46d-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-cw2zd\" (UID: \"b6361df3-6ef7-42bc-8c50-c6f317d9d46d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cw2zd" Oct 30 17:58:45.836890 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.836820 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/29e7a0a9-5a42-47b2-bf89-eccde79a39fd-node-exporter-textfile\") pod \"node-exporter-6zxkn\" (UID: \"29e7a0a9-5a42-47b2-bf89-eccde79a39fd\") " pod="openshift-monitoring/node-exporter-6zxkn" Oct 30 17:58:45.836890 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.836856 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/29e7a0a9-5a42-47b2-bf89-eccde79a39fd-node-exporter-wtmp\") pod \"node-exporter-6zxkn\" (UID: \"29e7a0a9-5a42-47b2-bf89-eccde79a39fd\") " pod="openshift-monitoring/node-exporter-6zxkn" Oct 30 17:58:45.836890 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.836888 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b6361df3-6ef7-42bc-8c50-c6f317d9d46d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-cw2zd\" (UID: \"b6361df3-6ef7-42bc-8c50-c6f317d9d46d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cw2zd" Oct 30 17:58:45.837086 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.836912 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsb24\" (UniqueName: \"kubernetes.io/projected/b6361df3-6ef7-42bc-8c50-c6f317d9d46d-kube-api-access-lsb24\") pod \"kube-state-metrics-5dc8d7d89b-cw2zd\" (UID: \"b6361df3-6ef7-42bc-8c50-c6f317d9d46d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cw2zd" Oct 30 17:58:45.837086 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.836934 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/29e7a0a9-5a42-47b2-bf89-eccde79a39fd-root\") pod \"node-exporter-6zxkn\" (UID: \"29e7a0a9-5a42-47b2-bf89-eccde79a39fd\") " pod="openshift-monitoring/node-exporter-6zxkn" Oct 30 17:58:45.837086 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.836948 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b6361df3-6ef7-42bc-8c50-c6f317d9d46d-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-cw2zd\" (UID: \"b6361df3-6ef7-42bc-8c50-c6f317d9d46d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cw2zd" Oct 30 17:58:45.837086 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.836967 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29e7a0a9-5a42-47b2-bf89-eccde79a39fd-metrics-client-ca\") pod \"node-exporter-6zxkn\" (UID: \"29e7a0a9-5a42-47b2-bf89-eccde79a39fd\") " pod="openshift-monitoring/node-exporter-6zxkn" Oct 30 17:58:45.837086 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.836993 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b6361df3-6ef7-42bc-8c50-c6f317d9d46d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-cw2zd\" (UID: \"b6361df3-6ef7-42bc-8c50-c6f317d9d46d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cw2zd" Oct 30 17:58:45.837086 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.837042 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/29e7a0a9-5a42-47b2-bf89-eccde79a39fd-node-exporter-wtmp\") pod \"node-exporter-6zxkn\" (UID: \"29e7a0a9-5a42-47b2-bf89-eccde79a39fd\") " pod="openshift-monitoring/node-exporter-6zxkn" Oct 30 17:58:45.837086 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.837064 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/29e7a0a9-5a42-47b2-bf89-eccde79a39fd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6zxkn\" (UID: \"29e7a0a9-5a42-47b2-bf89-eccde79a39fd\") " pod="openshift-monitoring/node-exporter-6zxkn" Oct 30 17:58:45.837449 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.837098 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/29e7a0a9-5a42-47b2-bf89-eccde79a39fd-sys\") pod \"node-exporter-6zxkn\" (UID: \"29e7a0a9-5a42-47b2-bf89-eccde79a39fd\") " pod="openshift-monitoring/node-exporter-6zxkn" Oct 30 17:58:45.837449 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.837127 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6361df3-6ef7-42bc-8c50-c6f317d9d46d-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-cw2zd\" (UID: \"b6361df3-6ef7-42bc-8c50-c6f317d9d46d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cw2zd" Oct 30 17:58:45.837449 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.837192 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/29e7a0a9-5a42-47b2-bf89-eccde79a39fd-node-exporter-accelerators-collector-config\") pod \"node-exporter-6zxkn\" (UID: \"29e7a0a9-5a42-47b2-bf89-eccde79a39fd\") " pod="openshift-monitoring/node-exporter-6zxkn" Oct 30 17:58:45.837449 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.837290 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/29e7a0a9-5a42-47b2-bf89-eccde79a39fd-node-exporter-textfile\") pod \"node-exporter-6zxkn\" (UID: \"29e7a0a9-5a42-47b2-bf89-eccde79a39fd\") " pod="openshift-monitoring/node-exporter-6zxkn" Oct 30 17:58:45.837638 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.837594 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b6361df3-6ef7-42bc-8c50-c6f317d9d46d-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-cw2zd\" (UID: \"b6361df3-6ef7-42bc-8c50-c6f317d9d46d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cw2zd" Oct 30 17:58:45.837695 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.837633 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/29e7a0a9-5a42-47b2-bf89-eccde79a39fd-sys\") pod \"node-exporter-6zxkn\" (UID: \"29e7a0a9-5a42-47b2-bf89-eccde79a39fd\") " pod="openshift-monitoring/node-exporter-6zxkn" Oct 30 17:58:45.837695 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.837042 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/29e7a0a9-5a42-47b2-bf89-eccde79a39fd-root\") pod \"node-exporter-6zxkn\" (UID: \"29e7a0a9-5a42-47b2-bf89-eccde79a39fd\") " pod="openshift-monitoring/node-exporter-6zxkn" Oct 30 17:58:45.837776 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.837694 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29e7a0a9-5a42-47b2-bf89-eccde79a39fd-metrics-client-ca\") pod \"node-exporter-6zxkn\" (UID: \"29e7a0a9-5a42-47b2-bf89-eccde79a39fd\") " pod="openshift-monitoring/node-exporter-6zxkn" Oct 30 17:58:45.837776 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.837716 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b6361df3-6ef7-42bc-8c50-c6f317d9d46d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-cw2zd\" (UID: \"b6361df3-6ef7-42bc-8c50-c6f317d9d46d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cw2zd" Oct 30 17:58:45.838178 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.838154 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/29e7a0a9-5a42-47b2-bf89-eccde79a39fd-node-exporter-accelerators-collector-config\") pod \"node-exporter-6zxkn\" (UID: \"29e7a0a9-5a42-47b2-bf89-eccde79a39fd\") " pod="openshift-monitoring/node-exporter-6zxkn" Oct 30 17:58:45.838493 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.838465 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6361df3-6ef7-42bc-8c50-c6f317d9d46d-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-cw2zd\" (UID: \"b6361df3-6ef7-42bc-8c50-c6f317d9d46d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cw2zd" Oct 30 17:58:45.839551 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.839524 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/29e7a0a9-5a42-47b2-bf89-eccde79a39fd-node-exporter-tls\") pod \"node-exporter-6zxkn\" (UID: \"29e7a0a9-5a42-47b2-bf89-eccde79a39fd\") " pod="openshift-monitoring/node-exporter-6zxkn" Oct 30 17:58:45.839622 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.839604 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b6361df3-6ef7-42bc-8c50-c6f317d9d46d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-cw2zd\" (UID: \"b6361df3-6ef7-42bc-8c50-c6f317d9d46d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cw2zd" Oct 30 17:58:45.839988 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.839969 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6361df3-6ef7-42bc-8c50-c6f317d9d46d-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-cw2zd\" (UID: \"b6361df3-6ef7-42bc-8c50-c6f317d9d46d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cw2zd" Oct 30 17:58:45.840024 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.839978 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/29e7a0a9-5a42-47b2-bf89-eccde79a39fd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6zxkn\" (UID: \"29e7a0a9-5a42-47b2-bf89-eccde79a39fd\") " pod="openshift-monitoring/node-exporter-6zxkn" Oct 30 17:58:45.855062 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.855041 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsb24\" (UniqueName: \"kubernetes.io/projected/b6361df3-6ef7-42bc-8c50-c6f317d9d46d-kube-api-access-lsb24\") pod \"kube-state-metrics-5dc8d7d89b-cw2zd\" (UID: \"b6361df3-6ef7-42bc-8c50-c6f317d9d46d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cw2zd" Oct 30 17:58:45.855204 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.855187 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52dqg\" (UniqueName: \"kubernetes.io/projected/29e7a0a9-5a42-47b2-bf89-eccde79a39fd-kube-api-access-52dqg\") pod \"node-exporter-6zxkn\" (UID: \"29e7a0a9-5a42-47b2-bf89-eccde79a39fd\") " pod="openshift-monitoring/node-exporter-6zxkn" Oct 30 17:58:45.928182 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.928153 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cw2zd" Oct 30 17:58:45.937675 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:45.937656 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6zxkn" Oct 30 17:58:45.951726 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:45.951701 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29e7a0a9_5a42_47b2_bf89_eccde79a39fd.slice/crio-d4732b55d86fd659b8ff8d4500e6fd33f53c2295639c19d60ddb7899634192e6 WatchSource:0}: Error finding container d4732b55d86fd659b8ff8d4500e6fd33f53c2295639c19d60ddb7899634192e6: Status 404 returned error can't find the container with id d4732b55d86fd659b8ff8d4500e6fd33f53c2295639c19d60ddb7899634192e6 Oct 30 17:58:46.179675 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:46.179584 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6zxkn" event={"ID":"29e7a0a9-5a42-47b2-bf89-eccde79a39fd","Type":"ContainerStarted","Data":"d4732b55d86fd659b8ff8d4500e6fd33f53c2295639c19d60ddb7899634192e6"} Oct 30 17:58:46.236374 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:46.236337 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-cw2zd"] Oct 30 17:58:46.242433 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:46.242405 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6361df3_6ef7_42bc_8c50_c6f317d9d46d.slice/crio-314ebafeb01b0fa2f32cebc7a934ce23417b9bb7958567987ef5d219b86555fd WatchSource:0}: Error finding container 314ebafeb01b0fa2f32cebc7a934ce23417b9bb7958567987ef5d219b86555fd: Status 404 returned error can't find the container with id 314ebafeb01b0fa2f32cebc7a934ce23417b9bb7958567987ef5d219b86555fd Oct 30 17:58:47.189782 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:47.189746 4442 generic.go:356] "Generic (PLEG): container finished" podID="29e7a0a9-5a42-47b2-bf89-eccde79a39fd" containerID="b96d3f98e742378353f51b2807bc2ced1c357e85132beb05db5e0e802f528f11" exitCode=0 Oct 30 17:58:47.190292 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:47.189844 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6zxkn" event={"ID":"29e7a0a9-5a42-47b2-bf89-eccde79a39fd","Type":"ContainerDied","Data":"b96d3f98e742378353f51b2807bc2ced1c357e85132beb05db5e0e802f528f11"} Oct 30 17:58:47.191196 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:47.191176 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cw2zd" event={"ID":"b6361df3-6ef7-42bc-8c50-c6f317d9d46d","Type":"ContainerStarted","Data":"314ebafeb01b0fa2f32cebc7a934ce23417b9bb7958567987ef5d219b86555fd"} Oct 30 17:58:47.564623 ip-10-0-138-176 kubenswrapper[4442]: E1030 17:58:47.564589 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 17:58:48.196869 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:48.196833 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6zxkn" event={"ID":"29e7a0a9-5a42-47b2-bf89-eccde79a39fd","Type":"ContainerStarted","Data":"7a6ea5b3f43a398e1a2480a17dbcd6233633a589e77c6a1a29465cc5e7fb99bb"} Oct 30 17:58:48.197527 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:48.196888 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6zxkn" event={"ID":"29e7a0a9-5a42-47b2-bf89-eccde79a39fd","Type":"ContainerStarted","Data":"d4c561ab40f706b0d5fd241a48abd89abc8338fbf43cb0806eb86a0e67ea274f"} Oct 30 17:58:48.199549 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:48.199056 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cw2zd" event={"ID":"b6361df3-6ef7-42bc-8c50-c6f317d9d46d","Type":"ContainerStarted","Data":"13b57394019664dadf39e8f6a8385be6e3fa0ee18f0cdf78bbee4871c30cc5dd"} Oct 30 17:58:48.199549 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:48.199107 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cw2zd" event={"ID":"b6361df3-6ef7-42bc-8c50-c6f317d9d46d","Type":"ContainerStarted","Data":"86939e5e9e69c1a656fb27fb19b4a08d397dec0afef6017f86645430135f92c2"} Oct 30 17:58:48.199549 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:48.199200 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cw2zd" event={"ID":"b6361df3-6ef7-42bc-8c50-c6f317d9d46d","Type":"ContainerStarted","Data":"ab2d805376c844ab4bbd30285eacbda361efaf2700ad4fd9941f9bf78a2b821e"} Oct 30 17:58:48.261263 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:48.261215 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6zxkn" podStartSLOduration=2.366647589 podStartE2EDuration="3.261198677s" totalImagesPullingTime="810.151337ms" totalInitContainerRuntime="84.399751ms" isStatefulPod=true podCreationTimestamp="2026-10-30 17:58:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:48.260670125 +0000 UTC m=+25.350732464" watchObservedRunningTime="2026-10-30 17:58:48.261198677 +0000 UTC m=+25.351261017" Oct 30 17:58:48.300081 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:48.300032 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cw2zd" podStartSLOduration=2.014098222 podStartE2EDuration="3.300020237s" totalImagesPullingTime="1.285922015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:58:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:48.299585608 +0000 UTC m=+25.389647946" watchObservedRunningTime="2026-10-30 17:58:48.300020237 +0000 UTC m=+25.390082576" Oct 30 17:58:50.163276 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:50.163243 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-78c74bf4c6-prjr2"] Oct 30 17:58:50.168217 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:50.168189 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-78c74bf4c6-prjr2" Oct 30 17:58:50.183679 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:50.183655 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 17:58:50.194654 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:50.194630 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 17:58:50.194789 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:50.194718 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 17:58:50.194862 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:50.194814 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 17:58:50.194862 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:50.194825 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-t942w\"" Oct 30 17:58:50.195421 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:50.195404 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-84lfnknjq3jfj\"" Oct 30 17:58:50.196482 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:50.196461 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-78c74bf4c6-prjr2"] Oct 30 17:58:50.270209 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:50.270178 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9101c57f-0762-487c-9f61-c1efa4900965-secret-metrics-server-client-certs\") pod \"metrics-server-78c74bf4c6-prjr2\" (UID: \"9101c57f-0762-487c-9f61-c1efa4900965\") " pod="openshift-monitoring/metrics-server-78c74bf4c6-prjr2" Oct 30 17:58:50.270209 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:50.270209 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9101c57f-0762-487c-9f61-c1efa4900965-audit-log\") pod \"metrics-server-78c74bf4c6-prjr2\" (UID: \"9101c57f-0762-487c-9f61-c1efa4900965\") " pod="openshift-monitoring/metrics-server-78c74bf4c6-prjr2" Oct 30 17:58:50.270452 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:50.270230 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9101c57f-0762-487c-9f61-c1efa4900965-metrics-server-audit-profiles\") pod \"metrics-server-78c74bf4c6-prjr2\" (UID: \"9101c57f-0762-487c-9f61-c1efa4900965\") " pod="openshift-monitoring/metrics-server-78c74bf4c6-prjr2" Oct 30 17:58:50.270452 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:50.270255 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9101c57f-0762-487c-9f61-c1efa4900965-client-ca-bundle\") pod \"metrics-server-78c74bf4c6-prjr2\" (UID: \"9101c57f-0762-487c-9f61-c1efa4900965\") " pod="openshift-monitoring/metrics-server-78c74bf4c6-prjr2" Oct 30 17:58:50.270452 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:50.270390 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9101c57f-0762-487c-9f61-c1efa4900965-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-78c74bf4c6-prjr2\" (UID: \"9101c57f-0762-487c-9f61-c1efa4900965\") " pod="openshift-monitoring/metrics-server-78c74bf4c6-prjr2" Oct 30 17:58:50.270557 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:50.270457 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9101c57f-0762-487c-9f61-c1efa4900965-secret-metrics-server-tls\") pod \"metrics-server-78c74bf4c6-prjr2\" (UID: \"9101c57f-0762-487c-9f61-c1efa4900965\") " pod="openshift-monitoring/metrics-server-78c74bf4c6-prjr2" Oct 30 17:58:50.270557 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:50.270487 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-489pl\" (UniqueName: \"kubernetes.io/projected/9101c57f-0762-487c-9f61-c1efa4900965-kube-api-access-489pl\") pod \"metrics-server-78c74bf4c6-prjr2\" (UID: \"9101c57f-0762-487c-9f61-c1efa4900965\") " pod="openshift-monitoring/metrics-server-78c74bf4c6-prjr2" Oct 30 17:58:50.371431 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:50.371392 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9101c57f-0762-487c-9f61-c1efa4900965-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-78c74bf4c6-prjr2\" (UID: \"9101c57f-0762-487c-9f61-c1efa4900965\") " pod="openshift-monitoring/metrics-server-78c74bf4c6-prjr2" Oct 30 17:58:50.371431 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:50.371437 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9101c57f-0762-487c-9f61-c1efa4900965-secret-metrics-server-tls\") pod \"metrics-server-78c74bf4c6-prjr2\" (UID: \"9101c57f-0762-487c-9f61-c1efa4900965\") " pod="openshift-monitoring/metrics-server-78c74bf4c6-prjr2" Oct 30 17:58:50.371683 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:50.371551 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-489pl\" (UniqueName: \"kubernetes.io/projected/9101c57f-0762-487c-9f61-c1efa4900965-kube-api-access-489pl\") pod \"metrics-server-78c74bf4c6-prjr2\" (UID: \"9101c57f-0762-487c-9f61-c1efa4900965\") " pod="openshift-monitoring/metrics-server-78c74bf4c6-prjr2" Oct 30 17:58:50.371683 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:50.371598 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9101c57f-0762-487c-9f61-c1efa4900965-secret-metrics-server-client-certs\") pod \"metrics-server-78c74bf4c6-prjr2\" (UID: \"9101c57f-0762-487c-9f61-c1efa4900965\") " pod="openshift-monitoring/metrics-server-78c74bf4c6-prjr2" Oct 30 17:58:50.371683 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:50.371621 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9101c57f-0762-487c-9f61-c1efa4900965-audit-log\") pod \"metrics-server-78c74bf4c6-prjr2\" (UID: \"9101c57f-0762-487c-9f61-c1efa4900965\") " pod="openshift-monitoring/metrics-server-78c74bf4c6-prjr2" Oct 30 17:58:50.371683 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:50.371656 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9101c57f-0762-487c-9f61-c1efa4900965-metrics-server-audit-profiles\") pod \"metrics-server-78c74bf4c6-prjr2\" (UID: \"9101c57f-0762-487c-9f61-c1efa4900965\") " pod="openshift-monitoring/metrics-server-78c74bf4c6-prjr2" Oct 30 17:58:50.371893 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:50.371690 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9101c57f-0762-487c-9f61-c1efa4900965-client-ca-bundle\") pod \"metrics-server-78c74bf4c6-prjr2\" (UID: \"9101c57f-0762-487c-9f61-c1efa4900965\") " pod="openshift-monitoring/metrics-server-78c74bf4c6-prjr2" Oct 30 17:58:50.372106 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:50.372055 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9101c57f-0762-487c-9f61-c1efa4900965-audit-log\") pod \"metrics-server-78c74bf4c6-prjr2\" (UID: \"9101c57f-0762-487c-9f61-c1efa4900965\") " pod="openshift-monitoring/metrics-server-78c74bf4c6-prjr2" Oct 30 17:58:50.372426 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:50.372282 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9101c57f-0762-487c-9f61-c1efa4900965-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-78c74bf4c6-prjr2\" (UID: \"9101c57f-0762-487c-9f61-c1efa4900965\") " pod="openshift-monitoring/metrics-server-78c74bf4c6-prjr2" Oct 30 17:58:50.372750 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:50.372729 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9101c57f-0762-487c-9f61-c1efa4900965-metrics-server-audit-profiles\") pod \"metrics-server-78c74bf4c6-prjr2\" (UID: \"9101c57f-0762-487c-9f61-c1efa4900965\") " pod="openshift-monitoring/metrics-server-78c74bf4c6-prjr2" Oct 30 17:58:50.374189 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:50.374167 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9101c57f-0762-487c-9f61-c1efa4900965-secret-metrics-server-client-certs\") pod \"metrics-server-78c74bf4c6-prjr2\" (UID: \"9101c57f-0762-487c-9f61-c1efa4900965\") " pod="openshift-monitoring/metrics-server-78c74bf4c6-prjr2" Oct 30 17:58:50.374638 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:50.374618 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9101c57f-0762-487c-9f61-c1efa4900965-secret-metrics-server-tls\") pod \"metrics-server-78c74bf4c6-prjr2\" (UID: \"9101c57f-0762-487c-9f61-c1efa4900965\") " pod="openshift-monitoring/metrics-server-78c74bf4c6-prjr2" Oct 30 17:58:50.374727 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:50.374671 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9101c57f-0762-487c-9f61-c1efa4900965-client-ca-bundle\") pod \"metrics-server-78c74bf4c6-prjr2\" (UID: \"9101c57f-0762-487c-9f61-c1efa4900965\") " pod="openshift-monitoring/metrics-server-78c74bf4c6-prjr2" Oct 30 17:58:50.390256 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:50.390235 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-489pl\" (UniqueName: \"kubernetes.io/projected/9101c57f-0762-487c-9f61-c1efa4900965-kube-api-access-489pl\") pod \"metrics-server-78c74bf4c6-prjr2\" (UID: \"9101c57f-0762-487c-9f61-c1efa4900965\") " pod="openshift-monitoring/metrics-server-78c74bf4c6-prjr2" Oct 30 17:58:50.479657 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:50.479574 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-78c74bf4c6-prjr2" Oct 30 17:58:50.786481 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:50.786397 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-78c74bf4c6-prjr2"] Oct 30 17:58:50.791175 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:58:50.791147 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9101c57f_0762_487c_9f61_c1efa4900965.slice/crio-f0ad7c1c8dcd153770cc99756d2afd97085eb3832d62a8acb5384cc491e4c661 WatchSource:0}: Error finding container f0ad7c1c8dcd153770cc99756d2afd97085eb3832d62a8acb5384cc491e4c661: Status 404 returned error can't find the container with id f0ad7c1c8dcd153770cc99756d2afd97085eb3832d62a8acb5384cc491e4c661 Oct 30 17:58:51.210537 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:51.210499 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-78c74bf4c6-prjr2" event={"ID":"9101c57f-0762-487c-9f61-c1efa4900965","Type":"ContainerStarted","Data":"f0ad7c1c8dcd153770cc99756d2afd97085eb3832d62a8acb5384cc491e4c661"} Oct 30 17:58:52.216096 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:52.216066 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-78c74bf4c6-prjr2" event={"ID":"9101c57f-0762-487c-9f61-c1efa4900965","Type":"ContainerStarted","Data":"6100278dc93f144cca21707b2680a70ad9ff75c52701d116fcb38ea0621f56f5"} Oct 30 17:58:52.261993 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:52.261945 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-78c74bf4c6-prjr2" podStartSLOduration=0.962026679 podStartE2EDuration="2.261931541s" totalImagesPullingTime="1.299904862s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:58:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:52.261486003 +0000 UTC m=+29.351548357" watchObservedRunningTime="2026-10-30 17:58:52.261931541 +0000 UTC m=+29.351993935" Oct 30 17:58:57.315860 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:57.315829 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zpsg9" Oct 30 17:58:59.999727 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:58:59.999676 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6db45d6b4-7lt8m"] Oct 30 17:59:00.002815 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:00.002793 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6db45d6b4-7lt8m" Oct 30 17:59:00.018827 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:00.018803 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 17:59:00.018976 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:00.018956 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 17:59:00.019030 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:00.019013 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 17:59:00.019094 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:00.019079 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 17:59:00.039784 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:00.039752 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p4lt\" (UniqueName: \"kubernetes.io/projected/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-kube-api-access-4p4lt\") pod \"console-6db45d6b4-7lt8m\" (UID: \"333e0ba0-df2b-4d51-9ac2-ff464c8bc348\") " pod="openshift-console/console-6db45d6b4-7lt8m" Oct 30 17:59:00.039974 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:00.039797 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-console-serving-cert\") pod \"console-6db45d6b4-7lt8m\" (UID: \"333e0ba0-df2b-4d51-9ac2-ff464c8bc348\") " pod="openshift-console/console-6db45d6b4-7lt8m" Oct 30 17:59:00.039974 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:00.039919 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-console-oauth-config\") pod \"console-6db45d6b4-7lt8m\" (UID: \"333e0ba0-df2b-4d51-9ac2-ff464c8bc348\") " pod="openshift-console/console-6db45d6b4-7lt8m" Oct 30 17:59:00.039974 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:00.039955 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-trusted-ca-bundle\") pod \"console-6db45d6b4-7lt8m\" (UID: \"333e0ba0-df2b-4d51-9ac2-ff464c8bc348\") " pod="openshift-console/console-6db45d6b4-7lt8m" Oct 30 17:59:00.040110 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:00.039984 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-service-ca\") pod \"console-6db45d6b4-7lt8m\" (UID: \"333e0ba0-df2b-4d51-9ac2-ff464c8bc348\") " pod="openshift-console/console-6db45d6b4-7lt8m" Oct 30 17:59:00.040110 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:00.040069 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-console-config\") pod \"console-6db45d6b4-7lt8m\" (UID: \"333e0ba0-df2b-4d51-9ac2-ff464c8bc348\") " pod="openshift-console/console-6db45d6b4-7lt8m" Oct 30 17:59:00.040203 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:00.040159 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-tmp\") pod \"console-6db45d6b4-7lt8m\" (UID: \"333e0ba0-df2b-4d51-9ac2-ff464c8bc348\") " pod="openshift-console/console-6db45d6b4-7lt8m" Oct 30 17:59:00.040256 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:00.040202 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-oauth-serving-cert\") pod \"console-6db45d6b4-7lt8m\" (UID: \"333e0ba0-df2b-4d51-9ac2-ff464c8bc348\") " pod="openshift-console/console-6db45d6b4-7lt8m" Oct 30 17:59:00.042758 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:00.042736 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 17:59:00.054908 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:00.054879 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 17:59:00.055722 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:00.055702 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6db45d6b4-7lt8m"] Oct 30 17:59:00.098510 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:00.098483 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 17:59:00.098711 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:00.098543 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-xxtjb\"" Oct 30 17:59:00.098711 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:00.098541 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 17:59:00.140554 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:00.140522 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-tmp\") pod \"console-6db45d6b4-7lt8m\" (UID: \"333e0ba0-df2b-4d51-9ac2-ff464c8bc348\") " pod="openshift-console/console-6db45d6b4-7lt8m" Oct 30 17:59:00.140728 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:00.140601 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-oauth-serving-cert\") pod \"console-6db45d6b4-7lt8m\" (UID: \"333e0ba0-df2b-4d51-9ac2-ff464c8bc348\") " pod="openshift-console/console-6db45d6b4-7lt8m" Oct 30 17:59:00.140728 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:00.140634 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p4lt\" (UniqueName: \"kubernetes.io/projected/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-kube-api-access-4p4lt\") pod \"console-6db45d6b4-7lt8m\" (UID: \"333e0ba0-df2b-4d51-9ac2-ff464c8bc348\") " pod="openshift-console/console-6db45d6b4-7lt8m" Oct 30 17:59:00.140728 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:00.140659 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-console-serving-cert\") pod \"console-6db45d6b4-7lt8m\" (UID: \"333e0ba0-df2b-4d51-9ac2-ff464c8bc348\") " pod="openshift-console/console-6db45d6b4-7lt8m" Oct 30 17:59:00.140882 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:00.140807 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-console-oauth-config\") pod \"console-6db45d6b4-7lt8m\" (UID: \"333e0ba0-df2b-4d51-9ac2-ff464c8bc348\") " pod="openshift-console/console-6db45d6b4-7lt8m" Oct 30 17:59:00.140882 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:00.140843 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-trusted-ca-bundle\") pod \"console-6db45d6b4-7lt8m\" (UID: \"333e0ba0-df2b-4d51-9ac2-ff464c8bc348\") " pod="openshift-console/console-6db45d6b4-7lt8m" Oct 30 17:59:00.140882 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:00.140872 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-service-ca\") pod \"console-6db45d6b4-7lt8m\" (UID: \"333e0ba0-df2b-4d51-9ac2-ff464c8bc348\") " pod="openshift-console/console-6db45d6b4-7lt8m" Oct 30 17:59:00.141027 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:00.140943 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-tmp\") pod \"console-6db45d6b4-7lt8m\" (UID: \"333e0ba0-df2b-4d51-9ac2-ff464c8bc348\") " pod="openshift-console/console-6db45d6b4-7lt8m" Oct 30 17:59:00.141027 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:00.140978 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-console-config\") pod \"console-6db45d6b4-7lt8m\" (UID: \"333e0ba0-df2b-4d51-9ac2-ff464c8bc348\") " pod="openshift-console/console-6db45d6b4-7lt8m" Oct 30 17:59:00.141407 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:00.141386 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-oauth-serving-cert\") pod \"console-6db45d6b4-7lt8m\" (UID: \"333e0ba0-df2b-4d51-9ac2-ff464c8bc348\") " pod="openshift-console/console-6db45d6b4-7lt8m" Oct 30 17:59:00.141729 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:00.141702 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-console-config\") pod \"console-6db45d6b4-7lt8m\" (UID: \"333e0ba0-df2b-4d51-9ac2-ff464c8bc348\") " pod="openshift-console/console-6db45d6b4-7lt8m" Oct 30 17:59:00.141729 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:00.141721 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-service-ca\") pod \"console-6db45d6b4-7lt8m\" (UID: \"333e0ba0-df2b-4d51-9ac2-ff464c8bc348\") " pod="openshift-console/console-6db45d6b4-7lt8m" Oct 30 17:59:00.141857 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:00.141741 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-trusted-ca-bundle\") pod \"console-6db45d6b4-7lt8m\" (UID: \"333e0ba0-df2b-4d51-9ac2-ff464c8bc348\") " pod="openshift-console/console-6db45d6b4-7lt8m" Oct 30 17:59:00.143383 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:00.143360 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-console-oauth-config\") pod \"console-6db45d6b4-7lt8m\" (UID: \"333e0ba0-df2b-4d51-9ac2-ff464c8bc348\") " pod="openshift-console/console-6db45d6b4-7lt8m" Oct 30 17:59:00.143480 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:00.143465 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-console-serving-cert\") pod \"console-6db45d6b4-7lt8m\" (UID: \"333e0ba0-df2b-4d51-9ac2-ff464c8bc348\") " pod="openshift-console/console-6db45d6b4-7lt8m" Oct 30 17:59:00.155404 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:00.155375 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-59dcddb9c7-k7vz2" Oct 30 17:59:00.162635 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:00.162610 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p4lt\" (UniqueName: \"kubernetes.io/projected/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-kube-api-access-4p4lt\") pod \"console-6db45d6b4-7lt8m\" (UID: \"333e0ba0-df2b-4d51-9ac2-ff464c8bc348\") " pod="openshift-console/console-6db45d6b4-7lt8m" Oct 30 17:59:00.313646 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:00.313564 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6db45d6b4-7lt8m" Oct 30 17:59:00.623827 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:00.623794 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6db45d6b4-7lt8m"] Oct 30 17:59:00.628418 ip-10-0-138-176 kubenswrapper[4442]: W1030 17:59:00.628391 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod333e0ba0_df2b_4d51_9ac2_ff464c8bc348.slice/crio-db10f49545d0fbfc077c8c1746a0201f0029c7aec3e0f9c24d18cfe0f4e94051 WatchSource:0}: Error finding container db10f49545d0fbfc077c8c1746a0201f0029c7aec3e0f9c24d18cfe0f4e94051: Status 404 returned error can't find the container with id db10f49545d0fbfc077c8c1746a0201f0029c7aec3e0f9c24d18cfe0f4e94051 Oct 30 17:59:01.247066 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:01.247024 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6db45d6b4-7lt8m" event={"ID":"333e0ba0-df2b-4d51-9ac2-ff464c8bc348","Type":"ContainerStarted","Data":"db10f49545d0fbfc077c8c1746a0201f0029c7aec3e0f9c24d18cfe0f4e94051"} Oct 30 17:59:03.255956 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:03.255923 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6db45d6b4-7lt8m" event={"ID":"333e0ba0-df2b-4d51-9ac2-ff464c8bc348","Type":"ContainerStarted","Data":"3a589481153d2ee24e92779bc9ba6a4a4131d55b626d36bcdbf191f6b58df6d1"} Oct 30 17:59:03.309544 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:03.309493 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6db45d6b4-7lt8m" podStartSLOduration=1.81241271 podStartE2EDuration="4.309478568s" totalImagesPullingTime="2.497065858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:58:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:59:03.297507271 +0000 UTC m=+40.387569612" watchObservedRunningTime="2026-10-30 17:59:03.309478568 +0000 UTC m=+40.399540943" Oct 30 17:59:10.314426 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:10.314341 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6db45d6b4-7lt8m" Oct 30 17:59:10.314426 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:10.314436 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6db45d6b4-7lt8m" Oct 30 17:59:10.319124 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:10.319105 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6db45d6b4-7lt8m" Oct 30 17:59:10.480045 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:10.480016 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-78c74bf4c6-prjr2" Oct 30 17:59:10.480233 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:10.480071 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-78c74bf4c6-prjr2" Oct 30 17:59:11.288911 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:11.288881 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6db45d6b4-7lt8m" Oct 30 17:59:17.557892 ip-10-0-138-176 kubenswrapper[4442]: E1030 17:59:17.557842 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 17:59:30.485205 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:30.485178 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-78c74bf4c6-prjr2" Oct 30 17:59:30.488790 ip-10-0-138-176 kubenswrapper[4442]: I1030 17:59:30.488768 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-78c74bf4c6-prjr2" Oct 30 17:59:47.561556 ip-10-0-138-176 kubenswrapper[4442]: E1030 17:59:47.561515 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:00:17.558433 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:00:17.558394 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:00:35.810280 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:00:35.810191 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5fb9f6977f-bl8xj"] Oct 30 18:00:35.813566 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:00:35.813544 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fb9f6977f-bl8xj" Oct 30 18:00:35.834860 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:00:35.834830 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fb9f6977f-bl8xj"] Oct 30 18:00:35.841349 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:00:35.841321 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05978eb8-6b34-4cfa-ab02-c57212d5d8d8-oauth-serving-cert\") pod \"console-5fb9f6977f-bl8xj\" (UID: \"05978eb8-6b34-4cfa-ab02-c57212d5d8d8\") " pod="openshift-console/console-5fb9f6977f-bl8xj" Oct 30 18:00:35.841349 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:00:35.841352 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05978eb8-6b34-4cfa-ab02-c57212d5d8d8-service-ca\") pod \"console-5fb9f6977f-bl8xj\" (UID: \"05978eb8-6b34-4cfa-ab02-c57212d5d8d8\") " pod="openshift-console/console-5fb9f6977f-bl8xj" Oct 30 18:00:35.841547 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:00:35.841381 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05978eb8-6b34-4cfa-ab02-c57212d5d8d8-console-oauth-config\") pod \"console-5fb9f6977f-bl8xj\" (UID: \"05978eb8-6b34-4cfa-ab02-c57212d5d8d8\") " pod="openshift-console/console-5fb9f6977f-bl8xj" Oct 30 18:00:35.841547 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:00:35.841413 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05978eb8-6b34-4cfa-ab02-c57212d5d8d8-console-config\") pod \"console-5fb9f6977f-bl8xj\" (UID: \"05978eb8-6b34-4cfa-ab02-c57212d5d8d8\") " pod="openshift-console/console-5fb9f6977f-bl8xj" Oct 30 18:00:35.841634 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:00:35.841542 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05978eb8-6b34-4cfa-ab02-c57212d5d8d8-tmp\") pod \"console-5fb9f6977f-bl8xj\" (UID: \"05978eb8-6b34-4cfa-ab02-c57212d5d8d8\") " pod="openshift-console/console-5fb9f6977f-bl8xj" Oct 30 18:00:35.841634 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:00:35.841603 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05978eb8-6b34-4cfa-ab02-c57212d5d8d8-trusted-ca-bundle\") pod \"console-5fb9f6977f-bl8xj\" (UID: \"05978eb8-6b34-4cfa-ab02-c57212d5d8d8\") " pod="openshift-console/console-5fb9f6977f-bl8xj" Oct 30 18:00:35.841717 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:00:35.841705 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05978eb8-6b34-4cfa-ab02-c57212d5d8d8-console-serving-cert\") pod \"console-5fb9f6977f-bl8xj\" (UID: \"05978eb8-6b34-4cfa-ab02-c57212d5d8d8\") " pod="openshift-console/console-5fb9f6977f-bl8xj" Oct 30 18:00:35.841756 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:00:35.841732 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcvx4\" (UniqueName: \"kubernetes.io/projected/05978eb8-6b34-4cfa-ab02-c57212d5d8d8-kube-api-access-hcvx4\") pod \"console-5fb9f6977f-bl8xj\" (UID: \"05978eb8-6b34-4cfa-ab02-c57212d5d8d8\") " pod="openshift-console/console-5fb9f6977f-bl8xj" Oct 30 18:00:35.943059 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:00:35.943017 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05978eb8-6b34-4cfa-ab02-c57212d5d8d8-oauth-serving-cert\") pod \"console-5fb9f6977f-bl8xj\" (UID: \"05978eb8-6b34-4cfa-ab02-c57212d5d8d8\") " pod="openshift-console/console-5fb9f6977f-bl8xj" Oct 30 18:00:35.943059 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:00:35.943059 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05978eb8-6b34-4cfa-ab02-c57212d5d8d8-service-ca\") pod \"console-5fb9f6977f-bl8xj\" (UID: \"05978eb8-6b34-4cfa-ab02-c57212d5d8d8\") " pod="openshift-console/console-5fb9f6977f-bl8xj" Oct 30 18:00:35.943325 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:00:35.943084 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05978eb8-6b34-4cfa-ab02-c57212d5d8d8-console-oauth-config\") pod \"console-5fb9f6977f-bl8xj\" (UID: \"05978eb8-6b34-4cfa-ab02-c57212d5d8d8\") " pod="openshift-console/console-5fb9f6977f-bl8xj" Oct 30 18:00:35.943325 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:00:35.943101 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05978eb8-6b34-4cfa-ab02-c57212d5d8d8-console-config\") pod \"console-5fb9f6977f-bl8xj\" (UID: \"05978eb8-6b34-4cfa-ab02-c57212d5d8d8\") " pod="openshift-console/console-5fb9f6977f-bl8xj" Oct 30 18:00:35.943325 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:00:35.943118 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05978eb8-6b34-4cfa-ab02-c57212d5d8d8-tmp\") pod \"console-5fb9f6977f-bl8xj\" (UID: \"05978eb8-6b34-4cfa-ab02-c57212d5d8d8\") " pod="openshift-console/console-5fb9f6977f-bl8xj" Oct 30 18:00:35.943325 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:00:35.943157 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05978eb8-6b34-4cfa-ab02-c57212d5d8d8-trusted-ca-bundle\") pod \"console-5fb9f6977f-bl8xj\" (UID: \"05978eb8-6b34-4cfa-ab02-c57212d5d8d8\") " pod="openshift-console/console-5fb9f6977f-bl8xj" Oct 30 18:00:35.943325 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:00:35.943203 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05978eb8-6b34-4cfa-ab02-c57212d5d8d8-console-serving-cert\") pod \"console-5fb9f6977f-bl8xj\" (UID: \"05978eb8-6b34-4cfa-ab02-c57212d5d8d8\") " pod="openshift-console/console-5fb9f6977f-bl8xj" Oct 30 18:00:35.943325 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:00:35.943223 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcvx4\" (UniqueName: \"kubernetes.io/projected/05978eb8-6b34-4cfa-ab02-c57212d5d8d8-kube-api-access-hcvx4\") pod \"console-5fb9f6977f-bl8xj\" (UID: \"05978eb8-6b34-4cfa-ab02-c57212d5d8d8\") " pod="openshift-console/console-5fb9f6977f-bl8xj" Oct 30 18:00:35.943653 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:00:35.943613 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05978eb8-6b34-4cfa-ab02-c57212d5d8d8-tmp\") pod \"console-5fb9f6977f-bl8xj\" (UID: \"05978eb8-6b34-4cfa-ab02-c57212d5d8d8\") " pod="openshift-console/console-5fb9f6977f-bl8xj" Oct 30 18:00:35.943950 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:00:35.943927 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05978eb8-6b34-4cfa-ab02-c57212d5d8d8-oauth-serving-cert\") pod \"console-5fb9f6977f-bl8xj\" (UID: \"05978eb8-6b34-4cfa-ab02-c57212d5d8d8\") " pod="openshift-console/console-5fb9f6977f-bl8xj" Oct 30 18:00:35.944063 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:00:35.944013 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05978eb8-6b34-4cfa-ab02-c57212d5d8d8-console-config\") pod \"console-5fb9f6977f-bl8xj\" (UID: \"05978eb8-6b34-4cfa-ab02-c57212d5d8d8\") " pod="openshift-console/console-5fb9f6977f-bl8xj" Oct 30 18:00:35.944404 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:00:35.944382 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05978eb8-6b34-4cfa-ab02-c57212d5d8d8-service-ca\") pod \"console-5fb9f6977f-bl8xj\" (UID: \"05978eb8-6b34-4cfa-ab02-c57212d5d8d8\") " pod="openshift-console/console-5fb9f6977f-bl8xj" Oct 30 18:00:35.944653 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:00:35.944630 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05978eb8-6b34-4cfa-ab02-c57212d5d8d8-trusted-ca-bundle\") pod \"console-5fb9f6977f-bl8xj\" (UID: \"05978eb8-6b34-4cfa-ab02-c57212d5d8d8\") " pod="openshift-console/console-5fb9f6977f-bl8xj" Oct 30 18:00:35.945641 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:00:35.945620 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05978eb8-6b34-4cfa-ab02-c57212d5d8d8-console-oauth-config\") pod \"console-5fb9f6977f-bl8xj\" (UID: \"05978eb8-6b34-4cfa-ab02-c57212d5d8d8\") " pod="openshift-console/console-5fb9f6977f-bl8xj" Oct 30 18:00:35.946049 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:00:35.946028 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05978eb8-6b34-4cfa-ab02-c57212d5d8d8-console-serving-cert\") pod \"console-5fb9f6977f-bl8xj\" (UID: \"05978eb8-6b34-4cfa-ab02-c57212d5d8d8\") " pod="openshift-console/console-5fb9f6977f-bl8xj" Oct 30 18:00:35.967016 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:00:35.966991 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcvx4\" (UniqueName: \"kubernetes.io/projected/05978eb8-6b34-4cfa-ab02-c57212d5d8d8-kube-api-access-hcvx4\") pod \"console-5fb9f6977f-bl8xj\" (UID: \"05978eb8-6b34-4cfa-ab02-c57212d5d8d8\") " pod="openshift-console/console-5fb9f6977f-bl8xj" Oct 30 18:00:36.124674 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:00:36.124640 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fb9f6977f-bl8xj" Oct 30 18:00:36.420970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:00:36.420887 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fb9f6977f-bl8xj"] Oct 30 18:00:36.425513 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:00:36.425489 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05978eb8_6b34_4cfa_ab02_c57212d5d8d8.slice/crio-e195aef388d317eea53b43bd983547c7d7b060cec2594140e38ae0cacd58864f WatchSource:0}: Error finding container e195aef388d317eea53b43bd983547c7d7b060cec2594140e38ae0cacd58864f: Status 404 returned error can't find the container with id e195aef388d317eea53b43bd983547c7d7b060cec2594140e38ae0cacd58864f Oct 30 18:00:36.568478 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:00:36.568442 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fb9f6977f-bl8xj" event={"ID":"05978eb8-6b34-4cfa-ab02-c57212d5d8d8","Type":"ContainerStarted","Data":"408a55690b3a2b60660120384fc149d5ad8e42dc1a8188581f16baeb4c15c06e"} Oct 30 18:00:36.568617 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:00:36.568517 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fb9f6977f-bl8xj" event={"ID":"05978eb8-6b34-4cfa-ab02-c57212d5d8d8","Type":"ContainerStarted","Data":"e195aef388d317eea53b43bd983547c7d7b060cec2594140e38ae0cacd58864f"} Oct 30 18:00:36.608980 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:00:36.608933 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5fb9f6977f-bl8xj" podStartSLOduration=1.6089180459999999 podStartE2EDuration="1.608918046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:36.608186957 +0000 UTC m=+133.698249297" watchObservedRunningTime="2026-10-30 18:00:36.608918046 +0000 UTC m=+133.698980384" Oct 30 18:00:46.125641 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:00:46.125611 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5fb9f6977f-bl8xj" Oct 30 18:00:46.126260 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:00:46.125671 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5fb9f6977f-bl8xj" Oct 30 18:00:46.130435 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:00:46.130413 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5fb9f6977f-bl8xj" Oct 30 18:00:46.604461 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:00:46.604376 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5fb9f6977f-bl8xj" Oct 30 18:00:46.675351 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:00:46.675317 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6db45d6b4-7lt8m"] Oct 30 18:00:47.558839 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:00:47.558807 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:01:11.695803 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:01:11.695759 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6db45d6b4-7lt8m" podUID="333e0ba0-df2b-4d51-9ac2-ff464c8bc348" containerName="console" containerID="cri-o://3a589481153d2ee24e92779bc9ba6a4a4131d55b626d36bcdbf191f6b58df6d1" gracePeriod=15 Oct 30 18:01:11.935167 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:01:11.935141 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6db45d6b4-7lt8m" Oct 30 18:01:12.019813 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:01:12.019733 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-console-oauth-config\") pod \"333e0ba0-df2b-4d51-9ac2-ff464c8bc348\" (UID: \"333e0ba0-df2b-4d51-9ac2-ff464c8bc348\") " Oct 30 18:01:12.019813 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:01:12.019770 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-tmp\" (UniqueName: \"kubernetes.io/empty-dir/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-tmp\") pod \"333e0ba0-df2b-4d51-9ac2-ff464c8bc348\" (UID: \"333e0ba0-df2b-4d51-9ac2-ff464c8bc348\") " Oct 30 18:01:12.019813 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:01:12.019806 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-service-ca\" (UniqueName: \"kubernetes.io/configmap/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-service-ca\") pod \"333e0ba0-df2b-4d51-9ac2-ff464c8bc348\" (UID: \"333e0ba0-df2b-4d51-9ac2-ff464c8bc348\") " Oct 30 18:01:12.020084 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:01:12.019830 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-oauth-serving-cert\") pod \"333e0ba0-df2b-4d51-9ac2-ff464c8bc348\" (UID: \"333e0ba0-df2b-4d51-9ac2-ff464c8bc348\") " Oct 30 18:01:12.020084 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:01:12.019876 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-kube-api-access-4p4lt\" (UniqueName: \"kubernetes.io/projected/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-kube-api-access-4p4lt\") pod \"333e0ba0-df2b-4d51-9ac2-ff464c8bc348\" (UID: \"333e0ba0-df2b-4d51-9ac2-ff464c8bc348\") " Oct 30 18:01:12.020084 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:01:12.019899 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-trusted-ca-bundle\") pod \"333e0ba0-df2b-4d51-9ac2-ff464c8bc348\" (UID: \"333e0ba0-df2b-4d51-9ac2-ff464c8bc348\") " Oct 30 18:01:12.020084 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:01:12.019921 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-console-config\" (UniqueName: \"kubernetes.io/configmap/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-console-config\") pod \"333e0ba0-df2b-4d51-9ac2-ff464c8bc348\" (UID: \"333e0ba0-df2b-4d51-9ac2-ff464c8bc348\") " Oct 30 18:01:12.020310 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:01:12.020069 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-tmp" pod "333e0ba0-df2b-4d51-9ac2-ff464c8bc348" (UID: "333e0ba0-df2b-4d51-9ac2-ff464c8bc348"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:01:12.020310 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:01:12.020112 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-console-serving-cert\") pod \"333e0ba0-df2b-4d51-9ac2-ff464c8bc348\" (UID: \"333e0ba0-df2b-4d51-9ac2-ff464c8bc348\") " Oct 30 18:01:12.020310 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:01:12.020285 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-service-ca" pod "333e0ba0-df2b-4d51-9ac2-ff464c8bc348" (UID: "333e0ba0-df2b-4d51-9ac2-ff464c8bc348"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:01:12.020463 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:01:12.020436 4442 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-tmp\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:01:12.020463 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:01:12.020455 4442 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:01:12.020568 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:01:12.020461 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-console-config" pod "333e0ba0-df2b-4d51-9ac2-ff464c8bc348" (UID: "333e0ba0-df2b-4d51-9ac2-ff464c8bc348"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:01:12.020622 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:01:12.020573 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-trusted-ca-bundle" pod "333e0ba0-df2b-4d51-9ac2-ff464c8bc348" (UID: "333e0ba0-df2b-4d51-9ac2-ff464c8bc348"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:01:12.020720 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:01:12.020700 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-oauth-serving-cert" pod "333e0ba0-df2b-4d51-9ac2-ff464c8bc348" (UID: "333e0ba0-df2b-4d51-9ac2-ff464c8bc348"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:01:12.022332 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:01:12.022309 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-console-oauth-config" pod "333e0ba0-df2b-4d51-9ac2-ff464c8bc348" (UID: "333e0ba0-df2b-4d51-9ac2-ff464c8bc348"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:01:12.022429 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:01:12.022384 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-console-serving-cert" pod "333e0ba0-df2b-4d51-9ac2-ff464c8bc348" (UID: "333e0ba0-df2b-4d51-9ac2-ff464c8bc348"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:01:12.022429 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:01:12.022404 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-kube-api-access-4p4lt" pod "333e0ba0-df2b-4d51-9ac2-ff464c8bc348" (UID: "333e0ba0-df2b-4d51-9ac2-ff464c8bc348"). InnerVolumeSpecName "kube-api-access-4p4lt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:01:12.121605 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:01:12.121558 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p4lt\" (UniqueName: \"kubernetes.io/projected/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-kube-api-access-4p4lt\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:01:12.121605 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:01:12.121600 4442 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-trusted-ca-bundle\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:01:12.121605 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:01:12.121611 4442 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-console-config\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:01:12.121605 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:01:12.121620 4442 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-console-serving-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:01:12.121871 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:01:12.121628 4442 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-console-oauth-config\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:01:12.121871 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:01:12.121637 4442 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/333e0ba0-df2b-4d51-9ac2-ff464c8bc348-oauth-serving-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:01:12.681249 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:01:12.681216 4442 generic.go:356] "Generic (PLEG): container finished" podID="333e0ba0-df2b-4d51-9ac2-ff464c8bc348" containerID="3a589481153d2ee24e92779bc9ba6a4a4131d55b626d36bcdbf191f6b58df6d1" exitCode=0 Oct 30 18:01:12.681418 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:01:12.681268 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6db45d6b4-7lt8m" Oct 30 18:01:12.681418 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:01:12.681302 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6db45d6b4-7lt8m" event={"ID":"333e0ba0-df2b-4d51-9ac2-ff464c8bc348","Type":"ContainerDied","Data":"3a589481153d2ee24e92779bc9ba6a4a4131d55b626d36bcdbf191f6b58df6d1"} Oct 30 18:01:12.681418 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:01:12.681355 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6db45d6b4-7lt8m" event={"ID":"333e0ba0-df2b-4d51-9ac2-ff464c8bc348","Type":"ContainerDied","Data":"db10f49545d0fbfc077c8c1746a0201f0029c7aec3e0f9c24d18cfe0f4e94051"} Oct 30 18:01:12.681418 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:01:12.681376 4442 scope.go:122] "RemoveContainer" containerID="3a589481153d2ee24e92779bc9ba6a4a4131d55b626d36bcdbf191f6b58df6d1" Oct 30 18:01:12.690523 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:01:12.690506 4442 scope.go:122] "RemoveContainer" containerID="3a589481153d2ee24e92779bc9ba6a4a4131d55b626d36bcdbf191f6b58df6d1" Oct 30 18:01:12.690793 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:01:12.690770 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a589481153d2ee24e92779bc9ba6a4a4131d55b626d36bcdbf191f6b58df6d1\": container with ID starting with 3a589481153d2ee24e92779bc9ba6a4a4131d55b626d36bcdbf191f6b58df6d1 not found: ID does not exist" containerID="3a589481153d2ee24e92779bc9ba6a4a4131d55b626d36bcdbf191f6b58df6d1" Oct 30 18:01:12.690889 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:01:12.690799 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a589481153d2ee24e92779bc9ba6a4a4131d55b626d36bcdbf191f6b58df6d1"} err="failed to get container status \"3a589481153d2ee24e92779bc9ba6a4a4131d55b626d36bcdbf191f6b58df6d1\": rpc error: code = NotFound desc = could not find container \"3a589481153d2ee24e92779bc9ba6a4a4131d55b626d36bcdbf191f6b58df6d1\": container with ID starting with 3a589481153d2ee24e92779bc9ba6a4a4131d55b626d36bcdbf191f6b58df6d1 not found: ID does not exist" Oct 30 18:01:12.730474 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:01:12.730442 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6db45d6b4-7lt8m"] Oct 30 18:01:12.742023 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:01:12.741992 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6db45d6b4-7lt8m"] Oct 30 18:01:13.061352 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:01:13.061284 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="333e0ba0-df2b-4d51-9ac2-ff464c8bc348" path="/var/lib/kubelet/pods/333e0ba0-df2b-4d51-9ac2-ff464c8bc348/volumes" Oct 30 18:01:17.560614 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:01:17.560573 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:01:47.557101 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:01:47.557068 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:02:17.558412 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:02:17.558334 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:02:29.660372 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:29.660340 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hdwk9"] Oct 30 18:02:29.660882 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:29.660590 4442 cpu_manager.go:414] "containerMap: removing container" podUID="333e0ba0-df2b-4d51-9ac2-ff464c8bc348" containerName="console" Oct 30 18:02:29.660882 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:29.660600 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="333e0ba0-df2b-4d51-9ac2-ff464c8bc348" containerName="console" Oct 30 18:02:29.660882 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:29.660659 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="333e0ba0-df2b-4d51-9ac2-ff464c8bc348" containerName="console" Oct 30 18:02:29.663828 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:29.663807 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hdwk9" Oct 30 18:02:29.693815 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:29.693779 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 18:02:29.694006 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:29.693910 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 18:02:29.696283 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:29.696248 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hdwk9"] Oct 30 18:02:29.705605 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:29.705575 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-m7kfq\"" Oct 30 18:02:29.746096 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:29.746063 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tfns"] Oct 30 18:02:29.749719 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:29.749678 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tfns" Oct 30 18:02:29.756234 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:29.756196 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/452375be-3092-46cf-a3e4-56111a75fb24-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hdwk9\" (UID: \"452375be-3092-46cf-a3e4-56111a75fb24\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hdwk9" Oct 30 18:02:29.756385 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:29.756302 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/452375be-3092-46cf-a3e4-56111a75fb24-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hdwk9\" (UID: \"452375be-3092-46cf-a3e4-56111a75fb24\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hdwk9" Oct 30 18:02:29.756385 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:29.756363 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zs8r\" (UniqueName: \"kubernetes.io/projected/452375be-3092-46cf-a3e4-56111a75fb24-kube-api-access-2zs8r\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hdwk9\" (UID: \"452375be-3092-46cf-a3e4-56111a75fb24\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hdwk9" Oct 30 18:02:29.766783 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:29.766752 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-mw499\"" Oct 30 18:02:29.768116 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:29.768071 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tfns"] Oct 30 18:02:29.857020 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:29.856985 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/452375be-3092-46cf-a3e4-56111a75fb24-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hdwk9\" (UID: \"452375be-3092-46cf-a3e4-56111a75fb24\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hdwk9" Oct 30 18:02:29.857020 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:29.857035 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zs8r\" (UniqueName: \"kubernetes.io/projected/452375be-3092-46cf-a3e4-56111a75fb24-kube-api-access-2zs8r\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hdwk9\" (UID: \"452375be-3092-46cf-a3e4-56111a75fb24\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hdwk9" Oct 30 18:02:29.857380 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:29.857060 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/452375be-3092-46cf-a3e4-56111a75fb24-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hdwk9\" (UID: \"452375be-3092-46cf-a3e4-56111a75fb24\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hdwk9" Oct 30 18:02:29.857380 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:29.857082 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmmxp\" (UniqueName: \"kubernetes.io/projected/2442891f-bd5e-47cd-b882-2121cdd96cd0-kube-api-access-qmmxp\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tfns\" (UID: \"2442891f-bd5e-47cd-b882-2121cdd96cd0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tfns" Oct 30 18:02:29.857380 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:29.857117 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2442891f-bd5e-47cd-b882-2121cdd96cd0-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tfns\" (UID: \"2442891f-bd5e-47cd-b882-2121cdd96cd0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tfns" Oct 30 18:02:29.857380 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:29.857166 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2442891f-bd5e-47cd-b882-2121cdd96cd0-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tfns\" (UID: \"2442891f-bd5e-47cd-b882-2121cdd96cd0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tfns" Oct 30 18:02:29.857552 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:29.857498 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/452375be-3092-46cf-a3e4-56111a75fb24-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hdwk9\" (UID: \"452375be-3092-46cf-a3e4-56111a75fb24\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hdwk9" Oct 30 18:02:29.857552 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:29.857511 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/452375be-3092-46cf-a3e4-56111a75fb24-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hdwk9\" (UID: \"452375be-3092-46cf-a3e4-56111a75fb24\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hdwk9" Oct 30 18:02:29.877031 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:29.876991 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zs8r\" (UniqueName: \"kubernetes.io/projected/452375be-3092-46cf-a3e4-56111a75fb24-kube-api-access-2zs8r\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hdwk9\" (UID: \"452375be-3092-46cf-a3e4-56111a75fb24\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hdwk9" Oct 30 18:02:29.958248 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:29.958160 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2442891f-bd5e-47cd-b882-2121cdd96cd0-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tfns\" (UID: \"2442891f-bd5e-47cd-b882-2121cdd96cd0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tfns" Oct 30 18:02:29.958400 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:29.958248 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmmxp\" (UniqueName: \"kubernetes.io/projected/2442891f-bd5e-47cd-b882-2121cdd96cd0-kube-api-access-qmmxp\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tfns\" (UID: \"2442891f-bd5e-47cd-b882-2121cdd96cd0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tfns" Oct 30 18:02:29.958400 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:29.958285 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2442891f-bd5e-47cd-b882-2121cdd96cd0-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tfns\" (UID: \"2442891f-bd5e-47cd-b882-2121cdd96cd0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tfns" Oct 30 18:02:29.958561 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:29.958544 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2442891f-bd5e-47cd-b882-2121cdd96cd0-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tfns\" (UID: \"2442891f-bd5e-47cd-b882-2121cdd96cd0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tfns" Oct 30 18:02:29.958593 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:29.958581 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2442891f-bd5e-47cd-b882-2121cdd96cd0-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tfns\" (UID: \"2442891f-bd5e-47cd-b882-2121cdd96cd0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tfns" Oct 30 18:02:29.979209 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:29.979179 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hdwk9" Oct 30 18:02:29.984940 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:29.984915 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmmxp\" (UniqueName: \"kubernetes.io/projected/2442891f-bd5e-47cd-b882-2121cdd96cd0-kube-api-access-qmmxp\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tfns\" (UID: \"2442891f-bd5e-47cd-b882-2121cdd96cd0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tfns" Oct 30 18:02:30.066408 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:30.066374 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tfns" Oct 30 18:02:30.285009 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:30.284911 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hdwk9"] Oct 30 18:02:30.288927 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:02:30.288903 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod452375be_3092_46cf_a3e4_56111a75fb24.slice/crio-f31c4c80df02d60051f881d874dbd74bb4520e20b407f46bede06a472ad84f6a WatchSource:0}: Error finding container f31c4c80df02d60051f881d874dbd74bb4520e20b407f46bede06a472ad84f6a: Status 404 returned error can't find the container with id f31c4c80df02d60051f881d874dbd74bb4520e20b407f46bede06a472ad84f6a Oct 30 18:02:30.361407 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:30.361373 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tfns"] Oct 30 18:02:30.365965 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:02:30.365940 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2442891f_bd5e_47cd_b882_2121cdd96cd0.slice/crio-03643f0bb4bcf674c93c890458d8bb336768083e28b61c2be0bf1a85b7bc5db0 WatchSource:0}: Error finding container 03643f0bb4bcf674c93c890458d8bb336768083e28b61c2be0bf1a85b7bc5db0: Status 404 returned error can't find the container with id 03643f0bb4bcf674c93c890458d8bb336768083e28b61c2be0bf1a85b7bc5db0 Oct 30 18:02:30.948254 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:30.948219 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tfns" event={"ID":"2442891f-bd5e-47cd-b882-2121cdd96cd0","Type":"ContainerStarted","Data":"03643f0bb4bcf674c93c890458d8bb336768083e28b61c2be0bf1a85b7bc5db0"} Oct 30 18:02:30.949887 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:30.949862 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hdwk9" event={"ID":"452375be-3092-46cf-a3e4-56111a75fb24","Type":"ContainerStarted","Data":"f31c4c80df02d60051f881d874dbd74bb4520e20b407f46bede06a472ad84f6a"} Oct 30 18:02:32.937619 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:32.937584 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5ddml"] Oct 30 18:02:32.942760 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:32.942268 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5ddml" Oct 30 18:02:32.966043 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:32.966010 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5ddml"] Oct 30 18:02:32.976209 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:32.976178 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-vb5l2\"" Oct 30 18:02:32.985045 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:32.985022 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cd769ce4-6c17-47da-be0a-fdb8bbc7f76b-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5ddml\" (UID: \"cd769ce4-6c17-47da-be0a-fdb8bbc7f76b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5ddml" Oct 30 18:02:32.985205 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:32.985064 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cd769ce4-6c17-47da-be0a-fdb8bbc7f76b-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5ddml\" (UID: \"cd769ce4-6c17-47da-be0a-fdb8bbc7f76b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5ddml" Oct 30 18:02:32.985275 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:32.985202 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5bq5\" (UniqueName: \"kubernetes.io/projected/cd769ce4-6c17-47da-be0a-fdb8bbc7f76b-kube-api-access-g5bq5\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5ddml\" (UID: \"cd769ce4-6c17-47da-be0a-fdb8bbc7f76b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5ddml" Oct 30 18:02:33.086851 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.086117 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cd769ce4-6c17-47da-be0a-fdb8bbc7f76b-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5ddml\" (UID: \"cd769ce4-6c17-47da-be0a-fdb8bbc7f76b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5ddml" Oct 30 18:02:33.086851 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.086192 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5bq5\" (UniqueName: \"kubernetes.io/projected/cd769ce4-6c17-47da-be0a-fdb8bbc7f76b-kube-api-access-g5bq5\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5ddml\" (UID: \"cd769ce4-6c17-47da-be0a-fdb8bbc7f76b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5ddml" Oct 30 18:02:33.086851 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.086269 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cd769ce4-6c17-47da-be0a-fdb8bbc7f76b-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5ddml\" (UID: \"cd769ce4-6c17-47da-be0a-fdb8bbc7f76b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5ddml" Oct 30 18:02:33.086851 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.086500 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cd769ce4-6c17-47da-be0a-fdb8bbc7f76b-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5ddml\" (UID: \"cd769ce4-6c17-47da-be0a-fdb8bbc7f76b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5ddml" Oct 30 18:02:33.086851 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.086554 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cd769ce4-6c17-47da-be0a-fdb8bbc7f76b-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5ddml\" (UID: \"cd769ce4-6c17-47da-be0a-fdb8bbc7f76b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5ddml" Oct 30 18:02:33.107238 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.107205 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5bq5\" (UniqueName: \"kubernetes.io/projected/cd769ce4-6c17-47da-be0a-fdb8bbc7f76b-kube-api-access-g5bq5\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5ddml\" (UID: \"cd769ce4-6c17-47da-be0a-fdb8bbc7f76b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5ddml" Oct 30 18:02:33.261343 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.261264 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5ddml" Oct 30 18:02:33.568310 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.568236 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5ddml"] Oct 30 18:02:33.619624 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.619590 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmksd"] Oct 30 18:02:33.624346 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.624317 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmksd" Oct 30 18:02:33.641655 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.641627 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-zp8b6\"" Oct 30 18:02:33.654419 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.654392 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmksd"] Oct 30 18:02:33.691579 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.691379 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f953747-4491-40e8-972d-e54b3951983b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmksd\" (UID: \"8f953747-4491-40e8-972d-e54b3951983b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmksd" Oct 30 18:02:33.691579 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.691454 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f953747-4491-40e8-972d-e54b3951983b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmksd\" (UID: \"8f953747-4491-40e8-972d-e54b3951983b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmksd" Oct 30 18:02:33.691579 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.691488 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plzh7\" (UniqueName: \"kubernetes.io/projected/8f953747-4491-40e8-972d-e54b3951983b-kube-api-access-plzh7\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmksd\" (UID: \"8f953747-4491-40e8-972d-e54b3951983b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmksd" Oct 30 18:02:33.722355 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.722317 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9xpmn"] Oct 30 18:02:33.726867 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.726845 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9xpmn" Oct 30 18:02:33.744203 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.744175 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-86tm8\"" Oct 30 18:02:33.745725 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.745698 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9xpmn"] Oct 30 18:02:33.792942 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.792909 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1db910f-a35a-475f-810e-d6a7a924bcbe-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9xpmn\" (UID: \"c1db910f-a35a-475f-810e-d6a7a924bcbe\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9xpmn" Oct 30 18:02:33.793119 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.792982 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f953747-4491-40e8-972d-e54b3951983b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmksd\" (UID: \"8f953747-4491-40e8-972d-e54b3951983b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmksd" Oct 30 18:02:33.793119 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.793050 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfkbr\" (UniqueName: \"kubernetes.io/projected/c1db910f-a35a-475f-810e-d6a7a924bcbe-kube-api-access-xfkbr\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9xpmn\" (UID: \"c1db910f-a35a-475f-810e-d6a7a924bcbe\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9xpmn" Oct 30 18:02:33.793119 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.793099 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1db910f-a35a-475f-810e-d6a7a924bcbe-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9xpmn\" (UID: \"c1db910f-a35a-475f-810e-d6a7a924bcbe\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9xpmn" Oct 30 18:02:33.793289 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.793195 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f953747-4491-40e8-972d-e54b3951983b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmksd\" (UID: \"8f953747-4491-40e8-972d-e54b3951983b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmksd" Oct 30 18:02:33.793289 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.793242 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plzh7\" (UniqueName: \"kubernetes.io/projected/8f953747-4491-40e8-972d-e54b3951983b-kube-api-access-plzh7\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmksd\" (UID: \"8f953747-4491-40e8-972d-e54b3951983b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmksd" Oct 30 18:02:33.793413 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.793395 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f953747-4491-40e8-972d-e54b3951983b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmksd\" (UID: \"8f953747-4491-40e8-972d-e54b3951983b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmksd" Oct 30 18:02:33.793686 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.793667 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f953747-4491-40e8-972d-e54b3951983b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmksd\" (UID: \"8f953747-4491-40e8-972d-e54b3951983b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmksd" Oct 30 18:02:33.814313 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.814280 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plzh7\" (UniqueName: \"kubernetes.io/projected/8f953747-4491-40e8-972d-e54b3951983b-kube-api-access-plzh7\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmksd\" (UID: \"8f953747-4491-40e8-972d-e54b3951983b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmksd" Oct 30 18:02:33.827158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.827044 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar96xg"] Oct 30 18:02:33.830622 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.830598 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar96xg" Oct 30 18:02:33.844799 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.844778 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-vpn6s\"" Oct 30 18:02:33.852553 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.852529 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar96xg"] Oct 30 18:02:33.893662 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.893631 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1db910f-a35a-475f-810e-d6a7a924bcbe-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9xpmn\" (UID: \"c1db910f-a35a-475f-810e-d6a7a924bcbe\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9xpmn" Oct 30 18:02:33.893823 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.893668 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grmmj\" (UniqueName: \"kubernetes.io/projected/a2299ff4-930f-4255-86f3-01a9c34c9e5b-kube-api-access-grmmj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar96xg\" (UID: \"a2299ff4-930f-4255-86f3-01a9c34c9e5b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar96xg" Oct 30 18:02:33.893823 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.893784 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2299ff4-930f-4255-86f3-01a9c34c9e5b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar96xg\" (UID: \"a2299ff4-930f-4255-86f3-01a9c34c9e5b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar96xg" Oct 30 18:02:33.893936 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.893865 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfkbr\" (UniqueName: \"kubernetes.io/projected/c1db910f-a35a-475f-810e-d6a7a924bcbe-kube-api-access-xfkbr\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9xpmn\" (UID: \"c1db910f-a35a-475f-810e-d6a7a924bcbe\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9xpmn" Oct 30 18:02:33.893936 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.893923 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1db910f-a35a-475f-810e-d6a7a924bcbe-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9xpmn\" (UID: \"c1db910f-a35a-475f-810e-d6a7a924bcbe\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9xpmn" Oct 30 18:02:33.894046 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.893967 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2299ff4-930f-4255-86f3-01a9c34c9e5b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar96xg\" (UID: \"a2299ff4-930f-4255-86f3-01a9c34c9e5b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar96xg" Oct 30 18:02:33.894096 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.894063 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1db910f-a35a-475f-810e-d6a7a924bcbe-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9xpmn\" (UID: \"c1db910f-a35a-475f-810e-d6a7a924bcbe\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9xpmn" Oct 30 18:02:33.894346 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.894326 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1db910f-a35a-475f-810e-d6a7a924bcbe-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9xpmn\" (UID: \"c1db910f-a35a-475f-810e-d6a7a924bcbe\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9xpmn" Oct 30 18:02:33.914440 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.914415 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfkbr\" (UniqueName: \"kubernetes.io/projected/c1db910f-a35a-475f-810e-d6a7a924bcbe-kube-api-access-xfkbr\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9xpmn\" (UID: \"c1db910f-a35a-475f-810e-d6a7a924bcbe\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9xpmn" Oct 30 18:02:33.917073 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.917047 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777xm2"] Oct 30 18:02:33.920553 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.920530 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777xm2" Oct 30 18:02:33.938635 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.938584 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-8kq2h\"" Oct 30 18:02:33.941165 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.940821 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777xm2"] Oct 30 18:02:33.942545 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.942370 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmksd" Oct 30 18:02:33.968799 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.968762 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5ddml" event={"ID":"cd769ce4-6c17-47da-be0a-fdb8bbc7f76b","Type":"ContainerStarted","Data":"ef522eea36a8b868fbebee650358d5f20e0639fae26363306caa430972f137a2"} Oct 30 18:02:33.995337 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.995302 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2299ff4-930f-4255-86f3-01a9c34c9e5b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar96xg\" (UID: \"a2299ff4-930f-4255-86f3-01a9c34c9e5b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar96xg" Oct 30 18:02:33.995526 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.995365 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grmmj\" (UniqueName: \"kubernetes.io/projected/a2299ff4-930f-4255-86f3-01a9c34c9e5b-kube-api-access-grmmj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar96xg\" (UID: \"a2299ff4-930f-4255-86f3-01a9c34c9e5b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar96xg" Oct 30 18:02:33.995526 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.995393 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45c0dc9a-baa9-40b5-a37d-f08973acbd6a-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777xm2\" (UID: \"45c0dc9a-baa9-40b5-a37d-f08973acbd6a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777xm2" Oct 30 18:02:33.995526 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.995427 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2299ff4-930f-4255-86f3-01a9c34c9e5b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar96xg\" (UID: \"a2299ff4-930f-4255-86f3-01a9c34c9e5b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar96xg" Oct 30 18:02:33.995526 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.995447 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45c0dc9a-baa9-40b5-a37d-f08973acbd6a-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777xm2\" (UID: \"45c0dc9a-baa9-40b5-a37d-f08973acbd6a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777xm2" Oct 30 18:02:33.995526 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.995482 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r584n\" (UniqueName: \"kubernetes.io/projected/45c0dc9a-baa9-40b5-a37d-f08973acbd6a-kube-api-access-r584n\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777xm2\" (UID: \"45c0dc9a-baa9-40b5-a37d-f08973acbd6a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777xm2" Oct 30 18:02:33.996556 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.996226 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2299ff4-930f-4255-86f3-01a9c34c9e5b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar96xg\" (UID: \"a2299ff4-930f-4255-86f3-01a9c34c9e5b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar96xg" Oct 30 18:02:33.996556 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:33.996511 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2299ff4-930f-4255-86f3-01a9c34c9e5b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar96xg\" (UID: \"a2299ff4-930f-4255-86f3-01a9c34c9e5b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar96xg" Oct 30 18:02:34.021520 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.021461 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grmmj\" (UniqueName: \"kubernetes.io/projected/a2299ff4-930f-4255-86f3-01a9c34c9e5b-kube-api-access-grmmj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar96xg\" (UID: \"a2299ff4-930f-4255-86f3-01a9c34c9e5b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar96xg" Oct 30 18:02:34.047455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.047201 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9xpmn" Oct 30 18:02:34.095206 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.090717 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2"] Oct 30 18:02:34.098592 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.096850 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45c0dc9a-baa9-40b5-a37d-f08973acbd6a-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777xm2\" (UID: \"45c0dc9a-baa9-40b5-a37d-f08973acbd6a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777xm2" Oct 30 18:02:34.098592 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.096898 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45c0dc9a-baa9-40b5-a37d-f08973acbd6a-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777xm2\" (UID: \"45c0dc9a-baa9-40b5-a37d-f08973acbd6a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777xm2" Oct 30 18:02:34.098592 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.096920 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r584n\" (UniqueName: \"kubernetes.io/projected/45c0dc9a-baa9-40b5-a37d-f08973acbd6a-kube-api-access-r584n\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777xm2\" (UID: \"45c0dc9a-baa9-40b5-a37d-f08973acbd6a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777xm2" Oct 30 18:02:34.098592 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.097651 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45c0dc9a-baa9-40b5-a37d-f08973acbd6a-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777xm2\" (UID: \"45c0dc9a-baa9-40b5-a37d-f08973acbd6a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777xm2" Oct 30 18:02:34.098592 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.097879 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45c0dc9a-baa9-40b5-a37d-f08973acbd6a-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777xm2\" (UID: \"45c0dc9a-baa9-40b5-a37d-f08973acbd6a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777xm2" Oct 30 18:02:34.100315 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.099192 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" Oct 30 18:02:34.122119 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.114999 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2"] Oct 30 18:02:34.122119 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.119689 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 18:02:34.122119 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.119811 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 18:02:34.126954 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.126923 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 18:02:34.127099 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.127034 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-xfzjq\"" Oct 30 18:02:34.136660 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.136634 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r584n\" (UniqueName: \"kubernetes.io/projected/45c0dc9a-baa9-40b5-a37d-f08973acbd6a-kube-api-access-r584n\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777xm2\" (UID: \"45c0dc9a-baa9-40b5-a37d-f08973acbd6a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777xm2" Oct 30 18:02:34.148821 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.147359 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar96xg" Oct 30 18:02:34.198217 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.198179 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e03523d4-1b6f-4a80-9006-26b0345d809d-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-x2rr2\" (UID: \"e03523d4-1b6f-4a80-9006-26b0345d809d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" Oct 30 18:02:34.198392 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.198275 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e03523d4-1b6f-4a80-9006-26b0345d809d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-x2rr2\" (UID: \"e03523d4-1b6f-4a80-9006-26b0345d809d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" Oct 30 18:02:34.198392 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.198327 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckdf6\" (UniqueName: \"kubernetes.io/projected/e03523d4-1b6f-4a80-9006-26b0345d809d-kube-api-access-ckdf6\") pod \"maas-default-gateway-openshift-default-76597d6855-x2rr2\" (UID: \"e03523d4-1b6f-4a80-9006-26b0345d809d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" Oct 30 18:02:34.198392 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.198364 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e03523d4-1b6f-4a80-9006-26b0345d809d-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-x2rr2\" (UID: \"e03523d4-1b6f-4a80-9006-26b0345d809d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" Oct 30 18:02:34.198392 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.198388 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e03523d4-1b6f-4a80-9006-26b0345d809d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-x2rr2\" (UID: \"e03523d4-1b6f-4a80-9006-26b0345d809d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" Oct 30 18:02:34.198608 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.198524 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e03523d4-1b6f-4a80-9006-26b0345d809d-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-x2rr2\" (UID: \"e03523d4-1b6f-4a80-9006-26b0345d809d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" Oct 30 18:02:34.198608 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.198556 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e03523d4-1b6f-4a80-9006-26b0345d809d-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-x2rr2\" (UID: \"e03523d4-1b6f-4a80-9006-26b0345d809d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" Oct 30 18:02:34.198608 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.198581 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e03523d4-1b6f-4a80-9006-26b0345d809d-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-x2rr2\" (UID: \"e03523d4-1b6f-4a80-9006-26b0345d809d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" Oct 30 18:02:34.198790 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.198615 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e03523d4-1b6f-4a80-9006-26b0345d809d-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-x2rr2\" (UID: \"e03523d4-1b6f-4a80-9006-26b0345d809d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" Oct 30 18:02:34.239275 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.239246 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777xm2" Oct 30 18:02:34.258851 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.258826 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c"] Oct 30 18:02:34.272563 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.272534 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmksd"] Oct 30 18:02:34.272730 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.272714 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c" Oct 30 18:02:34.282216 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.282105 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c"] Oct 30 18:02:34.292157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.291990 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-nxhnj\"" Oct 30 18:02:34.299910 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.299350 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e03523d4-1b6f-4a80-9006-26b0345d809d-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-x2rr2\" (UID: \"e03523d4-1b6f-4a80-9006-26b0345d809d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" Oct 30 18:02:34.299910 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.299401 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e03523d4-1b6f-4a80-9006-26b0345d809d-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-x2rr2\" (UID: \"e03523d4-1b6f-4a80-9006-26b0345d809d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" Oct 30 18:02:34.299910 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.299431 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e03523d4-1b6f-4a80-9006-26b0345d809d-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-x2rr2\" (UID: \"e03523d4-1b6f-4a80-9006-26b0345d809d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" Oct 30 18:02:34.299910 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.299464 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e03523d4-1b6f-4a80-9006-26b0345d809d-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-x2rr2\" (UID: \"e03523d4-1b6f-4a80-9006-26b0345d809d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" Oct 30 18:02:34.299910 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.299494 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e03523d4-1b6f-4a80-9006-26b0345d809d-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-x2rr2\" (UID: \"e03523d4-1b6f-4a80-9006-26b0345d809d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" Oct 30 18:02:34.299910 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.299597 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e03523d4-1b6f-4a80-9006-26b0345d809d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-x2rr2\" (UID: \"e03523d4-1b6f-4a80-9006-26b0345d809d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" Oct 30 18:02:34.299910 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.299644 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckdf6\" (UniqueName: \"kubernetes.io/projected/e03523d4-1b6f-4a80-9006-26b0345d809d-kube-api-access-ckdf6\") pod \"maas-default-gateway-openshift-default-76597d6855-x2rr2\" (UID: \"e03523d4-1b6f-4a80-9006-26b0345d809d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" Oct 30 18:02:34.299910 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.299683 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e03523d4-1b6f-4a80-9006-26b0345d809d-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-x2rr2\" (UID: \"e03523d4-1b6f-4a80-9006-26b0345d809d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" Oct 30 18:02:34.299910 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.299709 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e03523d4-1b6f-4a80-9006-26b0345d809d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-x2rr2\" (UID: \"e03523d4-1b6f-4a80-9006-26b0345d809d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" Oct 30 18:02:34.301067 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.301042 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e03523d4-1b6f-4a80-9006-26b0345d809d-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-x2rr2\" (UID: \"e03523d4-1b6f-4a80-9006-26b0345d809d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" Oct 30 18:02:34.301709 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.301281 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e03523d4-1b6f-4a80-9006-26b0345d809d-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-x2rr2\" (UID: \"e03523d4-1b6f-4a80-9006-26b0345d809d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" Oct 30 18:02:34.301709 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.301468 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e03523d4-1b6f-4a80-9006-26b0345d809d-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-x2rr2\" (UID: \"e03523d4-1b6f-4a80-9006-26b0345d809d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" Oct 30 18:02:34.302267 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.302055 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e03523d4-1b6f-4a80-9006-26b0345d809d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-x2rr2\" (UID: \"e03523d4-1b6f-4a80-9006-26b0345d809d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" Oct 30 18:02:34.303015 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.302983 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e03523d4-1b6f-4a80-9006-26b0345d809d-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-x2rr2\" (UID: \"e03523d4-1b6f-4a80-9006-26b0345d809d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" Oct 30 18:02:34.305564 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.305544 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e03523d4-1b6f-4a80-9006-26b0345d809d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-x2rr2\" (UID: \"e03523d4-1b6f-4a80-9006-26b0345d809d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" Oct 30 18:02:34.324223 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.320242 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e03523d4-1b6f-4a80-9006-26b0345d809d-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-x2rr2\" (UID: \"e03523d4-1b6f-4a80-9006-26b0345d809d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" Oct 30 18:02:34.324223 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.323347 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e03523d4-1b6f-4a80-9006-26b0345d809d-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-x2rr2\" (UID: \"e03523d4-1b6f-4a80-9006-26b0345d809d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" Oct 30 18:02:34.324223 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.324100 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckdf6\" (UniqueName: \"kubernetes.io/projected/e03523d4-1b6f-4a80-9006-26b0345d809d-kube-api-access-ckdf6\") pod \"maas-default-gateway-openshift-default-76597d6855-x2rr2\" (UID: \"e03523d4-1b6f-4a80-9006-26b0345d809d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" Oct 30 18:02:34.400385 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.400347 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa4194a4-34ec-4500-a28f-e7a55df72764-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c\" (UID: \"aa4194a4-34ec-4500-a28f-e7a55df72764\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c" Oct 30 18:02:34.400566 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.400401 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa4194a4-34ec-4500-a28f-e7a55df72764-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c\" (UID: \"aa4194a4-34ec-4500-a28f-e7a55df72764\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c" Oct 30 18:02:34.400622 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.400551 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa4194a4-34ec-4500-a28f-e7a55df72764-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c\" (UID: \"aa4194a4-34ec-4500-a28f-e7a55df72764\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c" Oct 30 18:02:34.400622 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.400591 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa4194a4-34ec-4500-a28f-e7a55df72764-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c\" (UID: \"aa4194a4-34ec-4500-a28f-e7a55df72764\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c" Oct 30 18:02:34.400730 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.400623 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa4194a4-34ec-4500-a28f-e7a55df72764-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c\" (UID: \"aa4194a4-34ec-4500-a28f-e7a55df72764\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c" Oct 30 18:02:34.400730 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.400660 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa4194a4-34ec-4500-a28f-e7a55df72764-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c\" (UID: \"aa4194a4-34ec-4500-a28f-e7a55df72764\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c" Oct 30 18:02:34.400730 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.400696 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa4194a4-34ec-4500-a28f-e7a55df72764-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c\" (UID: \"aa4194a4-34ec-4500-a28f-e7a55df72764\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c" Oct 30 18:02:34.400824 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.400779 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnr6d\" (UniqueName: \"kubernetes.io/projected/aa4194a4-34ec-4500-a28f-e7a55df72764-kube-api-access-jnr6d\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c\" (UID: \"aa4194a4-34ec-4500-a28f-e7a55df72764\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c" Oct 30 18:02:34.400876 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.400828 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa4194a4-34ec-4500-a28f-e7a55df72764-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c\" (UID: \"aa4194a4-34ec-4500-a28f-e7a55df72764\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c" Oct 30 18:02:34.412202 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.412172 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9xpmn"] Oct 30 18:02:34.414894 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:02:34.414852 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1db910f_a35a_475f_810e_d6a7a924bcbe.slice/crio-44f2cfdbe083fb2d27065020402e4d82b3dbfc2258437c7894d428097eacf892 WatchSource:0}: Error finding container 44f2cfdbe083fb2d27065020402e4d82b3dbfc2258437c7894d428097eacf892: Status 404 returned error can't find the container with id 44f2cfdbe083fb2d27065020402e4d82b3dbfc2258437c7894d428097eacf892 Oct 30 18:02:34.435807 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.435778 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" Oct 30 18:02:34.468756 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.468657 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar96xg"] Oct 30 18:02:34.474035 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:02:34.473998 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2299ff4_930f_4255_86f3_01a9c34c9e5b.slice/crio-2a432d20e4ff77098d52af6d9e3de00b12809f6bfb596aa82e461001ca28b13e WatchSource:0}: Error finding container 2a432d20e4ff77098d52af6d9e3de00b12809f6bfb596aa82e461001ca28b13e: Status 404 returned error can't find the container with id 2a432d20e4ff77098d52af6d9e3de00b12809f6bfb596aa82e461001ca28b13e Oct 30 18:02:34.501975 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.501795 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa4194a4-34ec-4500-a28f-e7a55df72764-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c\" (UID: \"aa4194a4-34ec-4500-a28f-e7a55df72764\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c" Oct 30 18:02:34.501975 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.501842 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa4194a4-34ec-4500-a28f-e7a55df72764-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c\" (UID: \"aa4194a4-34ec-4500-a28f-e7a55df72764\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c" Oct 30 18:02:34.501975 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.501871 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa4194a4-34ec-4500-a28f-e7a55df72764-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c\" (UID: \"aa4194a4-34ec-4500-a28f-e7a55df72764\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c" Oct 30 18:02:34.501975 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.501888 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa4194a4-34ec-4500-a28f-e7a55df72764-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c\" (UID: \"aa4194a4-34ec-4500-a28f-e7a55df72764\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c" Oct 30 18:02:34.501975 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.501906 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa4194a4-34ec-4500-a28f-e7a55df72764-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c\" (UID: \"aa4194a4-34ec-4500-a28f-e7a55df72764\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c" Oct 30 18:02:34.501975 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.501924 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa4194a4-34ec-4500-a28f-e7a55df72764-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c\" (UID: \"aa4194a4-34ec-4500-a28f-e7a55df72764\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c" Oct 30 18:02:34.501975 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.501945 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa4194a4-34ec-4500-a28f-e7a55df72764-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c\" (UID: \"aa4194a4-34ec-4500-a28f-e7a55df72764\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c" Oct 30 18:02:34.503525 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.501994 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnr6d\" (UniqueName: \"kubernetes.io/projected/aa4194a4-34ec-4500-a28f-e7a55df72764-kube-api-access-jnr6d\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c\" (UID: \"aa4194a4-34ec-4500-a28f-e7a55df72764\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c" Oct 30 18:02:34.503525 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.502022 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa4194a4-34ec-4500-a28f-e7a55df72764-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c\" (UID: \"aa4194a4-34ec-4500-a28f-e7a55df72764\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c" Oct 30 18:02:34.503525 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.502408 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa4194a4-34ec-4500-a28f-e7a55df72764-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c\" (UID: \"aa4194a4-34ec-4500-a28f-e7a55df72764\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c" Oct 30 18:02:34.503525 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.502925 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa4194a4-34ec-4500-a28f-e7a55df72764-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c\" (UID: \"aa4194a4-34ec-4500-a28f-e7a55df72764\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c" Oct 30 18:02:34.503525 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.503090 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa4194a4-34ec-4500-a28f-e7a55df72764-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c\" (UID: \"aa4194a4-34ec-4500-a28f-e7a55df72764\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c" Oct 30 18:02:34.503525 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.503446 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa4194a4-34ec-4500-a28f-e7a55df72764-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c\" (UID: \"aa4194a4-34ec-4500-a28f-e7a55df72764\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c" Oct 30 18:02:34.504434 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.504083 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa4194a4-34ec-4500-a28f-e7a55df72764-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c\" (UID: \"aa4194a4-34ec-4500-a28f-e7a55df72764\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c" Oct 30 18:02:34.506360 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.506335 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa4194a4-34ec-4500-a28f-e7a55df72764-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c\" (UID: \"aa4194a4-34ec-4500-a28f-e7a55df72764\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c" Oct 30 18:02:34.506819 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.506792 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa4194a4-34ec-4500-a28f-e7a55df72764-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c\" (UID: \"aa4194a4-34ec-4500-a28f-e7a55df72764\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c" Oct 30 18:02:34.531413 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.531380 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa4194a4-34ec-4500-a28f-e7a55df72764-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c\" (UID: \"aa4194a4-34ec-4500-a28f-e7a55df72764\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c" Oct 30 18:02:34.534384 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.534355 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnr6d\" (UniqueName: \"kubernetes.io/projected/aa4194a4-34ec-4500-a28f-e7a55df72764-kube-api-access-jnr6d\") pod \"openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c\" (UID: \"aa4194a4-34ec-4500-a28f-e7a55df72764\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c" Oct 30 18:02:34.559615 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.559534 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777xm2"] Oct 30 18:02:34.563177 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:02:34.563116 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45c0dc9a_baa9_40b5_a37d_f08973acbd6a.slice/crio-4aa09c9ab18605145502fd9733a8e5d98f7da65fd546c58a522d92fc6cfe1ad4 WatchSource:0}: Error finding container 4aa09c9ab18605145502fd9733a8e5d98f7da65fd546c58a522d92fc6cfe1ad4: Status 404 returned error can't find the container with id 4aa09c9ab18605145502fd9733a8e5d98f7da65fd546c58a522d92fc6cfe1ad4 Oct 30 18:02:34.598415 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.598381 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c" Oct 30 18:02:34.746298 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.746190 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2"] Oct 30 18:02:34.756505 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:02:34.756463 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode03523d4_1b6f_4a80_9006_26b0345d809d.slice/crio-ba3c9a9a0977439c5e6c9338c7011c57f2571ae0ab7ac0670bbd13eeb06fad67 WatchSource:0}: Error finding container ba3c9a9a0977439c5e6c9338c7011c57f2571ae0ab7ac0670bbd13eeb06fad67: Status 404 returned error can't find the container with id ba3c9a9a0977439c5e6c9338c7011c57f2571ae0ab7ac0670bbd13eeb06fad67 Oct 30 18:02:34.937181 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.937003 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c"] Oct 30 18:02:34.946214 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:02:34.946167 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa4194a4_34ec_4500_a28f_e7a55df72764.slice/crio-0b84fe2c5d0f20881c302b58df7f1719848c06c3618d51a69bd76a58aa3f2310 WatchSource:0}: Error finding container 0b84fe2c5d0f20881c302b58df7f1719848c06c3618d51a69bd76a58aa3f2310: Status 404 returned error can't find the container with id 0b84fe2c5d0f20881c302b58df7f1719848c06c3618d51a69bd76a58aa3f2310 Oct 30 18:02:34.977341 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.977303 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9xpmn" event={"ID":"c1db910f-a35a-475f-810e-d6a7a924bcbe","Type":"ContainerStarted","Data":"44f2cfdbe083fb2d27065020402e4d82b3dbfc2258437c7894d428097eacf892"} Oct 30 18:02:34.979758 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.979714 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmksd" event={"ID":"8f953747-4491-40e8-972d-e54b3951983b","Type":"ContainerStarted","Data":"d0137f01c2e0edd18a500c0d66c765a7bc5d5e9b821c50c06f67d4373f210074"} Oct 30 18:02:34.981536 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.981512 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar96xg" event={"ID":"a2299ff4-930f-4255-86f3-01a9c34c9e5b","Type":"ContainerStarted","Data":"2a432d20e4ff77098d52af6d9e3de00b12809f6bfb596aa82e461001ca28b13e"} Oct 30 18:02:34.983830 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.983799 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c" event={"ID":"aa4194a4-34ec-4500-a28f-e7a55df72764","Type":"ContainerStarted","Data":"0b84fe2c5d0f20881c302b58df7f1719848c06c3618d51a69bd76a58aa3f2310"} Oct 30 18:02:34.985792 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.985766 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" event={"ID":"e03523d4-1b6f-4a80-9006-26b0345d809d","Type":"ContainerStarted","Data":"ba3c9a9a0977439c5e6c9338c7011c57f2571ae0ab7ac0670bbd13eeb06fad67"} Oct 30 18:02:34.988631 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:34.988604 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777xm2" event={"ID":"45c0dc9a-baa9-40b5-a37d-f08973acbd6a","Type":"ContainerStarted","Data":"4aa09c9ab18605145502fd9733a8e5d98f7da65fd546c58a522d92fc6cfe1ad4"} Oct 30 18:02:35.922350 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:35.920424 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpvcl"] Oct 30 18:02:35.925413 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:35.924680 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpvcl" Oct 30 18:02:35.952731 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:35.952644 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpvcl"] Oct 30 18:02:36.015008 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:36.014692 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p846h\" (UniqueName: \"kubernetes.io/projected/4d1d6538-607a-4eec-881c-d1a189067df6-kube-api-access-p846h\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpvcl\" (UID: \"4d1d6538-607a-4eec-881c-d1a189067df6\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpvcl" Oct 30 18:02:36.015008 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:36.014768 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d1d6538-607a-4eec-881c-d1a189067df6-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpvcl\" (UID: \"4d1d6538-607a-4eec-881c-d1a189067df6\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpvcl" Oct 30 18:02:36.015008 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:36.014808 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d1d6538-607a-4eec-881c-d1a189067df6-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpvcl\" (UID: \"4d1d6538-607a-4eec-881c-d1a189067df6\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpvcl" Oct 30 18:02:36.117557 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:36.116352 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d1d6538-607a-4eec-881c-d1a189067df6-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpvcl\" (UID: \"4d1d6538-607a-4eec-881c-d1a189067df6\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpvcl" Oct 30 18:02:36.117557 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:36.116471 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p846h\" (UniqueName: \"kubernetes.io/projected/4d1d6538-607a-4eec-881c-d1a189067df6-kube-api-access-p846h\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpvcl\" (UID: \"4d1d6538-607a-4eec-881c-d1a189067df6\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpvcl" Oct 30 18:02:36.117557 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:36.116528 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d1d6538-607a-4eec-881c-d1a189067df6-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpvcl\" (UID: \"4d1d6538-607a-4eec-881c-d1a189067df6\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpvcl" Oct 30 18:02:36.117557 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:36.116943 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d1d6538-607a-4eec-881c-d1a189067df6-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpvcl\" (UID: \"4d1d6538-607a-4eec-881c-d1a189067df6\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpvcl" Oct 30 18:02:36.117557 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:36.117203 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d1d6538-607a-4eec-881c-d1a189067df6-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpvcl\" (UID: \"4d1d6538-607a-4eec-881c-d1a189067df6\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpvcl" Oct 30 18:02:36.160212 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:36.160173 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p846h\" (UniqueName: \"kubernetes.io/projected/4d1d6538-607a-4eec-881c-d1a189067df6-kube-api-access-p846h\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpvcl\" (UID: \"4d1d6538-607a-4eec-881c-d1a189067df6\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpvcl" Oct 30 18:02:36.206205 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:36.205580 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mmvlt"] Oct 30 18:02:36.210678 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:36.210657 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mmvlt" Oct 30 18:02:36.229546 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:36.228608 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mmvlt"] Oct 30 18:02:36.246551 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:36.246518 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpvcl" Oct 30 18:02:36.317916 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:36.317608 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnz6m\" (UniqueName: \"kubernetes.io/projected/16f76c49-259b-4c9b-bf5a-abc20a23ac41-kube-api-access-mnz6m\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mmvlt\" (UID: \"16f76c49-259b-4c9b-bf5a-abc20a23ac41\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mmvlt" Oct 30 18:02:36.317916 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:36.317686 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16f76c49-259b-4c9b-bf5a-abc20a23ac41-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mmvlt\" (UID: \"16f76c49-259b-4c9b-bf5a-abc20a23ac41\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mmvlt" Oct 30 18:02:36.317916 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:36.317793 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16f76c49-259b-4c9b-bf5a-abc20a23ac41-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mmvlt\" (UID: \"16f76c49-259b-4c9b-bf5a-abc20a23ac41\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mmvlt" Oct 30 18:02:36.419453 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:36.419412 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16f76c49-259b-4c9b-bf5a-abc20a23ac41-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mmvlt\" (UID: \"16f76c49-259b-4c9b-bf5a-abc20a23ac41\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mmvlt" Oct 30 18:02:36.419638 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:36.419547 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16f76c49-259b-4c9b-bf5a-abc20a23ac41-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mmvlt\" (UID: \"16f76c49-259b-4c9b-bf5a-abc20a23ac41\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mmvlt" Oct 30 18:02:36.419638 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:36.419591 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnz6m\" (UniqueName: \"kubernetes.io/projected/16f76c49-259b-4c9b-bf5a-abc20a23ac41-kube-api-access-mnz6m\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mmvlt\" (UID: \"16f76c49-259b-4c9b-bf5a-abc20a23ac41\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mmvlt" Oct 30 18:02:36.420617 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:36.420521 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16f76c49-259b-4c9b-bf5a-abc20a23ac41-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mmvlt\" (UID: \"16f76c49-259b-4c9b-bf5a-abc20a23ac41\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mmvlt" Oct 30 18:02:36.421290 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:36.421223 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16f76c49-259b-4c9b-bf5a-abc20a23ac41-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mmvlt\" (UID: \"16f76c49-259b-4c9b-bf5a-abc20a23ac41\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mmvlt" Oct 30 18:02:36.448247 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:36.448180 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnz6m\" (UniqueName: \"kubernetes.io/projected/16f76c49-259b-4c9b-bf5a-abc20a23ac41-kube-api-access-mnz6m\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mmvlt\" (UID: \"16f76c49-259b-4c9b-bf5a-abc20a23ac41\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mmvlt" Oct 30 18:02:36.540417 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:36.540102 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mmvlt" Oct 30 18:02:36.570842 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:36.570797 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpvcl"] Oct 30 18:02:36.920293 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:36.920206 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mmvlt"] Oct 30 18:02:36.934606 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:02:36.934542 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16f76c49_259b_4c9b_bf5a_abc20a23ac41.slice/crio-a31f0c1881e4a79f45eb4445a66ffda2ff312d3f649b2a521d2584ecfa7a0df5 WatchSource:0}: Error finding container a31f0c1881e4a79f45eb4445a66ffda2ff312d3f649b2a521d2584ecfa7a0df5: Status 404 returned error can't find the container with id a31f0c1881e4a79f45eb4445a66ffda2ff312d3f649b2a521d2584ecfa7a0df5 Oct 30 18:02:37.013433 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:37.013335 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mmvlt" event={"ID":"16f76c49-259b-4c9b-bf5a-abc20a23ac41","Type":"ContainerStarted","Data":"a31f0c1881e4a79f45eb4445a66ffda2ff312d3f649b2a521d2584ecfa7a0df5"} Oct 30 18:02:37.026289 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:37.025703 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpvcl" event={"ID":"4d1d6538-607a-4eec-881c-d1a189067df6","Type":"ContainerStarted","Data":"edf0b5e34262c883e72050324e7d28188a896c4ccb97d6309ae9a716f0ca03e2"} Oct 30 18:02:44.678038 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:44.677490 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:02:44.678038 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:44.677578 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:02:44.678038 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:44.677623 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:02:44.860341 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:44.859869 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:02:44.860341 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:44.859934 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:02:44.860341 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:44.859970 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:02:45.122196 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:45.121251 4442 generic.go:356] "Generic (PLEG): container finished" podID="cd769ce4-6c17-47da-be0a-fdb8bbc7f76b" containerID="5d1df8ed87ade16fcf7aa8614eeb53a7558bd2f0ac1b6b26ef7508315e22d2ac" exitCode=0 Oct 30 18:02:45.122196 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:45.121954 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5ddml" event={"ID":"cd769ce4-6c17-47da-be0a-fdb8bbc7f76b","Type":"ContainerDied","Data":"5d1df8ed87ade16fcf7aa8614eeb53a7558bd2f0ac1b6b26ef7508315e22d2ac"} Oct 30 18:02:45.132499 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:45.131025 4442 generic.go:356] "Generic (PLEG): container finished" podID="2442891f-bd5e-47cd-b882-2121cdd96cd0" containerID="3e5a483e63b1095ff448758ecae0689b7f82cb31b282210bf78d0c5058ef9d10" exitCode=0 Oct 30 18:02:45.132499 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:45.131118 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tfns" event={"ID":"2442891f-bd5e-47cd-b882-2121cdd96cd0","Type":"ContainerDied","Data":"3e5a483e63b1095ff448758ecae0689b7f82cb31b282210bf78d0c5058ef9d10"} Oct 30 18:02:45.144126 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:45.143468 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c" event={"ID":"aa4194a4-34ec-4500-a28f-e7a55df72764","Type":"ContainerStarted","Data":"0636b028f567b9a4fffefdaafff3024063558bb25d520ec4302181ecd172df86"} Oct 30 18:02:45.149572 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:45.148853 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" event={"ID":"e03523d4-1b6f-4a80-9006-26b0345d809d","Type":"ContainerStarted","Data":"f32a9e2f167f134d719f6dc9b1f94211c22dbd5921034d7500cccb171b8b3321"} Oct 30 18:02:45.270413 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:45.269018 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" podStartSLOduration=1.3525886 podStartE2EDuration="11.268995017s" totalImagesPullingTime="9.916406417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:02:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:02:45.260528295 +0000 UTC m=+262.350590634" watchObservedRunningTime="2026-10-30 18:02:45.268995017 +0000 UTC m=+262.359057359" Oct 30 18:02:45.303834 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:45.302984 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c" podStartSLOduration=1.393072027 podStartE2EDuration="11.302963925s" totalImagesPullingTime="9.909891898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:02:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:02:45.301669382 +0000 UTC m=+262.391731743" watchObservedRunningTime="2026-10-30 18:02:45.302963925 +0000 UTC m=+262.393026264" Oct 30 18:02:45.437007 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:45.436939 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" Oct 30 18:02:45.442794 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:45.442555 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" Oct 30 18:02:45.598708 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:45.598624 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c" Oct 30 18:02:46.157342 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:46.157268 4442 generic.go:356] "Generic (PLEG): container finished" podID="45c0dc9a-baa9-40b5-a37d-f08973acbd6a" containerID="89d0a06d791bddb8f6892d836c9b1413f78d980ad6124f7a96c83ad6270e4484" exitCode=0 Oct 30 18:02:46.157831 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:46.157380 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777xm2" event={"ID":"45c0dc9a-baa9-40b5-a37d-f08973acbd6a","Type":"ContainerDied","Data":"89d0a06d791bddb8f6892d836c9b1413f78d980ad6124f7a96c83ad6270e4484"} Oct 30 18:02:46.162740 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:46.162717 4442 generic.go:356] "Generic (PLEG): container finished" podID="c1db910f-a35a-475f-810e-d6a7a924bcbe" containerID="90113c4403165923c05dddf18b4567312ef195e32017afe247bb7f1c0700dfba" exitCode=0 Oct 30 18:02:46.162837 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:46.162794 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9xpmn" event={"ID":"c1db910f-a35a-475f-810e-d6a7a924bcbe","Type":"ContainerDied","Data":"90113c4403165923c05dddf18b4567312ef195e32017afe247bb7f1c0700dfba"} Oct 30 18:02:46.167255 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:46.167174 4442 generic.go:356] "Generic (PLEG): container finished" podID="8f953747-4491-40e8-972d-e54b3951983b" containerID="74a7565f6eab803937035d00b082d52566df372cbcb52b8a6167a87504dc5ecb" exitCode=0 Oct 30 18:02:46.167320 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:46.167267 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmksd" event={"ID":"8f953747-4491-40e8-972d-e54b3951983b","Type":"ContainerDied","Data":"74a7565f6eab803937035d00b082d52566df372cbcb52b8a6167a87504dc5ecb"} Oct 30 18:02:46.171766 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:46.171637 4442 generic.go:356] "Generic (PLEG): container finished" podID="a2299ff4-930f-4255-86f3-01a9c34c9e5b" containerID="faaf17caed170a8b857de6d90893a4583ddb6a24cb7a480f286ff5656c789e8c" exitCode=0 Oct 30 18:02:46.171766 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:46.171672 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar96xg" event={"ID":"a2299ff4-930f-4255-86f3-01a9c34c9e5b","Type":"ContainerDied","Data":"faaf17caed170a8b857de6d90893a4583ddb6a24cb7a480f286ff5656c789e8c"} Oct 30 18:02:46.175672 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:46.175648 4442 generic.go:356] "Generic (PLEG): container finished" podID="2442891f-bd5e-47cd-b882-2121cdd96cd0" containerID="fddc0e5d11114bcce34fad98a7e2b2d2bdaaabe68ec66449879c5de59de5ff23" exitCode=0 Oct 30 18:02:46.175781 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:46.175751 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tfns" event={"ID":"2442891f-bd5e-47cd-b882-2121cdd96cd0","Type":"ContainerDied","Data":"fddc0e5d11114bcce34fad98a7e2b2d2bdaaabe68ec66449879c5de59de5ff23"} Oct 30 18:02:46.179244 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:46.179224 4442 generic.go:356] "Generic (PLEG): container finished" podID="452375be-3092-46cf-a3e4-56111a75fb24" containerID="22e12ca52bb267ddc8e9f3c5e74caa95550dd9e165bfb2be3e21c246d4185c85" exitCode=0 Oct 30 18:02:46.180961 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:46.180939 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hdwk9" event={"ID":"452375be-3092-46cf-a3e4-56111a75fb24","Type":"ContainerDied","Data":"22e12ca52bb267ddc8e9f3c5e74caa95550dd9e165bfb2be3e21c246d4185c85"} Oct 30 18:02:46.181099 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:46.180983 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" Oct 30 18:02:46.182690 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:46.182668 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" Oct 30 18:02:46.605054 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:46.604983 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c" Oct 30 18:02:47.198166 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:47.196502 4442 generic.go:356] "Generic (PLEG): container finished" podID="45c0dc9a-baa9-40b5-a37d-f08973acbd6a" containerID="facbdfb887d94e34bfbfcff83e97e7d5a1c10dbda96e4660b4716c5711545ffa" exitCode=0 Oct 30 18:02:47.198166 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:47.196604 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777xm2" event={"ID":"45c0dc9a-baa9-40b5-a37d-f08973acbd6a","Type":"ContainerDied","Data":"facbdfb887d94e34bfbfcff83e97e7d5a1c10dbda96e4660b4716c5711545ffa"} Oct 30 18:02:47.200736 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:47.200631 4442 generic.go:356] "Generic (PLEG): container finished" podID="8f953747-4491-40e8-972d-e54b3951983b" containerID="89eddd4b464e48c393b57a6cee7315c0bb92e42ac75655291ade0ba5081f7d23" exitCode=0 Oct 30 18:02:47.200736 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:47.200712 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmksd" event={"ID":"8f953747-4491-40e8-972d-e54b3951983b","Type":"ContainerDied","Data":"89eddd4b464e48c393b57a6cee7315c0bb92e42ac75655291ade0ba5081f7d23"} Oct 30 18:02:47.205037 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:47.205011 4442 generic.go:356] "Generic (PLEG): container finished" podID="4d1d6538-607a-4eec-881c-d1a189067df6" containerID="18922ec32f71cd70c72f6a0b011c7cc521d37f263d00b5baa25b58b620c5a9d4" exitCode=0 Oct 30 18:02:47.205127 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:47.205089 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpvcl" event={"ID":"4d1d6538-607a-4eec-881c-d1a189067df6","Type":"ContainerDied","Data":"18922ec32f71cd70c72f6a0b011c7cc521d37f263d00b5baa25b58b620c5a9d4"} Oct 30 18:02:47.205920 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:47.205901 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c" Oct 30 18:02:47.206941 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:47.206924 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c" Oct 30 18:02:47.574941 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:02:47.574654 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:02:48.220181 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:48.219975 4442 generic.go:356] "Generic (PLEG): container finished" podID="16f76c49-259b-4c9b-bf5a-abc20a23ac41" containerID="d51625285c416bd2cbeaebb02a61e4ebe546a1b1dcd6cfc1140e038ce4ea8115" exitCode=0 Oct 30 18:02:48.220181 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:48.220085 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mmvlt" event={"ID":"16f76c49-259b-4c9b-bf5a-abc20a23ac41","Type":"ContainerDied","Data":"d51625285c416bd2cbeaebb02a61e4ebe546a1b1dcd6cfc1140e038ce4ea8115"} Oct 30 18:02:48.228299 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:48.227318 4442 generic.go:356] "Generic (PLEG): container finished" podID="a2299ff4-930f-4255-86f3-01a9c34c9e5b" containerID="cd5f1e8e9ee5225149e7bb0714998e388be633fd72af5d822cbebec6d1d3faad" exitCode=0 Oct 30 18:02:48.228299 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:48.227477 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar96xg" event={"ID":"a2299ff4-930f-4255-86f3-01a9c34c9e5b","Type":"ContainerDied","Data":"cd5f1e8e9ee5225149e7bb0714998e388be633fd72af5d822cbebec6d1d3faad"} Oct 30 18:02:48.236635 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:48.233285 4442 generic.go:356] "Generic (PLEG): container finished" podID="c1db910f-a35a-475f-810e-d6a7a924bcbe" containerID="cebf0ba459fe41e9aae162a7840798feb0009856d1475c13b1f91a582256db87" exitCode=0 Oct 30 18:02:48.236635 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:48.234436 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9xpmn" event={"ID":"c1db910f-a35a-475f-810e-d6a7a924bcbe","Type":"ContainerDied","Data":"cebf0ba459fe41e9aae162a7840798feb0009856d1475c13b1f91a582256db87"} Oct 30 18:02:49.255096 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:49.253952 4442 generic.go:356] "Generic (PLEG): container finished" podID="16f76c49-259b-4c9b-bf5a-abc20a23ac41" containerID="5863f7d43cc835af8fbbcb18bb111f362f5c866a1354cb5fd2b37ebe1f1ce08d" exitCode=0 Oct 30 18:02:49.255096 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:49.254037 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mmvlt" event={"ID":"16f76c49-259b-4c9b-bf5a-abc20a23ac41","Type":"ContainerDied","Data":"5863f7d43cc835af8fbbcb18bb111f362f5c866a1354cb5fd2b37ebe1f1ce08d"} Oct 30 18:02:49.265335 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:49.264209 4442 generic.go:356] "Generic (PLEG): container finished" podID="cd769ce4-6c17-47da-be0a-fdb8bbc7f76b" containerID="837c6b59d61d4d77997027661302599618050f579e80706472c19a93fb87e7ca" exitCode=0 Oct 30 18:02:49.265335 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:49.265189 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5ddml" event={"ID":"cd769ce4-6c17-47da-be0a-fdb8bbc7f76b","Type":"ContainerDied","Data":"837c6b59d61d4d77997027661302599618050f579e80706472c19a93fb87e7ca"} Oct 30 18:02:52.300844 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:52.300803 4442 generic.go:356] "Generic (PLEG): container finished" podID="452375be-3092-46cf-a3e4-56111a75fb24" containerID="5a9eb600701bc13c298746b29e2ba0d91e8feb8f78823f17a58219039fecf146" exitCode=0 Oct 30 18:02:52.301518 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:52.301495 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hdwk9" event={"ID":"452375be-3092-46cf-a3e4-56111a75fb24","Type":"ContainerDied","Data":"5a9eb600701bc13c298746b29e2ba0d91e8feb8f78823f17a58219039fecf146"} Oct 30 18:02:53.326274 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:53.325358 4442 generic.go:356] "Generic (PLEG): container finished" podID="4d1d6538-607a-4eec-881c-d1a189067df6" containerID="5a59b2dfd3e3d083845e84a7617f7d7c1e3ed24e8a1a024addc624d23bab7931" exitCode=0 Oct 30 18:02:53.326274 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:02:53.325405 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpvcl" event={"ID":"4d1d6538-607a-4eec-881c-d1a189067df6","Type":"ContainerDied","Data":"5a59b2dfd3e3d083845e84a7617f7d7c1e3ed24e8a1a024addc624d23bab7931"} Oct 30 18:03:04.457174 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:04.456478 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tfns" event={"ID":"2442891f-bd5e-47cd-b882-2121cdd96cd0","Type":"ContainerStarted","Data":"3a8f0ca9306abbd001e9c2c4c292045665074bc4de71a0615803b7ac25c0fa4e"} Oct 30 18:03:04.500601 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:04.499888 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tfns" podStartSLOduration=3.114539993 podStartE2EDuration="35.499868085s" totalImagesPullingTime="32.385328092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:02:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:04.497864199 +0000 UTC m=+281.587926564" watchObservedRunningTime="2026-10-30 18:03:04.499868085 +0000 UTC m=+281.589930428" Oct 30 18:03:05.465603 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:05.465564 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmksd" event={"ID":"8f953747-4491-40e8-972d-e54b3951983b","Type":"ContainerStarted","Data":"333cd23bf9efae97ae3e3b134d0ed91647733abf27f4ee010e9c71efa37eddac"} Oct 30 18:03:05.468344 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:05.468312 4442 generic.go:356] "Generic (PLEG): container finished" podID="4d1d6538-607a-4eec-881c-d1a189067df6" containerID="9e6184a65b19e38ef3127503e9d02d3606a7d52b7a1d05c76362e5a016ae6155" exitCode=0 Oct 30 18:03:05.468484 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:05.468402 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpvcl" event={"ID":"4d1d6538-607a-4eec-881c-d1a189067df6","Type":"ContainerDied","Data":"9e6184a65b19e38ef3127503e9d02d3606a7d52b7a1d05c76362e5a016ae6155"} Oct 30 18:03:05.471508 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:05.471278 4442 generic.go:356] "Generic (PLEG): container finished" podID="2442891f-bd5e-47cd-b882-2121cdd96cd0" containerID="3a8f0ca9306abbd001e9c2c4c292045665074bc4de71a0615803b7ac25c0fa4e" exitCode=0 Oct 30 18:03:05.471508 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:05.471452 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tfns" event={"ID":"2442891f-bd5e-47cd-b882-2121cdd96cd0","Type":"ContainerDied","Data":"3a8f0ca9306abbd001e9c2c4c292045665074bc4de71a0615803b7ac25c0fa4e"} Oct 30 18:03:05.474674 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:05.474646 4442 generic.go:356] "Generic (PLEG): container finished" podID="452375be-3092-46cf-a3e4-56111a75fb24" containerID="7ad2cf3d8c1f7b391d73e0c2e8d03b9948fb83d97d8f90bffab8d992fcd506f2" exitCode=0 Oct 30 18:03:05.474790 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:05.474687 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hdwk9" event={"ID":"452375be-3092-46cf-a3e4-56111a75fb24","Type":"ContainerDied","Data":"7ad2cf3d8c1f7b391d73e0c2e8d03b9948fb83d97d8f90bffab8d992fcd506f2"} Oct 30 18:03:05.505072 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:05.504978 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmksd" podStartSLOduration=2.528595662 podStartE2EDuration="32.504959317s" totalImagesPullingTime="29.976363655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:02:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:05.50331125 +0000 UTC m=+282.593373590" watchObservedRunningTime="2026-10-30 18:03:05.504959317 +0000 UTC m=+282.595021653" Oct 30 18:03:06.482741 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:06.482704 4442 generic.go:356] "Generic (PLEG): container finished" podID="8f953747-4491-40e8-972d-e54b3951983b" containerID="333cd23bf9efae97ae3e3b134d0ed91647733abf27f4ee010e9c71efa37eddac" exitCode=0 Oct 30 18:03:06.483197 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:06.482829 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmksd" event={"ID":"8f953747-4491-40e8-972d-e54b3951983b","Type":"ContainerDied","Data":"333cd23bf9efae97ae3e3b134d0ed91647733abf27f4ee010e9c71efa37eddac"} Oct 30 18:03:06.485999 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:06.485974 4442 generic.go:356] "Generic (PLEG): container finished" podID="a2299ff4-930f-4255-86f3-01a9c34c9e5b" containerID="599d5ec62f676901ac01e55bfacbd1cc1a061522cc93cb657370b59a7b452122" exitCode=0 Oct 30 18:03:06.486271 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:06.486251 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar96xg" event={"ID":"a2299ff4-930f-4255-86f3-01a9c34c9e5b","Type":"ContainerDied","Data":"599d5ec62f676901ac01e55bfacbd1cc1a061522cc93cb657370b59a7b452122"} Oct 30 18:03:06.490021 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:06.489999 4442 generic.go:356] "Generic (PLEG): container finished" podID="45c0dc9a-baa9-40b5-a37d-f08973acbd6a" containerID="05f3aa8e5b09e6df317b1f4a2e05bd772dbedfb652a4a052def6f517db105c57" exitCode=0 Oct 30 18:03:06.490154 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:06.490093 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777xm2" event={"ID":"45c0dc9a-baa9-40b5-a37d-f08973acbd6a","Type":"ContainerDied","Data":"05f3aa8e5b09e6df317b1f4a2e05bd772dbedfb652a4a052def6f517db105c57"} Oct 30 18:03:06.493385 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:06.493189 4442 generic.go:356] "Generic (PLEG): container finished" podID="c1db910f-a35a-475f-810e-d6a7a924bcbe" containerID="547674f5918b701b2bac6c24e8d0e246e35871d84a3d8acde311d682c2b2736a" exitCode=0 Oct 30 18:03:06.493385 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:06.493289 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9xpmn" event={"ID":"c1db910f-a35a-475f-810e-d6a7a924bcbe","Type":"ContainerDied","Data":"547674f5918b701b2bac6c24e8d0e246e35871d84a3d8acde311d682c2b2736a"} Oct 30 18:03:06.686054 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:06.686025 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpvcl" Oct 30 18:03:06.710422 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:06.710397 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tfns" Oct 30 18:03:06.722817 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:06.722792 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hdwk9" Oct 30 18:03:06.750635 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:06.750599 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d1d6538-607a-4eec-881c-d1a189067df6-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d1d6538-607a-4eec-881c-d1a189067df6-bundle\") pod \"4d1d6538-607a-4eec-881c-d1a189067df6\" (UID: \"4d1d6538-607a-4eec-881c-d1a189067df6\") " Oct 30 18:03:06.750635 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:06.750636 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/452375be-3092-46cf-a3e4-56111a75fb24-util\" (UniqueName: \"kubernetes.io/empty-dir/452375be-3092-46cf-a3e4-56111a75fb24-util\") pod \"452375be-3092-46cf-a3e4-56111a75fb24\" (UID: \"452375be-3092-46cf-a3e4-56111a75fb24\") " Oct 30 18:03:06.751047 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:06.750655 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2442891f-bd5e-47cd-b882-2121cdd96cd0-util\" (UniqueName: \"kubernetes.io/empty-dir/2442891f-bd5e-47cd-b882-2121cdd96cd0-util\") pod \"2442891f-bd5e-47cd-b882-2121cdd96cd0\" (UID: \"2442891f-bd5e-47cd-b882-2121cdd96cd0\") " Oct 30 18:03:06.751047 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:06.750677 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2442891f-bd5e-47cd-b882-2121cdd96cd0-kube-api-access-qmmxp\" (UniqueName: \"kubernetes.io/projected/2442891f-bd5e-47cd-b882-2121cdd96cd0-kube-api-access-qmmxp\") pod \"2442891f-bd5e-47cd-b882-2121cdd96cd0\" (UID: \"2442891f-bd5e-47cd-b882-2121cdd96cd0\") " Oct 30 18:03:06.751047 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:06.750702 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d1d6538-607a-4eec-881c-d1a189067df6-util\" (UniqueName: \"kubernetes.io/empty-dir/4d1d6538-607a-4eec-881c-d1a189067df6-util\") pod \"4d1d6538-607a-4eec-881c-d1a189067df6\" (UID: \"4d1d6538-607a-4eec-881c-d1a189067df6\") " Oct 30 18:03:06.751047 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:06.750781 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/452375be-3092-46cf-a3e4-56111a75fb24-bundle\" (UniqueName: \"kubernetes.io/empty-dir/452375be-3092-46cf-a3e4-56111a75fb24-bundle\") pod \"452375be-3092-46cf-a3e4-56111a75fb24\" (UID: \"452375be-3092-46cf-a3e4-56111a75fb24\") " Oct 30 18:03:06.751047 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:06.750818 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d1d6538-607a-4eec-881c-d1a189067df6-kube-api-access-p846h\" (UniqueName: \"kubernetes.io/projected/4d1d6538-607a-4eec-881c-d1a189067df6-kube-api-access-p846h\") pod \"4d1d6538-607a-4eec-881c-d1a189067df6\" (UID: \"4d1d6538-607a-4eec-881c-d1a189067df6\") " Oct 30 18:03:06.751047 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:06.750845 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/452375be-3092-46cf-a3e4-56111a75fb24-kube-api-access-2zs8r\" (UniqueName: \"kubernetes.io/projected/452375be-3092-46cf-a3e4-56111a75fb24-kube-api-access-2zs8r\") pod \"452375be-3092-46cf-a3e4-56111a75fb24\" (UID: \"452375be-3092-46cf-a3e4-56111a75fb24\") " Oct 30 18:03:06.751377 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:06.751097 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d1d6538-607a-4eec-881c-d1a189067df6-bundle" pod "4d1d6538-607a-4eec-881c-d1a189067df6" (UID: "4d1d6538-607a-4eec-881c-d1a189067df6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:06.752750 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:06.752719 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452375be-3092-46cf-a3e4-56111a75fb24-bundle" pod "452375be-3092-46cf-a3e4-56111a75fb24" (UID: "452375be-3092-46cf-a3e4-56111a75fb24"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:06.755161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:06.755112 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d1d6538-607a-4eec-881c-d1a189067df6-kube-api-access-p846h" pod "4d1d6538-607a-4eec-881c-d1a189067df6" (UID: "4d1d6538-607a-4eec-881c-d1a189067df6"). InnerVolumeSpecName "kube-api-access-p846h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:03:06.755611 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:06.755579 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/452375be-3092-46cf-a3e4-56111a75fb24-kube-api-access-2zs8r" pod "452375be-3092-46cf-a3e4-56111a75fb24" (UID: "452375be-3092-46cf-a3e4-56111a75fb24"). InnerVolumeSpecName "kube-api-access-2zs8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:03:06.756915 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:06.756896 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d1d6538-607a-4eec-881c-d1a189067df6-util" pod "4d1d6538-607a-4eec-881c-d1a189067df6" (UID: "4d1d6538-607a-4eec-881c-d1a189067df6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:06.757841 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:06.757820 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2442891f-bd5e-47cd-b882-2121cdd96cd0-util" pod "2442891f-bd5e-47cd-b882-2121cdd96cd0" (UID: "2442891f-bd5e-47cd-b882-2121cdd96cd0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:06.758717 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:06.758690 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2442891f-bd5e-47cd-b882-2121cdd96cd0-kube-api-access-qmmxp" pod "2442891f-bd5e-47cd-b882-2121cdd96cd0" (UID: "2442891f-bd5e-47cd-b882-2121cdd96cd0"). InnerVolumeSpecName "kube-api-access-qmmxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:03:06.760046 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:06.760022 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452375be-3092-46cf-a3e4-56111a75fb24-util" pod "452375be-3092-46cf-a3e4-56111a75fb24" (UID: "452375be-3092-46cf-a3e4-56111a75fb24"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:06.851767 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:06.851727 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2442891f-bd5e-47cd-b882-2121cdd96cd0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2442891f-bd5e-47cd-b882-2121cdd96cd0-bundle\") pod \"2442891f-bd5e-47cd-b882-2121cdd96cd0\" (UID: \"2442891f-bd5e-47cd-b882-2121cdd96cd0\") " Oct 30 18:03:06.852029 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:06.852010 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmmxp\" (UniqueName: \"kubernetes.io/projected/2442891f-bd5e-47cd-b882-2121cdd96cd0-kube-api-access-qmmxp\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:03:06.852086 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:06.852036 4442 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d1d6538-607a-4eec-881c-d1a189067df6-util\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:03:06.852086 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:06.852052 4442 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/452375be-3092-46cf-a3e4-56111a75fb24-bundle\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:03:06.852086 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:06.852066 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p846h\" (UniqueName: \"kubernetes.io/projected/4d1d6538-607a-4eec-881c-d1a189067df6-kube-api-access-p846h\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:03:06.852086 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:06.852081 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zs8r\" (UniqueName: \"kubernetes.io/projected/452375be-3092-46cf-a3e4-56111a75fb24-kube-api-access-2zs8r\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:03:06.852234 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:06.852094 4442 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d1d6538-607a-4eec-881c-d1a189067df6-bundle\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:03:06.852234 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:06.852107 4442 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/452375be-3092-46cf-a3e4-56111a75fb24-util\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:03:06.852234 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:06.852118 4442 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2442891f-bd5e-47cd-b882-2121cdd96cd0-util\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:03:06.852319 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:06.852246 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2442891f-bd5e-47cd-b882-2121cdd96cd0-bundle" pod "2442891f-bd5e-47cd-b882-2121cdd96cd0" (UID: "2442891f-bd5e-47cd-b882-2121cdd96cd0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:06.952848 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:06.952770 4442 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2442891f-bd5e-47cd-b882-2121cdd96cd0-bundle\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:03:07.499634 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.499546 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpvcl" event={"ID":"4d1d6538-607a-4eec-881c-d1a189067df6","Type":"ContainerDied","Data":"edf0b5e34262c883e72050324e7d28188a896c4ccb97d6309ae9a716f0ca03e2"} Oct 30 18:03:07.499634 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.499601 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edf0b5e34262c883e72050324e7d28188a896c4ccb97d6309ae9a716f0ca03e2" Oct 30 18:03:07.499634 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.499561 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpvcl" Oct 30 18:03:07.501974 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.501947 4442 generic.go:356] "Generic (PLEG): container finished" podID="cd769ce4-6c17-47da-be0a-fdb8bbc7f76b" containerID="b63f463e69d42ebf7a366197489151e0f705957776e3001146cfadedca5d1e1e" exitCode=0 Oct 30 18:03:07.502100 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.502030 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5ddml" event={"ID":"cd769ce4-6c17-47da-be0a-fdb8bbc7f76b","Type":"ContainerDied","Data":"b63f463e69d42ebf7a366197489151e0f705957776e3001146cfadedca5d1e1e"} Oct 30 18:03:07.503937 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.503909 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tfns" event={"ID":"2442891f-bd5e-47cd-b882-2121cdd96cd0","Type":"ContainerDied","Data":"03643f0bb4bcf674c93c890458d8bb336768083e28b61c2be0bf1a85b7bc5db0"} Oct 30 18:03:07.503937 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.503933 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715tfns" Oct 30 18:03:07.504093 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.503953 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03643f0bb4bcf674c93c890458d8bb336768083e28b61c2be0bf1a85b7bc5db0" Oct 30 18:03:07.505700 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.505678 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hdwk9" Oct 30 18:03:07.505789 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.505717 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hdwk9" event={"ID":"452375be-3092-46cf-a3e4-56111a75fb24","Type":"ContainerDied","Data":"f31c4c80df02d60051f881d874dbd74bb4520e20b407f46bede06a472ad84f6a"} Oct 30 18:03:07.505789 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.505773 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f31c4c80df02d60051f881d874dbd74bb4520e20b407f46bede06a472ad84f6a" Oct 30 18:03:07.672676 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.672117 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar96xg" Oct 30 18:03:07.724734 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.724709 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777xm2" Oct 30 18:03:07.727908 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.727889 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9xpmn" Oct 30 18:03:07.731731 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.731714 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmksd" Oct 30 18:03:07.760812 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.760734 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f953747-4491-40e8-972d-e54b3951983b-kube-api-access-plzh7\" (UniqueName: \"kubernetes.io/projected/8f953747-4491-40e8-972d-e54b3951983b-kube-api-access-plzh7\") pod \"8f953747-4491-40e8-972d-e54b3951983b\" (UID: \"8f953747-4491-40e8-972d-e54b3951983b\") " Oct 30 18:03:07.760812 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.760787 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f953747-4491-40e8-972d-e54b3951983b-util\" (UniqueName: \"kubernetes.io/empty-dir/8f953747-4491-40e8-972d-e54b3951983b-util\") pod \"8f953747-4491-40e8-972d-e54b3951983b\" (UID: \"8f953747-4491-40e8-972d-e54b3951983b\") " Oct 30 18:03:07.760812 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.760810 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f953747-4491-40e8-972d-e54b3951983b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f953747-4491-40e8-972d-e54b3951983b-bundle\") pod \"8f953747-4491-40e8-972d-e54b3951983b\" (UID: \"8f953747-4491-40e8-972d-e54b3951983b\") " Oct 30 18:03:07.761069 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.760829 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45c0dc9a-baa9-40b5-a37d-f08973acbd6a-kube-api-access-r584n\" (UniqueName: \"kubernetes.io/projected/45c0dc9a-baa9-40b5-a37d-f08973acbd6a-kube-api-access-r584n\") pod \"45c0dc9a-baa9-40b5-a37d-f08973acbd6a\" (UID: \"45c0dc9a-baa9-40b5-a37d-f08973acbd6a\") " Oct 30 18:03:07.761069 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.760857 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2299ff4-930f-4255-86f3-01a9c34c9e5b-kube-api-access-grmmj\" (UniqueName: \"kubernetes.io/projected/a2299ff4-930f-4255-86f3-01a9c34c9e5b-kube-api-access-grmmj\") pod \"a2299ff4-930f-4255-86f3-01a9c34c9e5b\" (UID: \"a2299ff4-930f-4255-86f3-01a9c34c9e5b\") " Oct 30 18:03:07.761069 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.760898 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1db910f-a35a-475f-810e-d6a7a924bcbe-util\" (UniqueName: \"kubernetes.io/empty-dir/c1db910f-a35a-475f-810e-d6a7a924bcbe-util\") pod \"c1db910f-a35a-475f-810e-d6a7a924bcbe\" (UID: \"c1db910f-a35a-475f-810e-d6a7a924bcbe\") " Oct 30 18:03:07.761069 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.760931 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2299ff4-930f-4255-86f3-01a9c34c9e5b-util\" (UniqueName: \"kubernetes.io/empty-dir/a2299ff4-930f-4255-86f3-01a9c34c9e5b-util\") pod \"a2299ff4-930f-4255-86f3-01a9c34c9e5b\" (UID: \"a2299ff4-930f-4255-86f3-01a9c34c9e5b\") " Oct 30 18:03:07.761069 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.760961 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1db910f-a35a-475f-810e-d6a7a924bcbe-kube-api-access-xfkbr\" (UniqueName: \"kubernetes.io/projected/c1db910f-a35a-475f-810e-d6a7a924bcbe-kube-api-access-xfkbr\") pod \"c1db910f-a35a-475f-810e-d6a7a924bcbe\" (UID: \"c1db910f-a35a-475f-810e-d6a7a924bcbe\") " Oct 30 18:03:07.761069 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.760990 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1db910f-a35a-475f-810e-d6a7a924bcbe-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1db910f-a35a-475f-810e-d6a7a924bcbe-bundle\") pod \"c1db910f-a35a-475f-810e-d6a7a924bcbe\" (UID: \"c1db910f-a35a-475f-810e-d6a7a924bcbe\") " Oct 30 18:03:07.761069 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.761038 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45c0dc9a-baa9-40b5-a37d-f08973acbd6a-util\" (UniqueName: \"kubernetes.io/empty-dir/45c0dc9a-baa9-40b5-a37d-f08973acbd6a-util\") pod \"45c0dc9a-baa9-40b5-a37d-f08973acbd6a\" (UID: \"45c0dc9a-baa9-40b5-a37d-f08973acbd6a\") " Oct 30 18:03:07.761417 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.761097 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2299ff4-930f-4255-86f3-01a9c34c9e5b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2299ff4-930f-4255-86f3-01a9c34c9e5b-bundle\") pod \"a2299ff4-930f-4255-86f3-01a9c34c9e5b\" (UID: \"a2299ff4-930f-4255-86f3-01a9c34c9e5b\") " Oct 30 18:03:07.761417 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.761127 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45c0dc9a-baa9-40b5-a37d-f08973acbd6a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/45c0dc9a-baa9-40b5-a37d-f08973acbd6a-bundle\") pod \"45c0dc9a-baa9-40b5-a37d-f08973acbd6a\" (UID: \"45c0dc9a-baa9-40b5-a37d-f08973acbd6a\") " Oct 30 18:03:07.761417 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.761348 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f953747-4491-40e8-972d-e54b3951983b-bundle" pod "8f953747-4491-40e8-972d-e54b3951983b" (UID: "8f953747-4491-40e8-972d-e54b3951983b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:07.762118 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.762086 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45c0dc9a-baa9-40b5-a37d-f08973acbd6a-bundle" pod "45c0dc9a-baa9-40b5-a37d-f08973acbd6a" (UID: "45c0dc9a-baa9-40b5-a37d-f08973acbd6a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:07.762841 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.762814 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2299ff4-930f-4255-86f3-01a9c34c9e5b-bundle" pod "a2299ff4-930f-4255-86f3-01a9c34c9e5b" (UID: "a2299ff4-930f-4255-86f3-01a9c34c9e5b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:07.763251 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.763180 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1db910f-a35a-475f-810e-d6a7a924bcbe-bundle" pod "c1db910f-a35a-475f-810e-d6a7a924bcbe" (UID: "c1db910f-a35a-475f-810e-d6a7a924bcbe"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:07.763872 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.763843 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45c0dc9a-baa9-40b5-a37d-f08973acbd6a-kube-api-access-r584n" pod "45c0dc9a-baa9-40b5-a37d-f08973acbd6a" (UID: "45c0dc9a-baa9-40b5-a37d-f08973acbd6a"). InnerVolumeSpecName "kube-api-access-r584n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:03:07.764311 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.764277 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2299ff4-930f-4255-86f3-01a9c34c9e5b-kube-api-access-grmmj" pod "a2299ff4-930f-4255-86f3-01a9c34c9e5b" (UID: "a2299ff4-930f-4255-86f3-01a9c34c9e5b"). InnerVolumeSpecName "kube-api-access-grmmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:03:07.764397 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.764286 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1db910f-a35a-475f-810e-d6a7a924bcbe-kube-api-access-xfkbr" pod "c1db910f-a35a-475f-810e-d6a7a924bcbe" (UID: "c1db910f-a35a-475f-810e-d6a7a924bcbe"). InnerVolumeSpecName "kube-api-access-xfkbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:03:07.767555 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.767528 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2299ff4-930f-4255-86f3-01a9c34c9e5b-util" pod "a2299ff4-930f-4255-86f3-01a9c34c9e5b" (UID: "a2299ff4-930f-4255-86f3-01a9c34c9e5b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:07.770806 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.770778 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45c0dc9a-baa9-40b5-a37d-f08973acbd6a-util" pod "45c0dc9a-baa9-40b5-a37d-f08973acbd6a" (UID: "45c0dc9a-baa9-40b5-a37d-f08973acbd6a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:07.770926 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.770905 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f953747-4491-40e8-972d-e54b3951983b-kube-api-access-plzh7" pod "8f953747-4491-40e8-972d-e54b3951983b" (UID: "8f953747-4491-40e8-972d-e54b3951983b"). InnerVolumeSpecName "kube-api-access-plzh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:03:07.771090 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.771069 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f953747-4491-40e8-972d-e54b3951983b-util" pod "8f953747-4491-40e8-972d-e54b3951983b" (UID: "8f953747-4491-40e8-972d-e54b3951983b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:07.772361 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.772331 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1db910f-a35a-475f-810e-d6a7a924bcbe-util" pod "c1db910f-a35a-475f-810e-d6a7a924bcbe" (UID: "c1db910f-a35a-475f-810e-d6a7a924bcbe"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:07.861906 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.861877 4442 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1db910f-a35a-475f-810e-d6a7a924bcbe-util\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:03:07.861906 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.861903 4442 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2299ff4-930f-4255-86f3-01a9c34c9e5b-util\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:03:07.861906 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.861913 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfkbr\" (UniqueName: \"kubernetes.io/projected/c1db910f-a35a-475f-810e-d6a7a924bcbe-kube-api-access-xfkbr\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:03:07.862172 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.861926 4442 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1db910f-a35a-475f-810e-d6a7a924bcbe-bundle\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:03:07.862172 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.861939 4442 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45c0dc9a-baa9-40b5-a37d-f08973acbd6a-util\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:03:07.862172 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.861948 4442 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2299ff4-930f-4255-86f3-01a9c34c9e5b-bundle\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:03:07.862172 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.861955 4442 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45c0dc9a-baa9-40b5-a37d-f08973acbd6a-bundle\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:03:07.862172 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.861965 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plzh7\" (UniqueName: \"kubernetes.io/projected/8f953747-4491-40e8-972d-e54b3951983b-kube-api-access-plzh7\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:03:07.862172 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.861972 4442 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f953747-4491-40e8-972d-e54b3951983b-util\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:03:07.862172 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.861980 4442 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f953747-4491-40e8-972d-e54b3951983b-bundle\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:03:07.862172 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.861987 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r584n\" (UniqueName: \"kubernetes.io/projected/45c0dc9a-baa9-40b5-a37d-f08973acbd6a-kube-api-access-r584n\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:03:07.862172 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:07.861997 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grmmj\" (UniqueName: \"kubernetes.io/projected/a2299ff4-930f-4255-86f3-01a9c34c9e5b-kube-api-access-grmmj\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:03:08.511944 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:08.511864 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmksd" event={"ID":"8f953747-4491-40e8-972d-e54b3951983b","Type":"ContainerDied","Data":"d0137f01c2e0edd18a500c0d66c765a7bc5d5e9b821c50c06f67d4373f210074"} Oct 30 18:03:08.511944 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:08.511909 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0137f01c2e0edd18a500c0d66c765a7bc5d5e9b821c50c06f67d4373f210074" Oct 30 18:03:08.511944 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:08.511876 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmksd" Oct 30 18:03:08.513940 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:08.513915 4442 generic.go:356] "Generic (PLEG): container finished" podID="16f76c49-259b-4c9b-bf5a-abc20a23ac41" containerID="abc51a4afd08d29ba30cb159b5a1f2c7a39513f5b94daaef6b75cbdb8c65aa88" exitCode=0 Oct 30 18:03:08.514065 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:08.513995 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mmvlt" event={"ID":"16f76c49-259b-4c9b-bf5a-abc20a23ac41","Type":"ContainerDied","Data":"abc51a4afd08d29ba30cb159b5a1f2c7a39513f5b94daaef6b75cbdb8c65aa88"} Oct 30 18:03:08.515971 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:08.515952 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar96xg" Oct 30 18:03:08.516056 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:08.515974 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar96xg" event={"ID":"a2299ff4-930f-4255-86f3-01a9c34c9e5b","Type":"ContainerDied","Data":"2a432d20e4ff77098d52af6d9e3de00b12809f6bfb596aa82e461001ca28b13e"} Oct 30 18:03:08.516056 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:08.516007 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a432d20e4ff77098d52af6d9e3de00b12809f6bfb596aa82e461001ca28b13e" Oct 30 18:03:08.517977 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:08.517955 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777xm2" event={"ID":"45c0dc9a-baa9-40b5-a37d-f08973acbd6a","Type":"ContainerDied","Data":"4aa09c9ab18605145502fd9733a8e5d98f7da65fd546c58a522d92fc6cfe1ad4"} Oct 30 18:03:08.518090 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:08.517995 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4aa09c9ab18605145502fd9733a8e5d98f7da65fd546c58a522d92fc6cfe1ad4" Oct 30 18:03:08.518090 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:08.517959 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777xm2" Oct 30 18:03:08.520047 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:08.520025 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9xpmn" event={"ID":"c1db910f-a35a-475f-810e-d6a7a924bcbe","Type":"ContainerDied","Data":"44f2cfdbe083fb2d27065020402e4d82b3dbfc2258437c7894d428097eacf892"} Oct 30 18:03:08.520213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:08.520196 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44f2cfdbe083fb2d27065020402e4d82b3dbfc2258437c7894d428097eacf892" Oct 30 18:03:08.520317 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:08.520235 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9xpmn" Oct 30 18:03:08.648200 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:08.648167 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5ddml" Oct 30 18:03:08.666745 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:08.666716 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd769ce4-6c17-47da-be0a-fdb8bbc7f76b-kube-api-access-g5bq5\" (UniqueName: \"kubernetes.io/projected/cd769ce4-6c17-47da-be0a-fdb8bbc7f76b-kube-api-access-g5bq5\") pod \"cd769ce4-6c17-47da-be0a-fdb8bbc7f76b\" (UID: \"cd769ce4-6c17-47da-be0a-fdb8bbc7f76b\") " Oct 30 18:03:08.666893 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:08.666775 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd769ce4-6c17-47da-be0a-fdb8bbc7f76b-util\" (UniqueName: \"kubernetes.io/empty-dir/cd769ce4-6c17-47da-be0a-fdb8bbc7f76b-util\") pod \"cd769ce4-6c17-47da-be0a-fdb8bbc7f76b\" (UID: \"cd769ce4-6c17-47da-be0a-fdb8bbc7f76b\") " Oct 30 18:03:08.666893 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:08.666811 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd769ce4-6c17-47da-be0a-fdb8bbc7f76b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/cd769ce4-6c17-47da-be0a-fdb8bbc7f76b-bundle\") pod \"cd769ce4-6c17-47da-be0a-fdb8bbc7f76b\" (UID: \"cd769ce4-6c17-47da-be0a-fdb8bbc7f76b\") " Oct 30 18:03:08.667504 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:08.667414 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd769ce4-6c17-47da-be0a-fdb8bbc7f76b-bundle" pod "cd769ce4-6c17-47da-be0a-fdb8bbc7f76b" (UID: "cd769ce4-6c17-47da-be0a-fdb8bbc7f76b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:08.668882 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:08.668860 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd769ce4-6c17-47da-be0a-fdb8bbc7f76b-kube-api-access-g5bq5" pod "cd769ce4-6c17-47da-be0a-fdb8bbc7f76b" (UID: "cd769ce4-6c17-47da-be0a-fdb8bbc7f76b"). InnerVolumeSpecName "kube-api-access-g5bq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:03:08.675289 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:08.675263 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd769ce4-6c17-47da-be0a-fdb8bbc7f76b-util" pod "cd769ce4-6c17-47da-be0a-fdb8bbc7f76b" (UID: "cd769ce4-6c17-47da-be0a-fdb8bbc7f76b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:08.767289 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:08.767207 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5bq5\" (UniqueName: \"kubernetes.io/projected/cd769ce4-6c17-47da-be0a-fdb8bbc7f76b-kube-api-access-g5bq5\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:03:08.767289 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:08.767235 4442 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cd769ce4-6c17-47da-be0a-fdb8bbc7f76b-util\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:03:08.767289 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:08.767247 4442 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cd769ce4-6c17-47da-be0a-fdb8bbc7f76b-bundle\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:03:09.526227 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:09.526199 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5ddml" Oct 30 18:03:09.526639 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:09.526227 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5ddml" event={"ID":"cd769ce4-6c17-47da-be0a-fdb8bbc7f76b","Type":"ContainerDied","Data":"ef522eea36a8b868fbebee650358d5f20e0639fae26363306caa430972f137a2"} Oct 30 18:03:09.526639 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:09.526271 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef522eea36a8b868fbebee650358d5f20e0639fae26363306caa430972f137a2" Oct 30 18:03:09.650373 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:09.650352 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mmvlt" Oct 30 18:03:09.672802 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:09.672776 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16f76c49-259b-4c9b-bf5a-abc20a23ac41-bundle\" (UniqueName: \"kubernetes.io/empty-dir/16f76c49-259b-4c9b-bf5a-abc20a23ac41-bundle\") pod \"16f76c49-259b-4c9b-bf5a-abc20a23ac41\" (UID: \"16f76c49-259b-4c9b-bf5a-abc20a23ac41\") " Oct 30 18:03:09.672961 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:09.672834 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16f76c49-259b-4c9b-bf5a-abc20a23ac41-util\" (UniqueName: \"kubernetes.io/empty-dir/16f76c49-259b-4c9b-bf5a-abc20a23ac41-util\") pod \"16f76c49-259b-4c9b-bf5a-abc20a23ac41\" (UID: \"16f76c49-259b-4c9b-bf5a-abc20a23ac41\") " Oct 30 18:03:09.672961 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:09.672855 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16f76c49-259b-4c9b-bf5a-abc20a23ac41-kube-api-access-mnz6m\" (UniqueName: \"kubernetes.io/projected/16f76c49-259b-4c9b-bf5a-abc20a23ac41-kube-api-access-mnz6m\") pod \"16f76c49-259b-4c9b-bf5a-abc20a23ac41\" (UID: \"16f76c49-259b-4c9b-bf5a-abc20a23ac41\") " Oct 30 18:03:09.673402 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:09.673373 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16f76c49-259b-4c9b-bf5a-abc20a23ac41-bundle" pod "16f76c49-259b-4c9b-bf5a-abc20a23ac41" (UID: "16f76c49-259b-4c9b-bf5a-abc20a23ac41"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:09.675108 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:09.675088 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16f76c49-259b-4c9b-bf5a-abc20a23ac41-kube-api-access-mnz6m" pod "16f76c49-259b-4c9b-bf5a-abc20a23ac41" (UID: "16f76c49-259b-4c9b-bf5a-abc20a23ac41"). InnerVolumeSpecName "kube-api-access-mnz6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:03:09.678236 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:09.678209 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16f76c49-259b-4c9b-bf5a-abc20a23ac41-util" pod "16f76c49-259b-4c9b-bf5a-abc20a23ac41" (UID: "16f76c49-259b-4c9b-bf5a-abc20a23ac41"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:09.773775 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:09.773740 4442 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16f76c49-259b-4c9b-bf5a-abc20a23ac41-bundle\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:03:09.773775 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:09.773769 4442 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16f76c49-259b-4c9b-bf5a-abc20a23ac41-util\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:03:09.773775 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:09.773778 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnz6m\" (UniqueName: \"kubernetes.io/projected/16f76c49-259b-4c9b-bf5a-abc20a23ac41-kube-api-access-mnz6m\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:03:10.531780 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:10.531750 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mmvlt" event={"ID":"16f76c49-259b-4c9b-bf5a-abc20a23ac41","Type":"ContainerDied","Data":"a31f0c1881e4a79f45eb4445a66ffda2ff312d3f649b2a521d2584ecfa7a0df5"} Oct 30 18:03:10.531780 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:10.531769 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mmvlt" Oct 30 18:03:10.532317 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:10.531794 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a31f0c1881e4a79f45eb4445a66ffda2ff312d3f649b2a521d2584ecfa7a0df5" Oct 30 18:03:15.191181 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.191148 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-dv6zs"] Oct 30 18:03:15.192277 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192248 4442 cpu_manager.go:414] "containerMap: removing container" podUID="452375be-3092-46cf-a3e4-56111a75fb24" containerName="util" Oct 30 18:03:15.192277 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192273 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="452375be-3092-46cf-a3e4-56111a75fb24" containerName="util" Oct 30 18:03:15.192277 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192284 4442 cpu_manager.go:414] "containerMap: removing container" podUID="c1db910f-a35a-475f-810e-d6a7a924bcbe" containerName="extract" Oct 30 18:03:15.192460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192290 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1db910f-a35a-475f-810e-d6a7a924bcbe" containerName="extract" Oct 30 18:03:15.192460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192298 4442 cpu_manager.go:414] "containerMap: removing container" podUID="16f76c49-259b-4c9b-bf5a-abc20a23ac41" containerName="extract" Oct 30 18:03:15.192460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192303 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16f76c49-259b-4c9b-bf5a-abc20a23ac41" containerName="extract" Oct 30 18:03:15.192460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192313 4442 cpu_manager.go:414] "containerMap: removing container" podUID="c1db910f-a35a-475f-810e-d6a7a924bcbe" containerName="pull" Oct 30 18:03:15.192460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192318 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1db910f-a35a-475f-810e-d6a7a924bcbe" containerName="pull" Oct 30 18:03:15.192460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192328 4442 cpu_manager.go:414] "containerMap: removing container" podUID="4d1d6538-607a-4eec-881c-d1a189067df6" containerName="pull" Oct 30 18:03:15.192460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192332 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d1d6538-607a-4eec-881c-d1a189067df6" containerName="pull" Oct 30 18:03:15.192460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192339 4442 cpu_manager.go:414] "containerMap: removing container" podUID="2442891f-bd5e-47cd-b882-2121cdd96cd0" containerName="pull" Oct 30 18:03:15.192460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192343 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2442891f-bd5e-47cd-b882-2121cdd96cd0" containerName="pull" Oct 30 18:03:15.192460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192348 4442 cpu_manager.go:414] "containerMap: removing container" podUID="45c0dc9a-baa9-40b5-a37d-f08973acbd6a" containerName="util" Oct 30 18:03:15.192460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192352 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45c0dc9a-baa9-40b5-a37d-f08973acbd6a" containerName="util" Oct 30 18:03:15.192460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192358 4442 cpu_manager.go:414] "containerMap: removing container" podUID="cd769ce4-6c17-47da-be0a-fdb8bbc7f76b" containerName="extract" Oct 30 18:03:15.192460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192361 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd769ce4-6c17-47da-be0a-fdb8bbc7f76b" containerName="extract" Oct 30 18:03:15.192460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192370 4442 cpu_manager.go:414] "containerMap: removing container" podUID="4d1d6538-607a-4eec-881c-d1a189067df6" containerName="util" Oct 30 18:03:15.192460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192374 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d1d6538-607a-4eec-881c-d1a189067df6" containerName="util" Oct 30 18:03:15.192460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192378 4442 cpu_manager.go:414] "containerMap: removing container" podUID="4d1d6538-607a-4eec-881c-d1a189067df6" containerName="extract" Oct 30 18:03:15.192460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192383 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d1d6538-607a-4eec-881c-d1a189067df6" containerName="extract" Oct 30 18:03:15.192460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192392 4442 cpu_manager.go:414] "containerMap: removing container" podUID="c1db910f-a35a-475f-810e-d6a7a924bcbe" containerName="util" Oct 30 18:03:15.192460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192399 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1db910f-a35a-475f-810e-d6a7a924bcbe" containerName="util" Oct 30 18:03:15.192460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192405 4442 cpu_manager.go:414] "containerMap: removing container" podUID="2442891f-bd5e-47cd-b882-2121cdd96cd0" containerName="util" Oct 30 18:03:15.192460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192409 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2442891f-bd5e-47cd-b882-2121cdd96cd0" containerName="util" Oct 30 18:03:15.192460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192414 4442 cpu_manager.go:414] "containerMap: removing container" podUID="a2299ff4-930f-4255-86f3-01a9c34c9e5b" containerName="util" Oct 30 18:03:15.192460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192419 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2299ff4-930f-4255-86f3-01a9c34c9e5b" containerName="util" Oct 30 18:03:15.192460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192425 4442 cpu_manager.go:414] "containerMap: removing container" podUID="a2299ff4-930f-4255-86f3-01a9c34c9e5b" containerName="pull" Oct 30 18:03:15.192460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192429 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2299ff4-930f-4255-86f3-01a9c34c9e5b" containerName="pull" Oct 30 18:03:15.192460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192434 4442 cpu_manager.go:414] "containerMap: removing container" podUID="8f953747-4491-40e8-972d-e54b3951983b" containerName="extract" Oct 30 18:03:15.192460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192438 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f953747-4491-40e8-972d-e54b3951983b" containerName="extract" Oct 30 18:03:15.192460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192443 4442 cpu_manager.go:414] "containerMap: removing container" podUID="a2299ff4-930f-4255-86f3-01a9c34c9e5b" containerName="extract" Oct 30 18:03:15.192460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192447 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2299ff4-930f-4255-86f3-01a9c34c9e5b" containerName="extract" Oct 30 18:03:15.192460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192454 4442 cpu_manager.go:414] "containerMap: removing container" podUID="45c0dc9a-baa9-40b5-a37d-f08973acbd6a" containerName="pull" Oct 30 18:03:15.192460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192458 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45c0dc9a-baa9-40b5-a37d-f08973acbd6a" containerName="pull" Oct 30 18:03:15.192460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192463 4442 cpu_manager.go:414] "containerMap: removing container" podUID="2442891f-bd5e-47cd-b882-2121cdd96cd0" containerName="extract" Oct 30 18:03:15.192460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192467 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2442891f-bd5e-47cd-b882-2121cdd96cd0" containerName="extract" Oct 30 18:03:15.192460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192473 4442 cpu_manager.go:414] "containerMap: removing container" podUID="452375be-3092-46cf-a3e4-56111a75fb24" containerName="extract" Oct 30 18:03:15.192460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192477 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="452375be-3092-46cf-a3e4-56111a75fb24" containerName="extract" Oct 30 18:03:15.193418 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192482 4442 cpu_manager.go:414] "containerMap: removing container" podUID="45c0dc9a-baa9-40b5-a37d-f08973acbd6a" containerName="extract" Oct 30 18:03:15.193418 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192486 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45c0dc9a-baa9-40b5-a37d-f08973acbd6a" containerName="extract" Oct 30 18:03:15.193418 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192493 4442 cpu_manager.go:414] "containerMap: removing container" podUID="cd769ce4-6c17-47da-be0a-fdb8bbc7f76b" containerName="pull" Oct 30 18:03:15.193418 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192496 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd769ce4-6c17-47da-be0a-fdb8bbc7f76b" containerName="pull" Oct 30 18:03:15.193418 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192506 4442 cpu_manager.go:414] "containerMap: removing container" podUID="8f953747-4491-40e8-972d-e54b3951983b" containerName="util" Oct 30 18:03:15.193418 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192510 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f953747-4491-40e8-972d-e54b3951983b" containerName="util" Oct 30 18:03:15.193418 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192516 4442 cpu_manager.go:414] "containerMap: removing container" podUID="8f953747-4491-40e8-972d-e54b3951983b" containerName="pull" Oct 30 18:03:15.193418 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192520 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f953747-4491-40e8-972d-e54b3951983b" containerName="pull" Oct 30 18:03:15.193418 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192525 4442 cpu_manager.go:414] "containerMap: removing container" podUID="cd769ce4-6c17-47da-be0a-fdb8bbc7f76b" containerName="util" Oct 30 18:03:15.193418 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192529 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd769ce4-6c17-47da-be0a-fdb8bbc7f76b" containerName="util" Oct 30 18:03:15.193418 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192534 4442 cpu_manager.go:414] "containerMap: removing container" podUID="16f76c49-259b-4c9b-bf5a-abc20a23ac41" containerName="pull" Oct 30 18:03:15.193418 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192538 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16f76c49-259b-4c9b-bf5a-abc20a23ac41" containerName="pull" Oct 30 18:03:15.193418 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192544 4442 cpu_manager.go:414] "containerMap: removing container" podUID="452375be-3092-46cf-a3e4-56111a75fb24" containerName="pull" Oct 30 18:03:15.193418 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192547 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="452375be-3092-46cf-a3e4-56111a75fb24" containerName="pull" Oct 30 18:03:15.193418 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192555 4442 cpu_manager.go:414] "containerMap: removing container" podUID="16f76c49-259b-4c9b-bf5a-abc20a23ac41" containerName="util" Oct 30 18:03:15.193418 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192558 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16f76c49-259b-4c9b-bf5a-abc20a23ac41" containerName="util" Oct 30 18:03:15.193418 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192603 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="16f76c49-259b-4c9b-bf5a-abc20a23ac41" containerName="extract" Oct 30 18:03:15.193418 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192612 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd769ce4-6c17-47da-be0a-fdb8bbc7f76b" containerName="extract" Oct 30 18:03:15.193418 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192618 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d1d6538-607a-4eec-881c-d1a189067df6" containerName="extract" Oct 30 18:03:15.193418 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192624 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f953747-4491-40e8-972d-e54b3951983b" containerName="extract" Oct 30 18:03:15.193418 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192630 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2299ff4-930f-4255-86f3-01a9c34c9e5b" containerName="extract" Oct 30 18:03:15.193418 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192635 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="2442891f-bd5e-47cd-b882-2121cdd96cd0" containerName="extract" Oct 30 18:03:15.193418 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192639 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="452375be-3092-46cf-a3e4-56111a75fb24" containerName="extract" Oct 30 18:03:15.193418 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192643 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="45c0dc9a-baa9-40b5-a37d-f08973acbd6a" containerName="extract" Oct 30 18:03:15.193418 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.192649 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1db910f-a35a-475f-810e-d6a7a924bcbe" containerName="extract" Oct 30 18:03:15.194565 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.194550 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dv6zs" Oct 30 18:03:15.210612 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.210581 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zngf9\" (UniqueName: \"kubernetes.io/projected/24752fc1-eb92-452c-8f20-91cd12fc0ab0-kube-api-access-zngf9\") pod \"limitador-operator-controller-manager-5778bb4449-dv6zs\" (UID: \"24752fc1-eb92-452c-8f20-91cd12fc0ab0\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dv6zs" Oct 30 18:03:15.221079 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.221054 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 18:03:15.221224 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.221098 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 18:03:15.221224 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.221113 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-rs745\"" Oct 30 18:03:15.223018 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.222997 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-dv6zs"] Oct 30 18:03:15.311713 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.311671 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zngf9\" (UniqueName: \"kubernetes.io/projected/24752fc1-eb92-452c-8f20-91cd12fc0ab0-kube-api-access-zngf9\") pod \"limitador-operator-controller-manager-5778bb4449-dv6zs\" (UID: \"24752fc1-eb92-452c-8f20-91cd12fc0ab0\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dv6zs" Oct 30 18:03:15.333485 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.333455 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zngf9\" (UniqueName: \"kubernetes.io/projected/24752fc1-eb92-452c-8f20-91cd12fc0ab0-kube-api-access-zngf9\") pod \"limitador-operator-controller-manager-5778bb4449-dv6zs\" (UID: \"24752fc1-eb92-452c-8f20-91cd12fc0ab0\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dv6zs" Oct 30 18:03:15.505948 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.505855 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dv6zs" Oct 30 18:03:15.799717 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:15.799627 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-dv6zs"] Oct 30 18:03:15.803479 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:03:15.803454 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24752fc1_eb92_452c_8f20_91cd12fc0ab0.slice/crio-a73f42d189a0c800c328e4d04181c45c6260a5c623dd183253a64cfaaf8a8fc5 WatchSource:0}: Error finding container a73f42d189a0c800c328e4d04181c45c6260a5c623dd183253a64cfaaf8a8fc5: Status 404 returned error can't find the container with id a73f42d189a0c800c328e4d04181c45c6260a5c623dd183253a64cfaaf8a8fc5 Oct 30 18:03:16.561868 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:16.561830 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dv6zs" event={"ID":"24752fc1-eb92-452c-8f20-91cd12fc0ab0","Type":"ContainerStarted","Data":"a73f42d189a0c800c328e4d04181c45c6260a5c623dd183253a64cfaaf8a8fc5"} Oct 30 18:03:17.562907 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:03:17.562872 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:03:18.571814 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:18.571720 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dv6zs" event={"ID":"24752fc1-eb92-452c-8f20-91cd12fc0ab0","Type":"ContainerStarted","Data":"9865e0e5d91816c3ab5b9b74e44ae2731cadc4e7989ef3848aee014bf7e171b2"} Oct 30 18:03:18.571814 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:18.571810 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dv6zs" Oct 30 18:03:18.610648 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:18.610590 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dv6zs" podStartSLOduration=1.267143557 podStartE2EDuration="3.610558516s" totalImagesPullingTime="2.343414959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:18.608886714 +0000 UTC m=+295.698949078" watchObservedRunningTime="2026-10-30 18:03:18.610558516 +0000 UTC m=+295.700620866" Oct 30 18:03:19.053191 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:19.053152 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-v4fq8"] Oct 30 18:03:19.055403 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:19.055387 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-v4fq8" Oct 30 18:03:19.070706 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:19.070684 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 18:03:19.081537 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:19.081516 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-mrcwk\"" Oct 30 18:03:19.089150 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:19.089099 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-v4fq8"] Oct 30 18:03:19.143024 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:19.142986 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjdfx\" (UniqueName: \"kubernetes.io/projected/7695bc48-f4a1-4380-b25f-a32c35f6a500-kube-api-access-cjdfx\") pod \"dns-operator-controller-manager-8557cbdb84-v4fq8\" (UID: \"7695bc48-f4a1-4380-b25f-a32c35f6a500\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-v4fq8" Oct 30 18:03:19.243450 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:19.243413 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjdfx\" (UniqueName: \"kubernetes.io/projected/7695bc48-f4a1-4380-b25f-a32c35f6a500-kube-api-access-cjdfx\") pod \"dns-operator-controller-manager-8557cbdb84-v4fq8\" (UID: \"7695bc48-f4a1-4380-b25f-a32c35f6a500\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-v4fq8" Oct 30 18:03:19.267479 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:19.267440 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjdfx\" (UniqueName: \"kubernetes.io/projected/7695bc48-f4a1-4380-b25f-a32c35f6a500-kube-api-access-cjdfx\") pod \"dns-operator-controller-manager-8557cbdb84-v4fq8\" (UID: \"7695bc48-f4a1-4380-b25f-a32c35f6a500\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-v4fq8" Oct 30 18:03:19.367296 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:19.367260 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-v4fq8" Oct 30 18:03:19.659479 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:19.659405 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-v4fq8"] Oct 30 18:03:19.663404 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:03:19.663375 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7695bc48_f4a1_4380_b25f_a32c35f6a500.slice/crio-4613dd2c93cb4687770d8fcb452166194aac5b445a0206b5c5a815179d91822d WatchSource:0}: Error finding container 4613dd2c93cb4687770d8fcb452166194aac5b445a0206b5c5a815179d91822d: Status 404 returned error can't find the container with id 4613dd2c93cb4687770d8fcb452166194aac5b445a0206b5c5a815179d91822d Oct 30 18:03:20.583810 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:20.583778 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-v4fq8" event={"ID":"7695bc48-f4a1-4380-b25f-a32c35f6a500","Type":"ContainerStarted","Data":"4613dd2c93cb4687770d8fcb452166194aac5b445a0206b5c5a815179d91822d"} Oct 30 18:03:21.589837 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:21.589759 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-v4fq8" event={"ID":"7695bc48-f4a1-4380-b25f-a32c35f6a500","Type":"ContainerStarted","Data":"ca98c361d8e9ac48589ee7d0cbb5a530f7c533334814b88465d13df96c5d5f75"} Oct 30 18:03:21.590194 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:21.589892 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-v4fq8" Oct 30 18:03:21.629116 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:21.629058 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-v4fq8" podStartSLOduration=0.992114063 podStartE2EDuration="2.629045012s" totalImagesPullingTime="1.636930949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:21.628029644 +0000 UTC m=+298.718091985" watchObservedRunningTime="2026-10-30 18:03:21.629045012 +0000 UTC m=+298.719107348" Oct 30 18:03:23.082424 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:23.082395 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52rn6_516c6b6c-fc40-49cf-93b5-1c4471b02cf0/ovn-acl-logging/0.log" Oct 30 18:03:23.082878 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:23.082468 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52rn6_516c6b6c-fc40-49cf-93b5-1c4471b02cf0/ovn-acl-logging/0.log" Oct 30 18:03:23.533572 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:23.533541 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-k8hhs"] Oct 30 18:03:23.536188 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:23.536169 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-k8hhs" Oct 30 18:03:23.551489 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:23.551469 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-kcz9t\"" Oct 30 18:03:23.567635 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:23.567610 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-k8hhs"] Oct 30 18:03:23.578882 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:23.578859 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdt5d\" (UniqueName: \"kubernetes.io/projected/ea74abc8-3819-4e41-9705-9c95421816ce-kube-api-access-sdt5d\") pod \"authorino-operator-7b7bb4bc77-k8hhs\" (UID: \"ea74abc8-3819-4e41-9705-9c95421816ce\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-k8hhs" Oct 30 18:03:23.680078 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:23.680045 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdt5d\" (UniqueName: \"kubernetes.io/projected/ea74abc8-3819-4e41-9705-9c95421816ce-kube-api-access-sdt5d\") pod \"authorino-operator-7b7bb4bc77-k8hhs\" (UID: \"ea74abc8-3819-4e41-9705-9c95421816ce\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-k8hhs" Oct 30 18:03:23.699533 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:23.699505 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdt5d\" (UniqueName: \"kubernetes.io/projected/ea74abc8-3819-4e41-9705-9c95421816ce-kube-api-access-sdt5d\") pod \"authorino-operator-7b7bb4bc77-k8hhs\" (UID: \"ea74abc8-3819-4e41-9705-9c95421816ce\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-k8hhs" Oct 30 18:03:23.847875 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:23.847783 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-k8hhs" Oct 30 18:03:24.145753 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:24.145715 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-k8hhs"] Oct 30 18:03:24.149727 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:03:24.149701 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea74abc8_3819_4e41_9705_9c95421816ce.slice/crio-7cfa691003b48f1c8a9167a1f192bf99e78d3ade8ec728ef1a276ee7d490a5b4 WatchSource:0}: Error finding container 7cfa691003b48f1c8a9167a1f192bf99e78d3ade8ec728ef1a276ee7d490a5b4: Status 404 returned error can't find the container with id 7cfa691003b48f1c8a9167a1f192bf99e78d3ade8ec728ef1a276ee7d490a5b4 Oct 30 18:03:24.603855 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:24.603770 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-k8hhs" event={"ID":"ea74abc8-3819-4e41-9705-9c95421816ce","Type":"ContainerStarted","Data":"7cfa691003b48f1c8a9167a1f192bf99e78d3ade8ec728ef1a276ee7d490a5b4"} Oct 30 18:03:25.610325 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:25.610272 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-k8hhs" event={"ID":"ea74abc8-3819-4e41-9705-9c95421816ce","Type":"ContainerStarted","Data":"90e1d23330fc91d81d2d0e78863e05ac874f6b318fee6dd19685015ba45ecc05"} Oct 30 18:03:25.610709 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:25.610378 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-k8hhs" Oct 30 18:03:25.653410 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:25.653301 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-k8hhs" podStartSLOduration=1.42501503 podStartE2EDuration="2.653287882s" totalImagesPullingTime="1.228272852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:25.652378391 +0000 UTC m=+302.742440741" watchObservedRunningTime="2026-10-30 18:03:25.653287882 +0000 UTC m=+302.743350220" Oct 30 18:03:26.757477 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:26.757444 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-nh2r9"] Oct 30 18:03:26.760168 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:26.760147 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-nh2r9" Oct 30 18:03:26.778907 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:26.778883 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 18:03:26.779261 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:26.778936 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 18:03:26.787217 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:26.787196 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 18:03:26.787344 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:26.787248 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 18:03:26.787344 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:26.787301 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-7dbhx\"" Oct 30 18:03:26.787344 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:26.787319 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 18:03:26.806562 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:26.806537 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1f64edd2-aeab-46cc-8bdc-bd0100da893d-webhook-cert\") pod \"opendatahub-operator-controller-manager-58c455bdd6-nh2r9\" (UID: \"1f64edd2-aeab-46cc-8bdc-bd0100da893d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-nh2r9" Oct 30 18:03:26.806679 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:26.806575 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1f64edd2-aeab-46cc-8bdc-bd0100da893d-manifests\") pod \"opendatahub-operator-controller-manager-58c455bdd6-nh2r9\" (UID: \"1f64edd2-aeab-46cc-8bdc-bd0100da893d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-nh2r9" Oct 30 18:03:26.806679 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:26.806609 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f64edd2-aeab-46cc-8bdc-bd0100da893d-tmp\") pod \"opendatahub-operator-controller-manager-58c455bdd6-nh2r9\" (UID: \"1f64edd2-aeab-46cc-8bdc-bd0100da893d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-nh2r9" Oct 30 18:03:26.806811 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:26.806687 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1f64edd2-aeab-46cc-8bdc-bd0100da893d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58c455bdd6-nh2r9\" (UID: \"1f64edd2-aeab-46cc-8bdc-bd0100da893d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-nh2r9" Oct 30 18:03:26.806811 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:26.806757 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbdwt\" (UniqueName: \"kubernetes.io/projected/1f64edd2-aeab-46cc-8bdc-bd0100da893d-kube-api-access-zbdwt\") pod \"opendatahub-operator-controller-manager-58c455bdd6-nh2r9\" (UID: \"1f64edd2-aeab-46cc-8bdc-bd0100da893d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-nh2r9" Oct 30 18:03:26.806811 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:26.806798 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1f64edd2-aeab-46cc-8bdc-bd0100da893d-metrics-cert\") pod \"opendatahub-operator-controller-manager-58c455bdd6-nh2r9\" (UID: \"1f64edd2-aeab-46cc-8bdc-bd0100da893d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-nh2r9" Oct 30 18:03:26.881910 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:26.881880 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-nh2r9"] Oct 30 18:03:26.908228 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:26.908202 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbdwt\" (UniqueName: \"kubernetes.io/projected/1f64edd2-aeab-46cc-8bdc-bd0100da893d-kube-api-access-zbdwt\") pod \"opendatahub-operator-controller-manager-58c455bdd6-nh2r9\" (UID: \"1f64edd2-aeab-46cc-8bdc-bd0100da893d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-nh2r9" Oct 30 18:03:26.908389 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:26.908238 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1f64edd2-aeab-46cc-8bdc-bd0100da893d-metrics-cert\") pod \"opendatahub-operator-controller-manager-58c455bdd6-nh2r9\" (UID: \"1f64edd2-aeab-46cc-8bdc-bd0100da893d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-nh2r9" Oct 30 18:03:26.908389 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:26.908278 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1f64edd2-aeab-46cc-8bdc-bd0100da893d-webhook-cert\") pod \"opendatahub-operator-controller-manager-58c455bdd6-nh2r9\" (UID: \"1f64edd2-aeab-46cc-8bdc-bd0100da893d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-nh2r9" Oct 30 18:03:26.908389 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:26.908307 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1f64edd2-aeab-46cc-8bdc-bd0100da893d-manifests\") pod \"opendatahub-operator-controller-manager-58c455bdd6-nh2r9\" (UID: \"1f64edd2-aeab-46cc-8bdc-bd0100da893d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-nh2r9" Oct 30 18:03:26.908389 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:26.908330 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f64edd2-aeab-46cc-8bdc-bd0100da893d-tmp\") pod \"opendatahub-operator-controller-manager-58c455bdd6-nh2r9\" (UID: \"1f64edd2-aeab-46cc-8bdc-bd0100da893d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-nh2r9" Oct 30 18:03:26.908389 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:26.908346 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1f64edd2-aeab-46cc-8bdc-bd0100da893d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58c455bdd6-nh2r9\" (UID: \"1f64edd2-aeab-46cc-8bdc-bd0100da893d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-nh2r9" Oct 30 18:03:26.908826 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:26.908795 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1f64edd2-aeab-46cc-8bdc-bd0100da893d-manifests\") pod \"opendatahub-operator-controller-manager-58c455bdd6-nh2r9\" (UID: \"1f64edd2-aeab-46cc-8bdc-bd0100da893d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-nh2r9" Oct 30 18:03:26.910856 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:26.910824 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f64edd2-aeab-46cc-8bdc-bd0100da893d-tmp\") pod \"opendatahub-operator-controller-manager-58c455bdd6-nh2r9\" (UID: \"1f64edd2-aeab-46cc-8bdc-bd0100da893d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-nh2r9" Oct 30 18:03:26.911020 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:26.910998 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1f64edd2-aeab-46cc-8bdc-bd0100da893d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58c455bdd6-nh2r9\" (UID: \"1f64edd2-aeab-46cc-8bdc-bd0100da893d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-nh2r9" Oct 30 18:03:26.911098 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:26.911021 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1f64edd2-aeab-46cc-8bdc-bd0100da893d-webhook-cert\") pod \"opendatahub-operator-controller-manager-58c455bdd6-nh2r9\" (UID: \"1f64edd2-aeab-46cc-8bdc-bd0100da893d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-nh2r9" Oct 30 18:03:26.911098 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:26.910998 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1f64edd2-aeab-46cc-8bdc-bd0100da893d-metrics-cert\") pod \"opendatahub-operator-controller-manager-58c455bdd6-nh2r9\" (UID: \"1f64edd2-aeab-46cc-8bdc-bd0100da893d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-nh2r9" Oct 30 18:03:26.931979 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:26.931949 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbdwt\" (UniqueName: \"kubernetes.io/projected/1f64edd2-aeab-46cc-8bdc-bd0100da893d-kube-api-access-zbdwt\") pod \"opendatahub-operator-controller-manager-58c455bdd6-nh2r9\" (UID: \"1f64edd2-aeab-46cc-8bdc-bd0100da893d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-nh2r9" Oct 30 18:03:27.079270 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:27.079186 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-nh2r9" Oct 30 18:03:27.434306 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:27.434285 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-nh2r9"] Oct 30 18:03:27.436796 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:03:27.436765 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f64edd2_aeab_46cc_8bdc_bd0100da893d.slice/crio-e5b28fad490e91dedfbb0655aa1af67756cb2aa7d58372f56599e56b0351912c WatchSource:0}: Error finding container e5b28fad490e91dedfbb0655aa1af67756cb2aa7d58372f56599e56b0351912c: Status 404 returned error can't find the container with id e5b28fad490e91dedfbb0655aa1af67756cb2aa7d58372f56599e56b0351912c Oct 30 18:03:27.438552 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:27.438534 4442 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:03:27.620016 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:27.619980 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-nh2r9" event={"ID":"1f64edd2-aeab-46cc-8bdc-bd0100da893d","Type":"ContainerStarted","Data":"e5b28fad490e91dedfbb0655aa1af67756cb2aa7d58372f56599e56b0351912c"} Oct 30 18:03:29.578834 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:29.578801 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dv6zs" Oct 30 18:03:30.637304 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:30.637267 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-nh2r9" event={"ID":"1f64edd2-aeab-46cc-8bdc-bd0100da893d","Type":"ContainerStarted","Data":"ce187de2ef1395455631c98bb2e30c9c5821eafe7020ef6e1a50a0ab97cd894d"} Oct 30 18:03:31.643619 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:31.643588 4442 generic.go:356] "Generic (PLEG): container finished" podID="1f64edd2-aeab-46cc-8bdc-bd0100da893d" containerID="ce187de2ef1395455631c98bb2e30c9c5821eafe7020ef6e1a50a0ab97cd894d" exitCode=0 Oct 30 18:03:31.644070 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:31.643671 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-nh2r9" event={"ID":"1f64edd2-aeab-46cc-8bdc-bd0100da893d","Type":"ContainerDied","Data":"ce187de2ef1395455631c98bb2e30c9c5821eafe7020ef6e1a50a0ab97cd894d"} Oct 30 18:03:32.260661 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:32.260573 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bsck5"] Oct 30 18:03:32.262830 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:32.262808 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bsck5" Oct 30 18:03:32.318494 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:32.318459 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bsck5"] Oct 30 18:03:32.352407 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:32.352377 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp4cr\" (UniqueName: \"kubernetes.io/projected/61b3c946-73f9-4659-a01b-879caab4d17d-kube-api-access-cp4cr\") pod \"kuadrant-operator-controller-manager-cf8469d7b-bsck5\" (UID: \"61b3c946-73f9-4659-a01b-879caab4d17d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bsck5" Oct 30 18:03:32.352584 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:32.352508 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/61b3c946-73f9-4659-a01b-879caab4d17d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-bsck5\" (UID: \"61b3c946-73f9-4659-a01b-879caab4d17d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bsck5" Oct 30 18:03:32.363881 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:32.363858 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-x8mnl\"" Oct 30 18:03:32.453453 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:32.453414 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp4cr\" (UniqueName: \"kubernetes.io/projected/61b3c946-73f9-4659-a01b-879caab4d17d-kube-api-access-cp4cr\") pod \"kuadrant-operator-controller-manager-cf8469d7b-bsck5\" (UID: \"61b3c946-73f9-4659-a01b-879caab4d17d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bsck5" Oct 30 18:03:32.453603 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:32.453503 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/61b3c946-73f9-4659-a01b-879caab4d17d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-bsck5\" (UID: \"61b3c946-73f9-4659-a01b-879caab4d17d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bsck5" Oct 30 18:03:32.453823 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:32.453809 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/61b3c946-73f9-4659-a01b-879caab4d17d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-bsck5\" (UID: \"61b3c946-73f9-4659-a01b-879caab4d17d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bsck5" Oct 30 18:03:32.484039 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:32.484001 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp4cr\" (UniqueName: \"kubernetes.io/projected/61b3c946-73f9-4659-a01b-879caab4d17d-kube-api-access-cp4cr\") pod \"kuadrant-operator-controller-manager-cf8469d7b-bsck5\" (UID: \"61b3c946-73f9-4659-a01b-879caab4d17d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bsck5" Oct 30 18:03:32.575187 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:32.575071 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bsck5" Oct 30 18:03:32.596703 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:32.596679 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-v4fq8" Oct 30 18:03:32.653406 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:32.653370 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-nh2r9" event={"ID":"1f64edd2-aeab-46cc-8bdc-bd0100da893d","Type":"ContainerStarted","Data":"07d193cf7a19fb9dcb37c2345a316f6f7cab5623af4dd7e14df36db579658cec"} Oct 30 18:03:32.654242 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:32.654222 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-nh2r9" Oct 30 18:03:32.719827 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:32.719777 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-nh2r9" podStartSLOduration=3.0071217 podStartE2EDuration="6.719763472s" totalImagesPullingTime="3.284908471s" totalInitContainerRuntime="427.733301ms" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:32.71772589 +0000 UTC m=+309.807788228" watchObservedRunningTime="2026-10-30 18:03:32.719763472 +0000 UTC m=+309.809825811" Oct 30 18:03:32.882754 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:32.882722 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bsck5"] Oct 30 18:03:32.886976 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:03:32.886948 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61b3c946_73f9_4659_a01b_879caab4d17d.slice/crio-b57858d6665bec6aaa8c84ba40c41da03aa5fa91027d5adaf787542d55003a59 WatchSource:0}: Error finding container b57858d6665bec6aaa8c84ba40c41da03aa5fa91027d5adaf787542d55003a59: Status 404 returned error can't find the container with id b57858d6665bec6aaa8c84ba40c41da03aa5fa91027d5adaf787542d55003a59 Oct 30 18:03:33.143354 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:33.143268 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-x9vk5"] Oct 30 18:03:33.146408 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:33.146391 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-x9vk5" Oct 30 18:03:33.167210 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:33.167182 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-gp2tj\"" Oct 30 18:03:33.167637 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:33.167619 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:03:33.170184 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:33.170168 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 30 18:03:33.187878 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:33.187847 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-x9vk5"] Oct 30 18:03:33.260849 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:33.260810 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa03b7e0-c562-42e6-b8e9-285a65eee8e1-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-x9vk5\" (UID: \"aa03b7e0-c562-42e6-b8e9-285a65eee8e1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-x9vk5" Oct 30 18:03:33.261003 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:33.260948 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxgqw\" (UniqueName: \"kubernetes.io/projected/aa03b7e0-c562-42e6-b8e9-285a65eee8e1-kube-api-access-dxgqw\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-x9vk5\" (UID: \"aa03b7e0-c562-42e6-b8e9-285a65eee8e1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-x9vk5" Oct 30 18:03:33.362328 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:33.362288 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxgqw\" (UniqueName: \"kubernetes.io/projected/aa03b7e0-c562-42e6-b8e9-285a65eee8e1-kube-api-access-dxgqw\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-x9vk5\" (UID: \"aa03b7e0-c562-42e6-b8e9-285a65eee8e1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-x9vk5" Oct 30 18:03:33.362328 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:33.362335 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa03b7e0-c562-42e6-b8e9-285a65eee8e1-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-x9vk5\" (UID: \"aa03b7e0-c562-42e6-b8e9-285a65eee8e1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-x9vk5" Oct 30 18:03:33.362700 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:33.362685 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa03b7e0-c562-42e6-b8e9-285a65eee8e1-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-x9vk5\" (UID: \"aa03b7e0-c562-42e6-b8e9-285a65eee8e1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-x9vk5" Oct 30 18:03:33.382248 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:33.382218 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxgqw\" (UniqueName: \"kubernetes.io/projected/aa03b7e0-c562-42e6-b8e9-285a65eee8e1-kube-api-access-dxgqw\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-x9vk5\" (UID: \"aa03b7e0-c562-42e6-b8e9-285a65eee8e1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-x9vk5" Oct 30 18:03:33.458293 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:33.458207 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-x9vk5" Oct 30 18:03:33.660840 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:33.660798 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bsck5" event={"ID":"61b3c946-73f9-4659-a01b-879caab4d17d","Type":"ContainerStarted","Data":"b57858d6665bec6aaa8c84ba40c41da03aa5fa91027d5adaf787542d55003a59"} Oct 30 18:03:33.804964 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:33.804932 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-x9vk5"] Oct 30 18:03:33.809498 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:03:33.809469 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa03b7e0_c562_42e6_b8e9_285a65eee8e1.slice/crio-712fe22efb5fb9f8dcd05e7a4f4faf9ad7b307244723d44d6f1e80d5f71c540a WatchSource:0}: Error finding container 712fe22efb5fb9f8dcd05e7a4f4faf9ad7b307244723d44d6f1e80d5f71c540a: Status 404 returned error can't find the container with id 712fe22efb5fb9f8dcd05e7a4f4faf9ad7b307244723d44d6f1e80d5f71c540a Oct 30 18:03:34.669046 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:34.668951 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-x9vk5" event={"ID":"aa03b7e0-c562-42e6-b8e9-285a65eee8e1","Type":"ContainerStarted","Data":"712fe22efb5fb9f8dcd05e7a4f4faf9ad7b307244723d44d6f1e80d5f71c540a"} Oct 30 18:03:36.617853 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:36.617824 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-k8hhs" Oct 30 18:03:36.683835 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:36.683288 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-x9vk5" event={"ID":"aa03b7e0-c562-42e6-b8e9-285a65eee8e1","Type":"ContainerStarted","Data":"24bde680cae9375dd46ce14c2a488e42e413b96cc92a26a850238409d7c5c749"} Oct 30 18:03:36.727347 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:36.725489 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-x9vk5" podStartSLOduration=1.299149899 podStartE2EDuration="3.725468689s" totalImagesPullingTime="2.42631879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:36.72402751 +0000 UTC m=+313.814089850" watchObservedRunningTime="2026-10-30 18:03:36.725468689 +0000 UTC m=+313.815531030" Oct 30 18:03:37.689094 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:37.689053 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bsck5" event={"ID":"61b3c946-73f9-4659-a01b-879caab4d17d","Type":"ContainerStarted","Data":"0ecffbde47d5c855e688d5256b5185116702cd09e248b1d310767e577b5998f1"} Oct 30 18:03:37.689573 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:37.689321 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bsck5" Oct 30 18:03:37.763773 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:37.763721 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bsck5" podStartSLOduration=1.08862141 podStartE2EDuration="5.763702633s" totalImagesPullingTime="4.675081223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:37.750601413 +0000 UTC m=+314.840663753" watchObservedRunningTime="2026-10-30 18:03:37.763702633 +0000 UTC m=+314.853764950" Oct 30 18:03:44.671831 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:44.671794 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-nh2r9" Oct 30 18:03:47.558027 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:03:47.557994 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:03:48.696868 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:48.696837 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bsck5" Oct 30 18:03:48.937300 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:48.937268 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-h2877"] Oct 30 18:03:48.941553 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:48.941537 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-h2877" Oct 30 18:03:48.965982 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:48.965953 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-h2877"] Oct 30 18:03:49.003294 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:49.003265 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwlg4\" (UniqueName: \"kubernetes.io/projected/ec2b6bb1-c8d6-43ee-8234-0556646382a7-kube-api-access-cwlg4\") pod \"kuadrant-operator-controller-manager-678d87766f-h2877\" (UID: \"ec2b6bb1-c8d6-43ee-8234-0556646382a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-h2877" Oct 30 18:03:49.003469 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:49.003302 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ec2b6bb1-c8d6-43ee-8234-0556646382a7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-h2877\" (UID: \"ec2b6bb1-c8d6-43ee-8234-0556646382a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-h2877" Oct 30 18:03:49.104251 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:49.104221 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwlg4\" (UniqueName: \"kubernetes.io/projected/ec2b6bb1-c8d6-43ee-8234-0556646382a7-kube-api-access-cwlg4\") pod \"kuadrant-operator-controller-manager-678d87766f-h2877\" (UID: \"ec2b6bb1-c8d6-43ee-8234-0556646382a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-h2877" Oct 30 18:03:49.104416 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:49.104264 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ec2b6bb1-c8d6-43ee-8234-0556646382a7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-h2877\" (UID: \"ec2b6bb1-c8d6-43ee-8234-0556646382a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-h2877" Oct 30 18:03:49.104679 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:49.104659 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ec2b6bb1-c8d6-43ee-8234-0556646382a7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-h2877\" (UID: \"ec2b6bb1-c8d6-43ee-8234-0556646382a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-h2877" Oct 30 18:03:49.123394 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:49.123368 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwlg4\" (UniqueName: \"kubernetes.io/projected/ec2b6bb1-c8d6-43ee-8234-0556646382a7-kube-api-access-cwlg4\") pod \"kuadrant-operator-controller-manager-678d87766f-h2877\" (UID: \"ec2b6bb1-c8d6-43ee-8234-0556646382a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-h2877" Oct 30 18:03:49.253082 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:49.252995 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-h2877" Oct 30 18:03:49.569075 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:49.568980 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-h2877"] Oct 30 18:03:49.573053 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:03:49.573032 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec2b6bb1_c8d6_43ee_8234_0556646382a7.slice/crio-3b8b0186827f8d50717c9e7bdc13fca3f896aff9d4f9f45f6db5473c834f49b4 WatchSource:0}: Error finding container 3b8b0186827f8d50717c9e7bdc13fca3f896aff9d4f9f45f6db5473c834f49b4: Status 404 returned error can't find the container with id 3b8b0186827f8d50717c9e7bdc13fca3f896aff9d4f9f45f6db5473c834f49b4 Oct 30 18:03:49.750085 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:49.750048 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-h2877" event={"ID":"ec2b6bb1-c8d6-43ee-8234-0556646382a7","Type":"ContainerStarted","Data":"66eb816062a0834ef2fb3583db048e258ac473db243183cf4cc57039b52636c8"} Oct 30 18:03:49.750571 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:49.750165 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-h2877" event={"ID":"ec2b6bb1-c8d6-43ee-8234-0556646382a7","Type":"ContainerStarted","Data":"3b8b0186827f8d50717c9e7bdc13fca3f896aff9d4f9f45f6db5473c834f49b4"} Oct 30 18:03:49.750571 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:49.750196 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-h2877" Oct 30 18:03:49.792571 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:03:49.792513 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-h2877" podStartSLOduration=1.7924970500000001 podStartE2EDuration="1.79249705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:49.791535365 +0000 UTC m=+326.881597704" watchObservedRunningTime="2026-10-30 18:03:49.79249705 +0000 UTC m=+326.882559389" Oct 30 18:04:00.758537 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:00.758508 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-h2877" Oct 30 18:04:00.850218 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:00.850186 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bsck5"] Oct 30 18:04:00.850485 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:00.850454 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bsck5" podUID="61b3c946-73f9-4659-a01b-879caab4d17d" containerName="manager" containerID="cri-o://0ecffbde47d5c855e688d5256b5185116702cd09e248b1d310767e577b5998f1" gracePeriod=10 Oct 30 18:04:01.092686 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:01.092659 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bsck5" Oct 30 18:04:01.207840 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:01.207807 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61b3c946-73f9-4659-a01b-879caab4d17d-kube-api-access-cp4cr\" (UniqueName: \"kubernetes.io/projected/61b3c946-73f9-4659-a01b-879caab4d17d-kube-api-access-cp4cr\") pod \"61b3c946-73f9-4659-a01b-879caab4d17d\" (UID: \"61b3c946-73f9-4659-a01b-879caab4d17d\") " Oct 30 18:04:01.208042 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:01.207884 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61b3c946-73f9-4659-a01b-879caab4d17d-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/61b3c946-73f9-4659-a01b-879caab4d17d-extensions-socket-volume\") pod \"61b3c946-73f9-4659-a01b-879caab4d17d\" (UID: \"61b3c946-73f9-4659-a01b-879caab4d17d\") " Oct 30 18:04:01.208444 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:01.208407 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61b3c946-73f9-4659-a01b-879caab4d17d-extensions-socket-volume" pod "61b3c946-73f9-4659-a01b-879caab4d17d" (UID: "61b3c946-73f9-4659-a01b-879caab4d17d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:01.210380 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:01.210348 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61b3c946-73f9-4659-a01b-879caab4d17d-kube-api-access-cp4cr" pod "61b3c946-73f9-4659-a01b-879caab4d17d" (UID: "61b3c946-73f9-4659-a01b-879caab4d17d"). InnerVolumeSpecName "kube-api-access-cp4cr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:04:01.309553 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:01.309466 4442 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/61b3c946-73f9-4659-a01b-879caab4d17d-extensions-socket-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:04:01.309553 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:01.309495 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp4cr\" (UniqueName: \"kubernetes.io/projected/61b3c946-73f9-4659-a01b-879caab4d17d-kube-api-access-cp4cr\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:04:01.811447 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:01.811407 4442 generic.go:356] "Generic (PLEG): container finished" podID="61b3c946-73f9-4659-a01b-879caab4d17d" containerID="0ecffbde47d5c855e688d5256b5185116702cd09e248b1d310767e577b5998f1" exitCode=0 Oct 30 18:04:01.811902 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:01.811461 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bsck5" Oct 30 18:04:01.811902 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:01.811495 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bsck5" event={"ID":"61b3c946-73f9-4659-a01b-879caab4d17d","Type":"ContainerDied","Data":"0ecffbde47d5c855e688d5256b5185116702cd09e248b1d310767e577b5998f1"} Oct 30 18:04:01.811902 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:01.811559 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bsck5" event={"ID":"61b3c946-73f9-4659-a01b-879caab4d17d","Type":"ContainerDied","Data":"b57858d6665bec6aaa8c84ba40c41da03aa5fa91027d5adaf787542d55003a59"} Oct 30 18:04:01.811902 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:01.811583 4442 scope.go:122] "RemoveContainer" containerID="0ecffbde47d5c855e688d5256b5185116702cd09e248b1d310767e577b5998f1" Oct 30 18:04:01.820393 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:01.820377 4442 scope.go:122] "RemoveContainer" containerID="0ecffbde47d5c855e688d5256b5185116702cd09e248b1d310767e577b5998f1" Oct 30 18:04:01.820680 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:04:01.820649 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ecffbde47d5c855e688d5256b5185116702cd09e248b1d310767e577b5998f1\": container with ID starting with 0ecffbde47d5c855e688d5256b5185116702cd09e248b1d310767e577b5998f1 not found: ID does not exist" containerID="0ecffbde47d5c855e688d5256b5185116702cd09e248b1d310767e577b5998f1" Oct 30 18:04:01.820739 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:01.820684 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ecffbde47d5c855e688d5256b5185116702cd09e248b1d310767e577b5998f1"} err="failed to get container status \"0ecffbde47d5c855e688d5256b5185116702cd09e248b1d310767e577b5998f1\": rpc error: code = NotFound desc = could not find container \"0ecffbde47d5c855e688d5256b5185116702cd09e248b1d310767e577b5998f1\": container with ID starting with 0ecffbde47d5c855e688d5256b5185116702cd09e248b1d310767e577b5998f1 not found: ID does not exist" Oct 30 18:04:01.892439 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:01.892412 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bsck5"] Oct 30 18:04:01.904345 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:01.904315 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bsck5"] Oct 30 18:04:02.364714 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:02.364685 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489hbqng"] Oct 30 18:04:02.365054 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:02.365042 4442 cpu_manager.go:414] "containerMap: removing container" podUID="61b3c946-73f9-4659-a01b-879caab4d17d" containerName="manager" Oct 30 18:04:02.365097 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:02.365056 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61b3c946-73f9-4659-a01b-879caab4d17d" containerName="manager" Oct 30 18:04:02.365157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:02.365110 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="61b3c946-73f9-4659-a01b-879caab4d17d" containerName="manager" Oct 30 18:04:02.368626 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:02.368606 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489hbqng" Oct 30 18:04:02.394832 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:02.394808 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-kn4c7\"" Oct 30 18:04:02.404523 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:02.404504 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489hbqng"] Oct 30 18:04:02.520408 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:02.520369 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c165fa36-f6e3-481f-a56e-05f046c197cd-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6545b48489hbqng\" (UID: \"c165fa36-f6e3-481f-a56e-05f046c197cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489hbqng" Oct 30 18:04:02.520408 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:02.520408 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c165fa36-f6e3-481f-a56e-05f046c197cd-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6545b48489hbqng\" (UID: \"c165fa36-f6e3-481f-a56e-05f046c197cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489hbqng" Oct 30 18:04:02.520635 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:02.520434 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c165fa36-f6e3-481f-a56e-05f046c197cd-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6545b48489hbqng\" (UID: \"c165fa36-f6e3-481f-a56e-05f046c197cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489hbqng" Oct 30 18:04:02.520635 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:02.520532 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c165fa36-f6e3-481f-a56e-05f046c197cd-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6545b48489hbqng\" (UID: \"c165fa36-f6e3-481f-a56e-05f046c197cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489hbqng" Oct 30 18:04:02.520635 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:02.520584 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c165fa36-f6e3-481f-a56e-05f046c197cd-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6545b48489hbqng\" (UID: \"c165fa36-f6e3-481f-a56e-05f046c197cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489hbqng" Oct 30 18:04:02.520730 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:02.520653 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tndqg\" (UniqueName: \"kubernetes.io/projected/c165fa36-f6e3-481f-a56e-05f046c197cd-kube-api-access-tndqg\") pod \"data-science-gateway-data-science-gateway-class-6545b48489hbqng\" (UID: \"c165fa36-f6e3-481f-a56e-05f046c197cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489hbqng" Oct 30 18:04:02.520730 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:02.520693 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c165fa36-f6e3-481f-a56e-05f046c197cd-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6545b48489hbqng\" (UID: \"c165fa36-f6e3-481f-a56e-05f046c197cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489hbqng" Oct 30 18:04:02.520730 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:02.520716 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c165fa36-f6e3-481f-a56e-05f046c197cd-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6545b48489hbqng\" (UID: \"c165fa36-f6e3-481f-a56e-05f046c197cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489hbqng" Oct 30 18:04:02.520824 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:02.520752 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c165fa36-f6e3-481f-a56e-05f046c197cd-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6545b48489hbqng\" (UID: \"c165fa36-f6e3-481f-a56e-05f046c197cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489hbqng" Oct 30 18:04:02.621553 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:02.621456 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c165fa36-f6e3-481f-a56e-05f046c197cd-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6545b48489hbqng\" (UID: \"c165fa36-f6e3-481f-a56e-05f046c197cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489hbqng" Oct 30 18:04:02.621553 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:02.621519 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c165fa36-f6e3-481f-a56e-05f046c197cd-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6545b48489hbqng\" (UID: \"c165fa36-f6e3-481f-a56e-05f046c197cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489hbqng" Oct 30 18:04:02.621553 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:02.621542 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c165fa36-f6e3-481f-a56e-05f046c197cd-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6545b48489hbqng\" (UID: \"c165fa36-f6e3-481f-a56e-05f046c197cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489hbqng" Oct 30 18:04:02.621838 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:02.621574 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c165fa36-f6e3-481f-a56e-05f046c197cd-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6545b48489hbqng\" (UID: \"c165fa36-f6e3-481f-a56e-05f046c197cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489hbqng" Oct 30 18:04:02.621838 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:02.621603 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c165fa36-f6e3-481f-a56e-05f046c197cd-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6545b48489hbqng\" (UID: \"c165fa36-f6e3-481f-a56e-05f046c197cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489hbqng" Oct 30 18:04:02.621838 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:02.621628 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c165fa36-f6e3-481f-a56e-05f046c197cd-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6545b48489hbqng\" (UID: \"c165fa36-f6e3-481f-a56e-05f046c197cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489hbqng" Oct 30 18:04:02.622001 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:02.621835 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tndqg\" (UniqueName: \"kubernetes.io/projected/c165fa36-f6e3-481f-a56e-05f046c197cd-kube-api-access-tndqg\") pod \"data-science-gateway-data-science-gateway-class-6545b48489hbqng\" (UID: \"c165fa36-f6e3-481f-a56e-05f046c197cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489hbqng" Oct 30 18:04:02.622001 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:02.621900 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c165fa36-f6e3-481f-a56e-05f046c197cd-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6545b48489hbqng\" (UID: \"c165fa36-f6e3-481f-a56e-05f046c197cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489hbqng" Oct 30 18:04:02.622001 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:02.621938 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c165fa36-f6e3-481f-a56e-05f046c197cd-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6545b48489hbqng\" (UID: \"c165fa36-f6e3-481f-a56e-05f046c197cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489hbqng" Oct 30 18:04:02.622001 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:02.621976 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c165fa36-f6e3-481f-a56e-05f046c197cd-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6545b48489hbqng\" (UID: \"c165fa36-f6e3-481f-a56e-05f046c197cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489hbqng" Oct 30 18:04:02.622321 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:02.622295 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c165fa36-f6e3-481f-a56e-05f046c197cd-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6545b48489hbqng\" (UID: \"c165fa36-f6e3-481f-a56e-05f046c197cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489hbqng" Oct 30 18:04:02.622619 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:02.622594 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c165fa36-f6e3-481f-a56e-05f046c197cd-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6545b48489hbqng\" (UID: \"c165fa36-f6e3-481f-a56e-05f046c197cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489hbqng" Oct 30 18:04:02.622754 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:02.622626 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c165fa36-f6e3-481f-a56e-05f046c197cd-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6545b48489hbqng\" (UID: \"c165fa36-f6e3-481f-a56e-05f046c197cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489hbqng" Oct 30 18:04:02.622855 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:02.622641 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c165fa36-f6e3-481f-a56e-05f046c197cd-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6545b48489hbqng\" (UID: \"c165fa36-f6e3-481f-a56e-05f046c197cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489hbqng" Oct 30 18:04:02.624274 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:02.624254 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c165fa36-f6e3-481f-a56e-05f046c197cd-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6545b48489hbqng\" (UID: \"c165fa36-f6e3-481f-a56e-05f046c197cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489hbqng" Oct 30 18:04:02.624274 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:02.624260 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c165fa36-f6e3-481f-a56e-05f046c197cd-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6545b48489hbqng\" (UID: \"c165fa36-f6e3-481f-a56e-05f046c197cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489hbqng" Oct 30 18:04:02.647486 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:02.647458 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tndqg\" (UniqueName: \"kubernetes.io/projected/c165fa36-f6e3-481f-a56e-05f046c197cd-kube-api-access-tndqg\") pod \"data-science-gateway-data-science-gateway-class-6545b48489hbqng\" (UID: \"c165fa36-f6e3-481f-a56e-05f046c197cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489hbqng" Oct 30 18:04:02.649293 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:02.649272 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c165fa36-f6e3-481f-a56e-05f046c197cd-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6545b48489hbqng\" (UID: \"c165fa36-f6e3-481f-a56e-05f046c197cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489hbqng" Oct 30 18:04:02.680368 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:02.680321 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489hbqng" Oct 30 18:04:03.031590 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:03.031519 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489hbqng"] Oct 30 18:04:03.034496 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:04:03.034472 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc165fa36_f6e3_481f_a56e_05f046c197cd.slice/crio-4d88511848daf2272e8d894ffc7a071365baef8370976e86451f3dbb9e6ed1c5 WatchSource:0}: Error finding container 4d88511848daf2272e8d894ffc7a071365baef8370976e86451f3dbb9e6ed1c5: Status 404 returned error can't find the container with id 4d88511848daf2272e8d894ffc7a071365baef8370976e86451f3dbb9e6ed1c5 Oct 30 18:04:03.036542 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:03.036513 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:04:03.036645 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:03.036573 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:04:03.036645 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:03.036602 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:04:03.062651 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:03.062624 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61b3c946-73f9-4659-a01b-879caab4d17d" path="/var/lib/kubelet/pods/61b3c946-73f9-4659-a01b-879caab4d17d/volumes" Oct 30 18:04:03.825449 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:03.825417 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489hbqng" event={"ID":"c165fa36-f6e3-481f-a56e-05f046c197cd","Type":"ContainerStarted","Data":"9544e7d0b362b655895a51c37bb0da58bc53c24f61f53b148ffbfc9945cefe82"} Oct 30 18:04:03.825628 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:03.825466 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489hbqng" event={"ID":"c165fa36-f6e3-481f-a56e-05f046c197cd","Type":"ContainerStarted","Data":"4d88511848daf2272e8d894ffc7a071365baef8370976e86451f3dbb9e6ed1c5"} Oct 30 18:04:03.879245 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:03.879197 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489hbqng" podStartSLOduration=1.879183878 podStartE2EDuration="1.879183878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:04:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:04:03.878659728 +0000 UTC m=+340.968722061" watchObservedRunningTime="2026-10-30 18:04:03.879183878 +0000 UTC m=+340.969246217" Oct 30 18:04:04.680591 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:04.680565 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489hbqng" Oct 30 18:04:04.685414 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:04.685391 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489hbqng" Oct 30 18:04:04.831212 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:04.831169 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489hbqng" Oct 30 18:04:04.832518 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:04.832491 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6545b48489hbqng" Oct 30 18:04:05.756315 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:05.756282 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-nxg2g"] Oct 30 18:04:05.767180 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:05.761048 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nxg2g" Oct 30 18:04:05.786916 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:05.786888 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-fkldn\"" Oct 30 18:04:05.811258 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:05.811227 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 18:04:05.811566 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:05.811548 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 30 18:04:05.813381 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:05.813356 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-nxg2g"] Oct 30 18:04:05.851674 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:05.851647 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dd0fb59a-f6bb-43fd-95e4-6f740ace1c6e-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-nxg2g\" (UID: \"dd0fb59a-f6bb-43fd-95e4-6f740ace1c6e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nxg2g" Oct 30 18:04:05.851674 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:05.851676 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd0fb59a-f6bb-43fd-95e4-6f740ace1c6e-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-nxg2g\" (UID: \"dd0fb59a-f6bb-43fd-95e4-6f740ace1c6e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nxg2g" Oct 30 18:04:05.851894 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:05.851709 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dd0fb59a-f6bb-43fd-95e4-6f740ace1c6e-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-nxg2g\" (UID: \"dd0fb59a-f6bb-43fd-95e4-6f740ace1c6e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nxg2g" Oct 30 18:04:05.851894 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:05.851874 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4877j\" (UniqueName: \"kubernetes.io/projected/dd0fb59a-f6bb-43fd-95e4-6f740ace1c6e-kube-api-access-4877j\") pod \"kserve-module-controller-manager-7566cf5fc8-nxg2g\" (UID: \"dd0fb59a-f6bb-43fd-95e4-6f740ace1c6e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nxg2g" Oct 30 18:04:05.876026 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:05.875995 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 18:04:05.952702 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:05.952668 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dd0fb59a-f6bb-43fd-95e4-6f740ace1c6e-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-nxg2g\" (UID: \"dd0fb59a-f6bb-43fd-95e4-6f740ace1c6e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nxg2g" Oct 30 18:04:05.952900 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:05.952711 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd0fb59a-f6bb-43fd-95e4-6f740ace1c6e-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-nxg2g\" (UID: \"dd0fb59a-f6bb-43fd-95e4-6f740ace1c6e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nxg2g" Oct 30 18:04:05.952900 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:05.952748 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dd0fb59a-f6bb-43fd-95e4-6f740ace1c6e-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-nxg2g\" (UID: \"dd0fb59a-f6bb-43fd-95e4-6f740ace1c6e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nxg2g" Oct 30 18:04:05.952900 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:05.952820 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4877j\" (UniqueName: \"kubernetes.io/projected/dd0fb59a-f6bb-43fd-95e4-6f740ace1c6e-kube-api-access-4877j\") pod \"kserve-module-controller-manager-7566cf5fc8-nxg2g\" (UID: \"dd0fb59a-f6bb-43fd-95e4-6f740ace1c6e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nxg2g" Oct 30 18:04:05.953242 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:05.953219 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd0fb59a-f6bb-43fd-95e4-6f740ace1c6e-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-nxg2g\" (UID: \"dd0fb59a-f6bb-43fd-95e4-6f740ace1c6e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nxg2g" Oct 30 18:04:05.953346 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:05.953329 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dd0fb59a-f6bb-43fd-95e4-6f740ace1c6e-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-nxg2g\" (UID: \"dd0fb59a-f6bb-43fd-95e4-6f740ace1c6e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nxg2g" Oct 30 18:04:05.953801 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:05.953785 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dd0fb59a-f6bb-43fd-95e4-6f740ace1c6e-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-nxg2g\" (UID: \"dd0fb59a-f6bb-43fd-95e4-6f740ace1c6e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nxg2g" Oct 30 18:04:05.971798 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:05.971771 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4877j\" (UniqueName: \"kubernetes.io/projected/dd0fb59a-f6bb-43fd-95e4-6f740ace1c6e-kube-api-access-4877j\") pod \"kserve-module-controller-manager-7566cf5fc8-nxg2g\" (UID: \"dd0fb59a-f6bb-43fd-95e4-6f740ace1c6e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nxg2g" Oct 30 18:04:06.078702 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:06.078623 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nxg2g" Oct 30 18:04:06.402614 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:06.402521 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-nxg2g"] Oct 30 18:04:06.404778 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:04:06.404752 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd0fb59a_f6bb_43fd_95e4_6f740ace1c6e.slice/crio-3b6d855fe750ab7d29aac5288fee77e3092aadcedc30d76e35d98e57d075c7f5 WatchSource:0}: Error finding container 3b6d855fe750ab7d29aac5288fee77e3092aadcedc30d76e35d98e57d075c7f5: Status 404 returned error can't find the container with id 3b6d855fe750ab7d29aac5288fee77e3092aadcedc30d76e35d98e57d075c7f5 Oct 30 18:04:06.847592 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:06.847501 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nxg2g" event={"ID":"dd0fb59a-f6bb-43fd-95e4-6f740ace1c6e","Type":"ContainerStarted","Data":"3b6d855fe750ab7d29aac5288fee77e3092aadcedc30d76e35d98e57d075c7f5"} Oct 30 18:04:08.860216 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:08.860175 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nxg2g" event={"ID":"dd0fb59a-f6bb-43fd-95e4-6f740ace1c6e","Type":"ContainerStarted","Data":"43fc76a3115ada38135ee4c071d80fab45528a63640dcfaf0d69cb587c357d20"} Oct 30 18:04:08.860607 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:08.860247 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nxg2g" Oct 30 18:04:08.911498 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:08.911396 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nxg2g" podStartSLOduration=1.667593323 podStartE2EDuration="3.911381302s" totalImagesPullingTime="2.243787979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:04:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:04:08.910126824 +0000 UTC m=+346.000189163" watchObservedRunningTime="2026-10-30 18:04:08.911381302 +0000 UTC m=+346.001443640" Oct 30 18:04:17.558604 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:04:17.558570 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:04:21.579398 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:21.579365 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-8bfcq"] Oct 30 18:04:21.582151 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:21.582114 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-8bfcq" Oct 30 18:04:21.590851 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:21.590830 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcbch\" (UniqueName: \"kubernetes.io/projected/c811e9f0-0012-4d3c-bfb7-5a1f724a8388-kube-api-access-zcbch\") pod \"limitador-limitador-74dfcfb894-8bfcq\" (UID: \"c811e9f0-0012-4d3c-bfb7-5a1f724a8388\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-8bfcq" Oct 30 18:04:21.590972 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:21.590922 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c811e9f0-0012-4d3c-bfb7-5a1f724a8388-config-file\") pod \"limitador-limitador-74dfcfb894-8bfcq\" (UID: \"c811e9f0-0012-4d3c-bfb7-5a1f724a8388\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-8bfcq" Oct 30 18:04:21.596768 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:21.596750 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 18:04:21.608454 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:21.608427 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-ks7zb\"" Oct 30 18:04:21.611856 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:21.611832 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-8bfcq"] Oct 30 18:04:21.691953 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:21.691914 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcbch\" (UniqueName: \"kubernetes.io/projected/c811e9f0-0012-4d3c-bfb7-5a1f724a8388-kube-api-access-zcbch\") pod \"limitador-limitador-74dfcfb894-8bfcq\" (UID: \"c811e9f0-0012-4d3c-bfb7-5a1f724a8388\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-8bfcq" Oct 30 18:04:21.692162 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:21.692015 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c811e9f0-0012-4d3c-bfb7-5a1f724a8388-config-file\") pod \"limitador-limitador-74dfcfb894-8bfcq\" (UID: \"c811e9f0-0012-4d3c-bfb7-5a1f724a8388\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-8bfcq" Oct 30 18:04:21.692781 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:21.692761 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c811e9f0-0012-4d3c-bfb7-5a1f724a8388-config-file\") pod \"limitador-limitador-74dfcfb894-8bfcq\" (UID: \"c811e9f0-0012-4d3c-bfb7-5a1f724a8388\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-8bfcq" Oct 30 18:04:21.711488 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:21.711461 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcbch\" (UniqueName: \"kubernetes.io/projected/c811e9f0-0012-4d3c-bfb7-5a1f724a8388-kube-api-access-zcbch\") pod \"limitador-limitador-74dfcfb894-8bfcq\" (UID: \"c811e9f0-0012-4d3c-bfb7-5a1f724a8388\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-8bfcq" Oct 30 18:04:21.717245 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:21.717218 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-8bfcq"] Oct 30 18:04:21.893859 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:21.893823 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-8bfcq" Oct 30 18:04:22.189103 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:22.189018 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-8bfcq"] Oct 30 18:04:22.192870 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:04:22.192845 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc811e9f0_0012_4d3c_bfb7_5a1f724a8388.slice/crio-85aad703f67c2b1ea536fc9278463fab101a1883c6c849ecedf8958313fac3a1 WatchSource:0}: Error finding container 85aad703f67c2b1ea536fc9278463fab101a1883c6c849ecedf8958313fac3a1: Status 404 returned error can't find the container with id 85aad703f67c2b1ea536fc9278463fab101a1883c6c849ecedf8958313fac3a1 Oct 30 18:04:22.438795 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:22.438762 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-tf6wx"] Oct 30 18:04:22.442020 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:22.441961 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-tf6wx" Oct 30 18:04:22.459970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:22.459942 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-tf6wx"] Oct 30 18:04:22.469294 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:22.469269 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-t6wvx\"" Oct 30 18:04:22.496538 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:22.496491 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5527\" (UniqueName: \"kubernetes.io/projected/f6b6bbce-fc39-45bd-9c86-980c9cf5d3c8-kube-api-access-m5527\") pod \"authorino-57f7f8659f-tf6wx\" (UID: \"f6b6bbce-fc39-45bd-9c86-980c9cf5d3c8\") " pod="kuadrant-system/authorino-57f7f8659f-tf6wx" Oct 30 18:04:22.597265 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:22.597234 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5527\" (UniqueName: \"kubernetes.io/projected/f6b6bbce-fc39-45bd-9c86-980c9cf5d3c8-kube-api-access-m5527\") pod \"authorino-57f7f8659f-tf6wx\" (UID: \"f6b6bbce-fc39-45bd-9c86-980c9cf5d3c8\") " pod="kuadrant-system/authorino-57f7f8659f-tf6wx" Oct 30 18:04:22.617555 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:22.617528 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5527\" (UniqueName: \"kubernetes.io/projected/f6b6bbce-fc39-45bd-9c86-980c9cf5d3c8-kube-api-access-m5527\") pod \"authorino-57f7f8659f-tf6wx\" (UID: \"f6b6bbce-fc39-45bd-9c86-980c9cf5d3c8\") " pod="kuadrant-system/authorino-57f7f8659f-tf6wx" Oct 30 18:04:22.766894 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:22.766806 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-tf6wx" Oct 30 18:04:22.932083 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:22.932041 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-8bfcq" event={"ID":"c811e9f0-0012-4d3c-bfb7-5a1f724a8388","Type":"ContainerStarted","Data":"85aad703f67c2b1ea536fc9278463fab101a1883c6c849ecedf8958313fac3a1"} Oct 30 18:04:23.066618 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:23.066586 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-tf6wx"] Oct 30 18:04:23.938781 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:23.938746 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-tf6wx" event={"ID":"f6b6bbce-fc39-45bd-9c86-980c9cf5d3c8","Type":"ContainerStarted","Data":"673a9195e07a22b8565911768eb3794dea19c4c43e89a5c0e4f756beffb014c9"} Oct 30 18:04:23.940978 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:23.940941 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-8bfcq" event={"ID":"c811e9f0-0012-4d3c-bfb7-5a1f724a8388","Type":"ContainerStarted","Data":"6b9246d62793bd4b3de3125dee5754f0fdb1d7bfdd59587a64812b471cbceeee"} Oct 30 18:04:23.941145 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:23.941086 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-8bfcq" Oct 30 18:04:23.980351 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:23.980305 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-8bfcq" podStartSLOduration=1.773060284 podStartE2EDuration="2.980285035s" totalImagesPullingTime="1.207224751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:04:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:04:23.979322361 +0000 UTC m=+361.069384702" watchObservedRunningTime="2026-10-30 18:04:23.980285035 +0000 UTC m=+361.070347417" Oct 30 18:04:24.951103 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:24.951058 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-tf6wx" event={"ID":"f6b6bbce-fc39-45bd-9c86-980c9cf5d3c8","Type":"ContainerStarted","Data":"32dfeef7d516d20244949de7bbf0c9ac140ae20067082e7388e6b7dd0cf9c39c"} Oct 30 18:04:24.988448 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:24.988392 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-tf6wx" podStartSLOduration=1.510888078 podStartE2EDuration="2.988378072s" totalImagesPullingTime="1.477489994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:04:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:04:24.98779291 +0000 UTC m=+362.077855246" watchObservedRunningTime="2026-10-30 18:04:24.988378072 +0000 UTC m=+362.078440436" Oct 30 18:04:26.682656 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:26.682626 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-tf6wx"] Oct 30 18:04:26.961014 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:26.960897 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-tf6wx" podUID="f6b6bbce-fc39-45bd-9c86-980c9cf5d3c8" containerName="authorino" containerID="cri-o://32dfeef7d516d20244949de7bbf0c9ac140ae20067082e7388e6b7dd0cf9c39c" gracePeriod=30 Oct 30 18:04:27.204256 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:27.204231 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-tf6wx" Oct 30 18:04:27.230316 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:27.230247 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6b6bbce-fc39-45bd-9c86-980c9cf5d3c8-kube-api-access-m5527\" (UniqueName: \"kubernetes.io/projected/f6b6bbce-fc39-45bd-9c86-980c9cf5d3c8-kube-api-access-m5527\") pod \"f6b6bbce-fc39-45bd-9c86-980c9cf5d3c8\" (UID: \"f6b6bbce-fc39-45bd-9c86-980c9cf5d3c8\") " Oct 30 18:04:27.232462 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:27.232435 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6b6bbce-fc39-45bd-9c86-980c9cf5d3c8-kube-api-access-m5527" pod "f6b6bbce-fc39-45bd-9c86-980c9cf5d3c8" (UID: "f6b6bbce-fc39-45bd-9c86-980c9cf5d3c8"). InnerVolumeSpecName "kube-api-access-m5527". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:04:27.331023 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:27.330984 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5527\" (UniqueName: \"kubernetes.io/projected/f6b6bbce-fc39-45bd-9c86-980c9cf5d3c8-kube-api-access-m5527\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:04:27.966266 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:27.966232 4442 generic.go:356] "Generic (PLEG): container finished" podID="f6b6bbce-fc39-45bd-9c86-980c9cf5d3c8" containerID="32dfeef7d516d20244949de7bbf0c9ac140ae20067082e7388e6b7dd0cf9c39c" exitCode=0 Oct 30 18:04:27.966779 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:27.966275 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-tf6wx" Oct 30 18:04:27.966779 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:27.966326 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-tf6wx" event={"ID":"f6b6bbce-fc39-45bd-9c86-980c9cf5d3c8","Type":"ContainerDied","Data":"32dfeef7d516d20244949de7bbf0c9ac140ae20067082e7388e6b7dd0cf9c39c"} Oct 30 18:04:27.966779 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:27.966379 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-tf6wx" event={"ID":"f6b6bbce-fc39-45bd-9c86-980c9cf5d3c8","Type":"ContainerDied","Data":"673a9195e07a22b8565911768eb3794dea19c4c43e89a5c0e4f756beffb014c9"} Oct 30 18:04:27.966779 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:27.966404 4442 scope.go:122] "RemoveContainer" containerID="32dfeef7d516d20244949de7bbf0c9ac140ae20067082e7388e6b7dd0cf9c39c" Oct 30 18:04:27.975571 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:27.975552 4442 scope.go:122] "RemoveContainer" containerID="32dfeef7d516d20244949de7bbf0c9ac140ae20067082e7388e6b7dd0cf9c39c" Oct 30 18:04:27.975862 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:04:27.975844 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32dfeef7d516d20244949de7bbf0c9ac140ae20067082e7388e6b7dd0cf9c39c\": container with ID starting with 32dfeef7d516d20244949de7bbf0c9ac140ae20067082e7388e6b7dd0cf9c39c not found: ID does not exist" containerID="32dfeef7d516d20244949de7bbf0c9ac140ae20067082e7388e6b7dd0cf9c39c" Oct 30 18:04:27.975926 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:27.975873 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32dfeef7d516d20244949de7bbf0c9ac140ae20067082e7388e6b7dd0cf9c39c"} err="failed to get container status \"32dfeef7d516d20244949de7bbf0c9ac140ae20067082e7388e6b7dd0cf9c39c\": rpc error: code = NotFound desc = could not find container \"32dfeef7d516d20244949de7bbf0c9ac140ae20067082e7388e6b7dd0cf9c39c\": container with ID starting with 32dfeef7d516d20244949de7bbf0c9ac140ae20067082e7388e6b7dd0cf9c39c not found: ID does not exist" Oct 30 18:04:28.024424 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:28.024379 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-tf6wx"] Oct 30 18:04:28.035674 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:28.035643 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-tf6wx"] Oct 30 18:04:29.061952 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:29.061920 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6b6bbce-fc39-45bd-9c86-980c9cf5d3c8" path="/var/lib/kubelet/pods/f6b6bbce-fc39-45bd-9c86-980c9cf5d3c8/volumes" Oct 30 18:04:29.866830 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:29.866800 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nxg2g" Oct 30 18:04:34.952872 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:34.952836 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-8bfcq" Oct 30 18:04:37.605682 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:37.605639 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-8bfcq"] Oct 30 18:04:37.606084 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:37.605933 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-8bfcq" podUID="c811e9f0-0012-4d3c-bfb7-5a1f724a8388" containerName="limitador" containerID="cri-o://6b9246d62793bd4b3de3125dee5754f0fdb1d7bfdd59587a64812b471cbceeee" gracePeriod=30 Oct 30 18:04:38.157619 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:38.157572 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-8bfcq" Oct 30 18:04:38.228889 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:38.228798 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c811e9f0-0012-4d3c-bfb7-5a1f724a8388-kube-api-access-zcbch\" (UniqueName: \"kubernetes.io/projected/c811e9f0-0012-4d3c-bfb7-5a1f724a8388-kube-api-access-zcbch\") pod \"c811e9f0-0012-4d3c-bfb7-5a1f724a8388\" (UID: \"c811e9f0-0012-4d3c-bfb7-5a1f724a8388\") " Oct 30 18:04:38.228889 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:38.228854 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c811e9f0-0012-4d3c-bfb7-5a1f724a8388-config-file\" (UniqueName: \"kubernetes.io/configmap/c811e9f0-0012-4d3c-bfb7-5a1f724a8388-config-file\") pod \"c811e9f0-0012-4d3c-bfb7-5a1f724a8388\" (UID: \"c811e9f0-0012-4d3c-bfb7-5a1f724a8388\") " Oct 30 18:04:38.229292 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:38.229266 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c811e9f0-0012-4d3c-bfb7-5a1f724a8388-config-file" pod "c811e9f0-0012-4d3c-bfb7-5a1f724a8388" (UID: "c811e9f0-0012-4d3c-bfb7-5a1f724a8388"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:04:38.231074 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:38.231031 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c811e9f0-0012-4d3c-bfb7-5a1f724a8388-kube-api-access-zcbch" pod "c811e9f0-0012-4d3c-bfb7-5a1f724a8388" (UID: "c811e9f0-0012-4d3c-bfb7-5a1f724a8388"). InnerVolumeSpecName "kube-api-access-zcbch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:04:38.329833 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:38.329800 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcbch\" (UniqueName: \"kubernetes.io/projected/c811e9f0-0012-4d3c-bfb7-5a1f724a8388-kube-api-access-zcbch\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:04:38.329833 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:38.329831 4442 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c811e9f0-0012-4d3c-bfb7-5a1f724a8388-config-file\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:04:39.021605 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:39.021576 4442 generic.go:356] "Generic (PLEG): container finished" podID="c811e9f0-0012-4d3c-bfb7-5a1f724a8388" containerID="6b9246d62793bd4b3de3125dee5754f0fdb1d7bfdd59587a64812b471cbceeee" exitCode=0 Oct 30 18:04:39.021605 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:39.021610 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-8bfcq" event={"ID":"c811e9f0-0012-4d3c-bfb7-5a1f724a8388","Type":"ContainerDied","Data":"6b9246d62793bd4b3de3125dee5754f0fdb1d7bfdd59587a64812b471cbceeee"} Oct 30 18:04:39.022094 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:39.021627 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-8bfcq" Oct 30 18:04:39.022094 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:39.021648 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-8bfcq" event={"ID":"c811e9f0-0012-4d3c-bfb7-5a1f724a8388","Type":"ContainerDied","Data":"85aad703f67c2b1ea536fc9278463fab101a1883c6c849ecedf8958313fac3a1"} Oct 30 18:04:39.022094 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:39.021668 4442 scope.go:122] "RemoveContainer" containerID="6b9246d62793bd4b3de3125dee5754f0fdb1d7bfdd59587a64812b471cbceeee" Oct 30 18:04:39.031814 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:39.031794 4442 scope.go:122] "RemoveContainer" containerID="6b9246d62793bd4b3de3125dee5754f0fdb1d7bfdd59587a64812b471cbceeee" Oct 30 18:04:39.032171 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:04:39.032118 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b9246d62793bd4b3de3125dee5754f0fdb1d7bfdd59587a64812b471cbceeee\": container with ID starting with 6b9246d62793bd4b3de3125dee5754f0fdb1d7bfdd59587a64812b471cbceeee not found: ID does not exist" containerID="6b9246d62793bd4b3de3125dee5754f0fdb1d7bfdd59587a64812b471cbceeee" Oct 30 18:04:39.032278 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:39.032172 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b9246d62793bd4b3de3125dee5754f0fdb1d7bfdd59587a64812b471cbceeee"} err="failed to get container status \"6b9246d62793bd4b3de3125dee5754f0fdb1d7bfdd59587a64812b471cbceeee\": rpc error: code = NotFound desc = could not find container \"6b9246d62793bd4b3de3125dee5754f0fdb1d7bfdd59587a64812b471cbceeee\": container with ID starting with 6b9246d62793bd4b3de3125dee5754f0fdb1d7bfdd59587a64812b471cbceeee not found: ID does not exist" Oct 30 18:04:39.075499 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:39.075467 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-8bfcq"] Oct 30 18:04:39.086593 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:39.086568 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-8bfcq"] Oct 30 18:04:40.209361 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:40.209320 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-d5s69"] Oct 30 18:04:40.210348 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:40.210315 4442 cpu_manager.go:414] "containerMap: removing container" podUID="c811e9f0-0012-4d3c-bfb7-5a1f724a8388" containerName="limitador" Oct 30 18:04:40.210348 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:40.210338 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c811e9f0-0012-4d3c-bfb7-5a1f724a8388" containerName="limitador" Oct 30 18:04:40.210535 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:40.210387 4442 cpu_manager.go:414] "containerMap: removing container" podUID="f6b6bbce-fc39-45bd-9c86-980c9cf5d3c8" containerName="authorino" Oct 30 18:04:40.210535 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:40.210395 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6b6bbce-fc39-45bd-9c86-980c9cf5d3c8" containerName="authorino" Oct 30 18:04:40.210535 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:40.210470 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="c811e9f0-0012-4d3c-bfb7-5a1f724a8388" containerName="limitador" Oct 30 18:04:40.210535 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:40.210487 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6b6bbce-fc39-45bd-9c86-980c9cf5d3c8" containerName="authorino" Oct 30 18:04:40.215786 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:40.215765 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-d5s69" Oct 30 18:04:40.231638 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:40.231591 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 30 18:04:40.233384 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:40.233355 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-d5s69"] Oct 30 18:04:40.242298 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:40.242274 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:04:40.242298 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:40.242281 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:04:40.242486 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:40.242282 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-qxdhq\"" Oct 30 18:04:40.246627 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:40.246607 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c8c00dd2-1002-436b-810d-7555fc4286df-data\") pod \"postgres-7cdbf6dc44-d5s69\" (UID: \"c8c00dd2-1002-436b-810d-7555fc4286df\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-d5s69" Oct 30 18:04:40.246755 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:40.246645 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2n5z\" (UniqueName: \"kubernetes.io/projected/c8c00dd2-1002-436b-810d-7555fc4286df-kube-api-access-k2n5z\") pod \"postgres-7cdbf6dc44-d5s69\" (UID: \"c8c00dd2-1002-436b-810d-7555fc4286df\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-d5s69" Oct 30 18:04:40.348022 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:40.347983 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c8c00dd2-1002-436b-810d-7555fc4286df-data\") pod \"postgres-7cdbf6dc44-d5s69\" (UID: \"c8c00dd2-1002-436b-810d-7555fc4286df\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-d5s69" Oct 30 18:04:40.348254 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:40.348051 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2n5z\" (UniqueName: \"kubernetes.io/projected/c8c00dd2-1002-436b-810d-7555fc4286df-kube-api-access-k2n5z\") pod \"postgres-7cdbf6dc44-d5s69\" (UID: \"c8c00dd2-1002-436b-810d-7555fc4286df\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-d5s69" Oct 30 18:04:40.348448 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:40.348427 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c8c00dd2-1002-436b-810d-7555fc4286df-data\") pod \"postgres-7cdbf6dc44-d5s69\" (UID: \"c8c00dd2-1002-436b-810d-7555fc4286df\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-d5s69" Oct 30 18:04:40.367880 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:40.367845 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2n5z\" (UniqueName: \"kubernetes.io/projected/c8c00dd2-1002-436b-810d-7555fc4286df-kube-api-access-k2n5z\") pod \"postgres-7cdbf6dc44-d5s69\" (UID: \"c8c00dd2-1002-436b-810d-7555fc4286df\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-d5s69" Oct 30 18:04:40.528582 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:40.528500 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-d5s69" Oct 30 18:04:40.823812 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:40.823729 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-d5s69"] Oct 30 18:04:41.035022 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:41.034983 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-d5s69" event={"ID":"c8c00dd2-1002-436b-810d-7555fc4286df","Type":"ContainerStarted","Data":"86bea855502171699d3af2f9d9a08175228eca5a501499c9bf80f66b47124923"} Oct 30 18:04:41.062523 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:41.062487 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c811e9f0-0012-4d3c-bfb7-5a1f724a8388" path="/var/lib/kubelet/pods/c811e9f0-0012-4d3c-bfb7-5a1f724a8388/volumes" Oct 30 18:04:47.074124 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:47.074090 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-d5s69" event={"ID":"c8c00dd2-1002-436b-810d-7555fc4286df","Type":"ContainerStarted","Data":"0f00e131fe7f0e45dedbab414dbff9095851d837354973a20a87e41422e12b63"} Oct 30 18:04:47.074586 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:47.074220 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-d5s69" Oct 30 18:04:47.111971 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:47.111921 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-d5s69" podStartSLOduration=1.753517715 podStartE2EDuration="7.111908037s" totalImagesPullingTime="5.358390322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:04:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:04:47.111805349 +0000 UTC m=+384.201867687" watchObservedRunningTime="2026-10-30 18:04:47.111908037 +0000 UTC m=+384.201970366" Oct 30 18:04:47.565889 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:04:47.565851 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:04:53.108562 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:04:53.108535 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-d5s69" Oct 30 18:05:00.745156 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:00.745107 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-76f74769d4-kf4sm"] Oct 30 18:05:00.753309 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:00.753281 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76f74769d4-kf4sm" Oct 30 18:05:00.791785 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:00.791754 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 18:05:00.799034 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:00.798987 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 18:05:00.799034 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:00.799018 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-rl5p8\"" Oct 30 18:05:00.799294 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:00.799082 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 18:05:00.802275 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:00.802251 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-76f74769d4-kf4sm"] Oct 30 18:05:00.828739 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:00.828702 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqhgr\" (UniqueName: \"kubernetes.io/projected/ab1e2145-cb53-4f9d-a76a-804b0545f59c-kube-api-access-gqhgr\") pod \"maas-controller-76f74769d4-kf4sm\" (UID: \"ab1e2145-cb53-4f9d-a76a-804b0545f59c\") " pod="opendatahub/maas-controller-76f74769d4-kf4sm" Oct 30 18:05:00.828739 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:00.828740 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ab1e2145-cb53-4f9d-a76a-804b0545f59c-webhook-cert\") pod \"maas-controller-76f74769d4-kf4sm\" (UID: \"ab1e2145-cb53-4f9d-a76a-804b0545f59c\") " pod="opendatahub/maas-controller-76f74769d4-kf4sm" Oct 30 18:05:00.828956 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:00.828773 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab1e2145-cb53-4f9d-a76a-804b0545f59c-metrics-certs\") pod \"maas-controller-76f74769d4-kf4sm\" (UID: \"ab1e2145-cb53-4f9d-a76a-804b0545f59c\") " pod="opendatahub/maas-controller-76f74769d4-kf4sm" Oct 30 18:05:00.929865 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:00.929833 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqhgr\" (UniqueName: \"kubernetes.io/projected/ab1e2145-cb53-4f9d-a76a-804b0545f59c-kube-api-access-gqhgr\") pod \"maas-controller-76f74769d4-kf4sm\" (UID: \"ab1e2145-cb53-4f9d-a76a-804b0545f59c\") " pod="opendatahub/maas-controller-76f74769d4-kf4sm" Oct 30 18:05:00.929865 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:00.929870 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ab1e2145-cb53-4f9d-a76a-804b0545f59c-webhook-cert\") pod \"maas-controller-76f74769d4-kf4sm\" (UID: \"ab1e2145-cb53-4f9d-a76a-804b0545f59c\") " pod="opendatahub/maas-controller-76f74769d4-kf4sm" Oct 30 18:05:00.930125 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:00.929997 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab1e2145-cb53-4f9d-a76a-804b0545f59c-metrics-certs\") pod \"maas-controller-76f74769d4-kf4sm\" (UID: \"ab1e2145-cb53-4f9d-a76a-804b0545f59c\") " pod="opendatahub/maas-controller-76f74769d4-kf4sm" Oct 30 18:05:00.932821 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:00.932789 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ab1e2145-cb53-4f9d-a76a-804b0545f59c-webhook-cert\") pod \"maas-controller-76f74769d4-kf4sm\" (UID: \"ab1e2145-cb53-4f9d-a76a-804b0545f59c\") " pod="opendatahub/maas-controller-76f74769d4-kf4sm" Oct 30 18:05:00.932821 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:00.932806 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab1e2145-cb53-4f9d-a76a-804b0545f59c-metrics-certs\") pod \"maas-controller-76f74769d4-kf4sm\" (UID: \"ab1e2145-cb53-4f9d-a76a-804b0545f59c\") " pod="opendatahub/maas-controller-76f74769d4-kf4sm" Oct 30 18:05:00.960326 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:00.960293 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqhgr\" (UniqueName: \"kubernetes.io/projected/ab1e2145-cb53-4f9d-a76a-804b0545f59c-kube-api-access-gqhgr\") pod \"maas-controller-76f74769d4-kf4sm\" (UID: \"ab1e2145-cb53-4f9d-a76a-804b0545f59c\") " pod="opendatahub/maas-controller-76f74769d4-kf4sm" Oct 30 18:05:01.066028 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:01.065946 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76f74769d4-kf4sm" Oct 30 18:05:01.306230 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:01.306192 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-67cc46c64-66wvf"] Oct 30 18:05:01.311018 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:01.310993 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67cc46c64-66wvf" Oct 30 18:05:01.339437 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:01.339405 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67cc46c64-66wvf"] Oct 30 18:05:01.380048 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:01.380002 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-76f74769d4-kf4sm"] Oct 30 18:05:01.383633 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:05:01.383605 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab1e2145_cb53_4f9d_a76a_804b0545f59c.slice/crio-8562987190bb4a5da28c87321f5249051f79908ec842bf0d6e8d0bb038b4d9e7 WatchSource:0}: Error finding container 8562987190bb4a5da28c87321f5249051f79908ec842bf0d6e8d0bb038b4d9e7: Status 404 returned error can't find the container with id 8562987190bb4a5da28c87321f5249051f79908ec842bf0d6e8d0bb038b4d9e7 Oct 30 18:05:01.434531 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:01.434497 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/024b2852-2e03-4f0b-a7dd-858dc9ed69ed-metrics-certs\") pod \"maas-controller-67cc46c64-66wvf\" (UID: \"024b2852-2e03-4f0b-a7dd-858dc9ed69ed\") " pod="opendatahub/maas-controller-67cc46c64-66wvf" Oct 30 18:05:01.434691 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:01.434544 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/024b2852-2e03-4f0b-a7dd-858dc9ed69ed-webhook-cert\") pod \"maas-controller-67cc46c64-66wvf\" (UID: \"024b2852-2e03-4f0b-a7dd-858dc9ed69ed\") " pod="opendatahub/maas-controller-67cc46c64-66wvf" Oct 30 18:05:01.434691 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:01.434641 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlbc6\" (UniqueName: \"kubernetes.io/projected/024b2852-2e03-4f0b-a7dd-858dc9ed69ed-kube-api-access-qlbc6\") pod \"maas-controller-67cc46c64-66wvf\" (UID: \"024b2852-2e03-4f0b-a7dd-858dc9ed69ed\") " pod="opendatahub/maas-controller-67cc46c64-66wvf" Oct 30 18:05:01.535702 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:01.535666 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/024b2852-2e03-4f0b-a7dd-858dc9ed69ed-webhook-cert\") pod \"maas-controller-67cc46c64-66wvf\" (UID: \"024b2852-2e03-4f0b-a7dd-858dc9ed69ed\") " pod="opendatahub/maas-controller-67cc46c64-66wvf" Oct 30 18:05:01.535912 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:01.535720 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlbc6\" (UniqueName: \"kubernetes.io/projected/024b2852-2e03-4f0b-a7dd-858dc9ed69ed-kube-api-access-qlbc6\") pod \"maas-controller-67cc46c64-66wvf\" (UID: \"024b2852-2e03-4f0b-a7dd-858dc9ed69ed\") " pod="opendatahub/maas-controller-67cc46c64-66wvf" Oct 30 18:05:01.535912 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:01.535796 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/024b2852-2e03-4f0b-a7dd-858dc9ed69ed-metrics-certs\") pod \"maas-controller-67cc46c64-66wvf\" (UID: \"024b2852-2e03-4f0b-a7dd-858dc9ed69ed\") " pod="opendatahub/maas-controller-67cc46c64-66wvf" Oct 30 18:05:01.538577 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:01.538550 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/024b2852-2e03-4f0b-a7dd-858dc9ed69ed-metrics-certs\") pod \"maas-controller-67cc46c64-66wvf\" (UID: \"024b2852-2e03-4f0b-a7dd-858dc9ed69ed\") " pod="opendatahub/maas-controller-67cc46c64-66wvf" Oct 30 18:05:01.538721 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:01.538550 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/024b2852-2e03-4f0b-a7dd-858dc9ed69ed-webhook-cert\") pod \"maas-controller-67cc46c64-66wvf\" (UID: \"024b2852-2e03-4f0b-a7dd-858dc9ed69ed\") " pod="opendatahub/maas-controller-67cc46c64-66wvf" Oct 30 18:05:01.556248 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:01.556222 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlbc6\" (UniqueName: \"kubernetes.io/projected/024b2852-2e03-4f0b-a7dd-858dc9ed69ed-kube-api-access-qlbc6\") pod \"maas-controller-67cc46c64-66wvf\" (UID: \"024b2852-2e03-4f0b-a7dd-858dc9ed69ed\") " pod="opendatahub/maas-controller-67cc46c64-66wvf" Oct 30 18:05:01.625524 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:01.625497 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67cc46c64-66wvf" Oct 30 18:05:01.848693 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:01.848664 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j44sc"] Oct 30 18:05:01.852493 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:01.852470 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j44sc" Oct 30 18:05:01.869687 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:01.869669 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 18:05:01.881263 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:01.880886 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 18:05:01.881263 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:01.880962 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 18:05:01.881263 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:01.880962 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-4jrp2\"" Oct 30 18:05:01.884186 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:01.884165 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j44sc"] Oct 30 18:05:01.939414 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:01.939384 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8e1a75a1-61d7-4285-bf95-ad9b23b57ad2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j44sc\" (UID: \"8e1a75a1-61d7-4285-bf95-ad9b23b57ad2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j44sc" Oct 30 18:05:01.939603 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:01.939438 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e1a75a1-61d7-4285-bf95-ad9b23b57ad2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j44sc\" (UID: \"8e1a75a1-61d7-4285-bf95-ad9b23b57ad2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j44sc" Oct 30 18:05:01.939603 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:01.939465 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8e1a75a1-61d7-4285-bf95-ad9b23b57ad2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j44sc\" (UID: \"8e1a75a1-61d7-4285-bf95-ad9b23b57ad2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j44sc" Oct 30 18:05:01.939726 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:01.939605 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8e1a75a1-61d7-4285-bf95-ad9b23b57ad2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j44sc\" (UID: \"8e1a75a1-61d7-4285-bf95-ad9b23b57ad2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j44sc" Oct 30 18:05:01.939726 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:01.939681 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e1a75a1-61d7-4285-bf95-ad9b23b57ad2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j44sc\" (UID: \"8e1a75a1-61d7-4285-bf95-ad9b23b57ad2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j44sc" Oct 30 18:05:01.939726 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:01.939711 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8e1a75a1-61d7-4285-bf95-ad9b23b57ad2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j44sc\" (UID: \"8e1a75a1-61d7-4285-bf95-ad9b23b57ad2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j44sc" Oct 30 18:05:01.939890 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:01.939750 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8xsr\" (UniqueName: \"kubernetes.io/projected/8e1a75a1-61d7-4285-bf95-ad9b23b57ad2-kube-api-access-j8xsr\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j44sc\" (UID: \"8e1a75a1-61d7-4285-bf95-ad9b23b57ad2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j44sc" Oct 30 18:05:01.942568 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:01.942543 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67cc46c64-66wvf"] Oct 30 18:05:01.946068 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:05:01.946040 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod024b2852_2e03_4f0b_a7dd_858dc9ed69ed.slice/crio-78cb4978d7685ffe8ca6d68c12dccd3bf98d1e9b8e8b63a54bb498439d3d8623 WatchSource:0}: Error finding container 78cb4978d7685ffe8ca6d68c12dccd3bf98d1e9b8e8b63a54bb498439d3d8623: Status 404 returned error can't find the container with id 78cb4978d7685ffe8ca6d68c12dccd3bf98d1e9b8e8b63a54bb498439d3d8623 Oct 30 18:05:02.041210 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:02.041167 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8e1a75a1-61d7-4285-bf95-ad9b23b57ad2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j44sc\" (UID: \"8e1a75a1-61d7-4285-bf95-ad9b23b57ad2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j44sc" Oct 30 18:05:02.041391 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:02.041261 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e1a75a1-61d7-4285-bf95-ad9b23b57ad2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j44sc\" (UID: \"8e1a75a1-61d7-4285-bf95-ad9b23b57ad2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j44sc" Oct 30 18:05:02.041391 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:02.041292 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8e1a75a1-61d7-4285-bf95-ad9b23b57ad2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j44sc\" (UID: \"8e1a75a1-61d7-4285-bf95-ad9b23b57ad2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j44sc" Oct 30 18:05:02.041391 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:02.041320 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8xsr\" (UniqueName: \"kubernetes.io/projected/8e1a75a1-61d7-4285-bf95-ad9b23b57ad2-kube-api-access-j8xsr\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j44sc\" (UID: \"8e1a75a1-61d7-4285-bf95-ad9b23b57ad2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j44sc" Oct 30 18:05:02.041391 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:02.041375 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8e1a75a1-61d7-4285-bf95-ad9b23b57ad2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j44sc\" (UID: \"8e1a75a1-61d7-4285-bf95-ad9b23b57ad2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j44sc" Oct 30 18:05:02.041583 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:02.041440 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e1a75a1-61d7-4285-bf95-ad9b23b57ad2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j44sc\" (UID: \"8e1a75a1-61d7-4285-bf95-ad9b23b57ad2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j44sc" Oct 30 18:05:02.041583 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:02.041467 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8e1a75a1-61d7-4285-bf95-ad9b23b57ad2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j44sc\" (UID: \"8e1a75a1-61d7-4285-bf95-ad9b23b57ad2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j44sc" Oct 30 18:05:02.042595 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:02.042564 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8e1a75a1-61d7-4285-bf95-ad9b23b57ad2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j44sc\" (UID: \"8e1a75a1-61d7-4285-bf95-ad9b23b57ad2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j44sc" Oct 30 18:05:02.044521 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:02.044475 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8e1a75a1-61d7-4285-bf95-ad9b23b57ad2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j44sc\" (UID: \"8e1a75a1-61d7-4285-bf95-ad9b23b57ad2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j44sc" Oct 30 18:05:02.045175 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:02.045117 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8e1a75a1-61d7-4285-bf95-ad9b23b57ad2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j44sc\" (UID: \"8e1a75a1-61d7-4285-bf95-ad9b23b57ad2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j44sc" Oct 30 18:05:02.045175 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:02.045172 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8e1a75a1-61d7-4285-bf95-ad9b23b57ad2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j44sc\" (UID: \"8e1a75a1-61d7-4285-bf95-ad9b23b57ad2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j44sc" Oct 30 18:05:02.046711 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:02.046677 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e1a75a1-61d7-4285-bf95-ad9b23b57ad2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j44sc\" (UID: \"8e1a75a1-61d7-4285-bf95-ad9b23b57ad2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j44sc" Oct 30 18:05:02.068809 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:02.068777 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8xsr\" (UniqueName: \"kubernetes.io/projected/8e1a75a1-61d7-4285-bf95-ad9b23b57ad2-kube-api-access-j8xsr\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j44sc\" (UID: \"8e1a75a1-61d7-4285-bf95-ad9b23b57ad2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j44sc" Oct 30 18:05:02.076437 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:02.076410 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e1a75a1-61d7-4285-bf95-ad9b23b57ad2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j44sc\" (UID: \"8e1a75a1-61d7-4285-bf95-ad9b23b57ad2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j44sc" Oct 30 18:05:02.162643 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:02.162521 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76f74769d4-kf4sm" event={"ID":"ab1e2145-cb53-4f9d-a76a-804b0545f59c","Type":"ContainerStarted","Data":"8562987190bb4a5da28c87321f5249051f79908ec842bf0d6e8d0bb038b4d9e7"} Oct 30 18:05:02.164868 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:02.164835 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67cc46c64-66wvf" event={"ID":"024b2852-2e03-4f0b-a7dd-858dc9ed69ed","Type":"ContainerStarted","Data":"78cb4978d7685ffe8ca6d68c12dccd3bf98d1e9b8e8b63a54bb498439d3d8623"} Oct 30 18:05:02.165555 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:02.165515 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j44sc" Oct 30 18:05:02.526481 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:02.526450 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j44sc"] Oct 30 18:05:02.532494 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:05:02.532464 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e1a75a1_61d7_4285_bf95_ad9b23b57ad2.slice/crio-d5fd2042dc651f015b292a2750afa2759b7213c1844e70f78c259d0477b48030 WatchSource:0}: Error finding container d5fd2042dc651f015b292a2750afa2759b7213c1844e70f78c259d0477b48030: Status 404 returned error can't find the container with id d5fd2042dc651f015b292a2750afa2759b7213c1844e70f78c259d0477b48030 Oct 30 18:05:03.173634 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:03.173546 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j44sc" event={"ID":"8e1a75a1-61d7-4285-bf95-ad9b23b57ad2","Type":"ContainerStarted","Data":"d5fd2042dc651f015b292a2750afa2759b7213c1844e70f78c259d0477b48030"} Oct 30 18:05:05.188014 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:05.187979 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76f74769d4-kf4sm" event={"ID":"ab1e2145-cb53-4f9d-a76a-804b0545f59c","Type":"ContainerStarted","Data":"49075b2e6cd43b728ade5603350a1cd49ae81762ae481618d5d067c3782ded68"} Oct 30 18:05:05.188405 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:05.188048 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-76f74769d4-kf4sm" Oct 30 18:05:05.238119 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:05.238067 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-76f74769d4-kf4sm" podStartSLOduration=1.9369792540000002 podStartE2EDuration="5.238053533s" totalImagesPullingTime="3.301074279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:05.236643776 +0000 UTC m=+402.326706114" watchObservedRunningTime="2026-10-30 18:05:05.238053533 +0000 UTC m=+402.328115872" Oct 30 18:05:05.412488 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:05.412457 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:05:05.412614 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:05.412534 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:05:06.197025 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:06.196985 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67cc46c64-66wvf" event={"ID":"024b2852-2e03-4f0b-a7dd-858dc9ed69ed","Type":"ContainerStarted","Data":"cd4569f68051b24db267d3934ac425d3ece913875a2e722cb1082a77b35ce067"} Oct 30 18:05:06.197519 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:06.197174 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-67cc46c64-66wvf" Oct 30 18:05:06.198760 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:06.198729 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j44sc" event={"ID":"8e1a75a1-61d7-4285-bf95-ad9b23b57ad2","Type":"ContainerStarted","Data":"4f796cb17f3a686cc07b5e8d2defb66aa0b097be655ae73ec826ca2050250a19"} Oct 30 18:05:06.198957 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:06.198943 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j44sc" Oct 30 18:05:06.201083 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:06.201057 4442 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-j44sc container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:05:06.201216 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:06.201108 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j44sc" podUID="8e1a75a1-61d7-4285-bf95-ad9b23b57ad2" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:05:06.239952 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:06.239903 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-67cc46c64-66wvf" podStartSLOduration=2.073612172 podStartE2EDuration="5.239886763s" totalImagesPullingTime="3.166274591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:06.238742324 +0000 UTC m=+403.328804654" watchObservedRunningTime="2026-10-30 18:05:06.239886763 +0000 UTC m=+403.329949324" Oct 30 18:05:06.284364 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:06.284296 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j44sc" podStartSLOduration=2.4145634129999998 podStartE2EDuration="5.284277317s" totalImagesPullingTime="2.869713904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:06.281457546 +0000 UTC m=+403.371519889" watchObservedRunningTime="2026-10-30 18:05:06.284277317 +0000 UTC m=+403.374339659" Oct 30 18:05:07.204079 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:07.204051 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j44sc" Oct 30 18:05:16.200890 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:16.200853 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-76f74769d4-kf4sm" Oct 30 18:05:17.205286 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:17.205259 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-67cc46c64-66wvf" Oct 30 18:05:17.293651 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:17.293616 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-76f74769d4-kf4sm"] Oct 30 18:05:17.293881 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:17.293837 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-76f74769d4-kf4sm" podUID="ab1e2145-cb53-4f9d-a76a-804b0545f59c" containerName="manager" containerID="cri-o://49075b2e6cd43b728ade5603350a1cd49ae81762ae481618d5d067c3782ded68" gracePeriod=30 Oct 30 18:05:17.559808 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:05:17.559723 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:05:18.041725 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:18.041701 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76f74769d4-kf4sm" Oct 30 18:05:18.202260 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:18.202168 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab1e2145-cb53-4f9d-a76a-804b0545f59c-kube-api-access-gqhgr\" (UniqueName: \"kubernetes.io/projected/ab1e2145-cb53-4f9d-a76a-804b0545f59c-kube-api-access-gqhgr\") pod \"ab1e2145-cb53-4f9d-a76a-804b0545f59c\" (UID: \"ab1e2145-cb53-4f9d-a76a-804b0545f59c\") " Oct 30 18:05:18.202260 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:18.202224 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab1e2145-cb53-4f9d-a76a-804b0545f59c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab1e2145-cb53-4f9d-a76a-804b0545f59c-metrics-certs\") pod \"ab1e2145-cb53-4f9d-a76a-804b0545f59c\" (UID: \"ab1e2145-cb53-4f9d-a76a-804b0545f59c\") " Oct 30 18:05:18.202260 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:18.202254 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab1e2145-cb53-4f9d-a76a-804b0545f59c-webhook-cert\" (UniqueName: \"kubernetes.io/secret/ab1e2145-cb53-4f9d-a76a-804b0545f59c-webhook-cert\") pod \"ab1e2145-cb53-4f9d-a76a-804b0545f59c\" (UID: \"ab1e2145-cb53-4f9d-a76a-804b0545f59c\") " Oct 30 18:05:18.204657 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:18.204624 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab1e2145-cb53-4f9d-a76a-804b0545f59c-metrics-certs" pod "ab1e2145-cb53-4f9d-a76a-804b0545f59c" (UID: "ab1e2145-cb53-4f9d-a76a-804b0545f59c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:05:18.204779 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:18.204701 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab1e2145-cb53-4f9d-a76a-804b0545f59c-webhook-cert" pod "ab1e2145-cb53-4f9d-a76a-804b0545f59c" (UID: "ab1e2145-cb53-4f9d-a76a-804b0545f59c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:05:18.204779 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:18.204730 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab1e2145-cb53-4f9d-a76a-804b0545f59c-kube-api-access-gqhgr" pod "ab1e2145-cb53-4f9d-a76a-804b0545f59c" (UID: "ab1e2145-cb53-4f9d-a76a-804b0545f59c"). InnerVolumeSpecName "kube-api-access-gqhgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:18.270186 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:18.270113 4442 generic.go:356] "Generic (PLEG): container finished" podID="ab1e2145-cb53-4f9d-a76a-804b0545f59c" containerID="49075b2e6cd43b728ade5603350a1cd49ae81762ae481618d5d067c3782ded68" exitCode=0 Oct 30 18:05:18.270626 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:18.270213 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76f74769d4-kf4sm" Oct 30 18:05:18.270626 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:18.270208 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76f74769d4-kf4sm" event={"ID":"ab1e2145-cb53-4f9d-a76a-804b0545f59c","Type":"ContainerDied","Data":"49075b2e6cd43b728ade5603350a1cd49ae81762ae481618d5d067c3782ded68"} Oct 30 18:05:18.270626 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:18.270305 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76f74769d4-kf4sm" event={"ID":"ab1e2145-cb53-4f9d-a76a-804b0545f59c","Type":"ContainerDied","Data":"8562987190bb4a5da28c87321f5249051f79908ec842bf0d6e8d0bb038b4d9e7"} Oct 30 18:05:18.270626 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:18.270334 4442 scope.go:122] "RemoveContainer" containerID="49075b2e6cd43b728ade5603350a1cd49ae81762ae481618d5d067c3782ded68" Oct 30 18:05:18.279718 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:18.279701 4442 scope.go:122] "RemoveContainer" containerID="49075b2e6cd43b728ade5603350a1cd49ae81762ae481618d5d067c3782ded68" Oct 30 18:05:18.280009 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:05:18.279988 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49075b2e6cd43b728ade5603350a1cd49ae81762ae481618d5d067c3782ded68\": container with ID starting with 49075b2e6cd43b728ade5603350a1cd49ae81762ae481618d5d067c3782ded68 not found: ID does not exist" containerID="49075b2e6cd43b728ade5603350a1cd49ae81762ae481618d5d067c3782ded68" Oct 30 18:05:18.280076 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:18.280019 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49075b2e6cd43b728ade5603350a1cd49ae81762ae481618d5d067c3782ded68"} err="failed to get container status \"49075b2e6cd43b728ade5603350a1cd49ae81762ae481618d5d067c3782ded68\": rpc error: code = NotFound desc = could not find container \"49075b2e6cd43b728ade5603350a1cd49ae81762ae481618d5d067c3782ded68\": container with ID starting with 49075b2e6cd43b728ade5603350a1cd49ae81762ae481618d5d067c3782ded68 not found: ID does not exist" Oct 30 18:05:18.303219 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:18.303192 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqhgr\" (UniqueName: \"kubernetes.io/projected/ab1e2145-cb53-4f9d-a76a-804b0545f59c-kube-api-access-gqhgr\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:05:18.303219 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:18.303216 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab1e2145-cb53-4f9d-a76a-804b0545f59c-metrics-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:05:18.303363 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:18.303226 4442 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ab1e2145-cb53-4f9d-a76a-804b0545f59c-webhook-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:05:18.318721 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:18.318680 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-76f74769d4-kf4sm"] Oct 30 18:05:18.322249 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:18.322227 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-76f74769d4-kf4sm"] Oct 30 18:05:19.062923 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:19.062892 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab1e2145-cb53-4f9d-a76a-804b0545f59c" path="/var/lib/kubelet/pods/ab1e2145-cb53-4f9d-a76a-804b0545f59c/volumes" Oct 30 18:05:47.223306 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:47.223265 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-x2rr2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:05:47.223785 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:47.223333 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" podUID="e03523d4-1b6f-4a80-9006-26b0345d809d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:05:47.561018 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:05:47.560937 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:05:48.417044 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:48.417014 4442 generic.go:356] "Generic (PLEG): container finished" podID="e03523d4-1b6f-4a80-9006-26b0345d809d" containerID="f32a9e2f167f134d719f6dc9b1f94211c22dbd5921034d7500cccb171b8b3321" exitCode=0 Oct 30 18:05:48.417495 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:48.417096 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" event={"ID":"e03523d4-1b6f-4a80-9006-26b0345d809d","Type":"ContainerDied","Data":"f32a9e2f167f134d719f6dc9b1f94211c22dbd5921034d7500cccb171b8b3321"} Oct 30 18:05:48.417536 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:48.417515 4442 scope.go:122] "RemoveContainer" containerID="f32a9e2f167f134d719f6dc9b1f94211c22dbd5921034d7500cccb171b8b3321" Oct 30 18:05:48.418648 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:48.418623 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:05:48.418713 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:48.418688 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:05:48.418749 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:48.418718 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:05:49.448175 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:49.445000 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" event={"ID":"e03523d4-1b6f-4a80-9006-26b0345d809d","Type":"ContainerStarted","Data":"535cf080e35833fb2fd74c327e7e095053c76ae3ecfb740a5517847d4f0e60f9"} Oct 30 18:05:50.448862 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:50.448489 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" Oct 30 18:05:50.448862 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:50.448640 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-x2rr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" start-of-body= Oct 30 18:05:50.448862 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:50.448690 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" podUID="e03523d4-1b6f-4a80-9006-26b0345d809d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" Oct 30 18:05:51.448551 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:51.448512 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-x2rr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" start-of-body= Oct 30 18:05:51.448761 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:51.448574 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" podUID="e03523d4-1b6f-4a80-9006-26b0345d809d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" Oct 30 18:05:52.441159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:52.440542 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-x2rr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" start-of-body= Oct 30 18:05:52.441159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:52.440623 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" podUID="e03523d4-1b6f-4a80-9006-26b0345d809d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" Oct 30 18:05:53.442157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:53.441546 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-x2rr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" start-of-body= Oct 30 18:05:53.442157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:53.441608 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" podUID="e03523d4-1b6f-4a80-9006-26b0345d809d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" Oct 30 18:05:54.439957 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:54.439926 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" Oct 30 18:05:54.440345 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:54.440320 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" Oct 30 18:05:54.440952 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:05:54.440933 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" Oct 30 18:06:17.559038 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:06:17.558999 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:06:47.559481 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:06:47.559438 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:07:17.559584 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:07:17.559544 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:07:22.962234 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:22.962165 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-dctgm"] Oct 30 18:07:22.962799 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:22.962721 4442 cpu_manager.go:414] "containerMap: removing container" podUID="ab1e2145-cb53-4f9d-a76a-804b0545f59c" containerName="manager" Oct 30 18:07:22.962799 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:22.962738 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab1e2145-cb53-4f9d-a76a-804b0545f59c" containerName="manager" Oct 30 18:07:22.962884 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:22.962807 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab1e2145-cb53-4f9d-a76a-804b0545f59c" containerName="manager" Oct 30 18:07:22.965985 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:22.965967 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dctgm" Oct 30 18:07:22.994478 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:22.994450 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 18:07:22.994658 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:22.994519 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-hdx68\"" Oct 30 18:07:22.994658 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:22.994650 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 18:07:22.996819 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:22.996799 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-dctgm"] Oct 30 18:07:23.031680 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:23.031650 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac100f87-aa55-49c3-8760-85523d3927db-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dctgm\" (UID: \"ac100f87-aa55-49c3-8760-85523d3927db\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dctgm" Oct 30 18:07:23.031680 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:23.031685 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac100f87-aa55-49c3-8760-85523d3927db-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dctgm\" (UID: \"ac100f87-aa55-49c3-8760-85523d3927db\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dctgm" Oct 30 18:07:23.032019 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:23.031713 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnvv4\" (UniqueName: \"kubernetes.io/projected/ac100f87-aa55-49c3-8760-85523d3927db-kube-api-access-dnvv4\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dctgm\" (UID: \"ac100f87-aa55-49c3-8760-85523d3927db\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dctgm" Oct 30 18:07:23.032019 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:23.031794 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac100f87-aa55-49c3-8760-85523d3927db-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dctgm\" (UID: \"ac100f87-aa55-49c3-8760-85523d3927db\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dctgm" Oct 30 18:07:23.032019 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:23.031920 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac100f87-aa55-49c3-8760-85523d3927db-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dctgm\" (UID: \"ac100f87-aa55-49c3-8760-85523d3927db\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dctgm" Oct 30 18:07:23.032019 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:23.031970 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac100f87-aa55-49c3-8760-85523d3927db-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dctgm\" (UID: \"ac100f87-aa55-49c3-8760-85523d3927db\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dctgm" Oct 30 18:07:23.082847 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:23.082821 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 30 18:07:23.132464 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:23.132431 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac100f87-aa55-49c3-8760-85523d3927db-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dctgm\" (UID: \"ac100f87-aa55-49c3-8760-85523d3927db\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dctgm" Oct 30 18:07:23.132464 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:23.132466 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac100f87-aa55-49c3-8760-85523d3927db-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dctgm\" (UID: \"ac100f87-aa55-49c3-8760-85523d3927db\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dctgm" Oct 30 18:07:23.132739 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:23.132492 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnvv4\" (UniqueName: \"kubernetes.io/projected/ac100f87-aa55-49c3-8760-85523d3927db-kube-api-access-dnvv4\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dctgm\" (UID: \"ac100f87-aa55-49c3-8760-85523d3927db\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dctgm" Oct 30 18:07:23.132739 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:23.132514 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac100f87-aa55-49c3-8760-85523d3927db-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dctgm\" (UID: \"ac100f87-aa55-49c3-8760-85523d3927db\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dctgm" Oct 30 18:07:23.132739 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:23.132535 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac100f87-aa55-49c3-8760-85523d3927db-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dctgm\" (UID: \"ac100f87-aa55-49c3-8760-85523d3927db\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dctgm" Oct 30 18:07:23.132739 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:23.132572 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac100f87-aa55-49c3-8760-85523d3927db-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dctgm\" (UID: \"ac100f87-aa55-49c3-8760-85523d3927db\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dctgm" Oct 30 18:07:23.132943 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:23.132916 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac100f87-aa55-49c3-8760-85523d3927db-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dctgm\" (UID: \"ac100f87-aa55-49c3-8760-85523d3927db\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dctgm" Oct 30 18:07:23.132989 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:23.132918 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac100f87-aa55-49c3-8760-85523d3927db-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dctgm\" (UID: \"ac100f87-aa55-49c3-8760-85523d3927db\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dctgm" Oct 30 18:07:23.133049 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:23.133027 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac100f87-aa55-49c3-8760-85523d3927db-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dctgm\" (UID: \"ac100f87-aa55-49c3-8760-85523d3927db\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dctgm" Oct 30 18:07:23.135013 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:23.134987 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac100f87-aa55-49c3-8760-85523d3927db-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dctgm\" (UID: \"ac100f87-aa55-49c3-8760-85523d3927db\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dctgm" Oct 30 18:07:23.135161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:23.135122 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac100f87-aa55-49c3-8760-85523d3927db-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dctgm\" (UID: \"ac100f87-aa55-49c3-8760-85523d3927db\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dctgm" Oct 30 18:07:23.162880 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:23.162853 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 18:07:23.183045 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:23.183023 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 18:07:23.192711 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:23.192680 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnvv4\" (UniqueName: \"kubernetes.io/projected/ac100f87-aa55-49c3-8760-85523d3927db-kube-api-access-dnvv4\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dctgm\" (UID: \"ac100f87-aa55-49c3-8760-85523d3927db\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dctgm" Oct 30 18:07:23.302650 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:23.302569 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-hdx68\"" Oct 30 18:07:23.309720 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:23.309698 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dctgm" Oct 30 18:07:23.608929 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:23.608903 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-dctgm"] Oct 30 18:07:23.614829 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:07:23.614799 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac100f87_aa55_49c3_8760_85523d3927db.slice/crio-0bda422b988430263205959912a88e6aa92895185be12337c510d2bf1a28a716 WatchSource:0}: Error finding container 0bda422b988430263205959912a88e6aa92895185be12337c510d2bf1a28a716: Status 404 returned error can't find the container with id 0bda422b988430263205959912a88e6aa92895185be12337c510d2bf1a28a716 Oct 30 18:07:23.911517 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:23.911473 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dctgm" event={"ID":"ac100f87-aa55-49c3-8760-85523d3927db","Type":"ContainerStarted","Data":"0bda422b988430263205959912a88e6aa92895185be12337c510d2bf1a28a716"} Oct 30 18:07:24.808946 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:24.808917 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-7s88v"] Oct 30 18:07:24.814355 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:24.814332 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7s88v" Oct 30 18:07:24.834020 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:24.833943 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-7s88v"] Oct 30 18:07:24.842656 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:24.842615 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 30 18:07:24.918268 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:24.918237 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dctgm" event={"ID":"ac100f87-aa55-49c3-8760-85523d3927db","Type":"ContainerStarted","Data":"cf2abbdcff8a97d42c6c1e6afc5eb8eecee42efcd126ce7016b222f090b6a867"} Oct 30 18:07:24.949513 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:24.949476 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/764a8232-9c6d-48aa-b055-8a15bb2e352d-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7s88v\" (UID: \"764a8232-9c6d-48aa-b055-8a15bb2e352d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7s88v" Oct 30 18:07:24.949700 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:24.949517 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/764a8232-9c6d-48aa-b055-8a15bb2e352d-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7s88v\" (UID: \"764a8232-9c6d-48aa-b055-8a15bb2e352d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7s88v" Oct 30 18:07:24.949700 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:24.949539 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vftw\" (UniqueName: \"kubernetes.io/projected/764a8232-9c6d-48aa-b055-8a15bb2e352d-kube-api-access-2vftw\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7s88v\" (UID: \"764a8232-9c6d-48aa-b055-8a15bb2e352d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7s88v" Oct 30 18:07:24.949827 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:24.949760 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/764a8232-9c6d-48aa-b055-8a15bb2e352d-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7s88v\" (UID: \"764a8232-9c6d-48aa-b055-8a15bb2e352d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7s88v" Oct 30 18:07:24.949827 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:24.949811 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/764a8232-9c6d-48aa-b055-8a15bb2e352d-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7s88v\" (UID: \"764a8232-9c6d-48aa-b055-8a15bb2e352d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7s88v" Oct 30 18:07:24.949921 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:24.949901 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/764a8232-9c6d-48aa-b055-8a15bb2e352d-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7s88v\" (UID: \"764a8232-9c6d-48aa-b055-8a15bb2e352d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7s88v" Oct 30 18:07:24.958027 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:24.957979 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dctgm" podStartSLOduration=1.7999474800000002 podStartE2EDuration="2.957965957s" totalImagesPullingTime="1.158018477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:07:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:07:24.956863634 +0000 UTC m=+542.046925986" watchObservedRunningTime="2026-10-30 18:07:24.957965957 +0000 UTC m=+542.048028296" Oct 30 18:07:25.050697 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:25.050617 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/764a8232-9c6d-48aa-b055-8a15bb2e352d-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7s88v\" (UID: \"764a8232-9c6d-48aa-b055-8a15bb2e352d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7s88v" Oct 30 18:07:25.050697 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:25.050657 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/764a8232-9c6d-48aa-b055-8a15bb2e352d-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7s88v\" (UID: \"764a8232-9c6d-48aa-b055-8a15bb2e352d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7s88v" Oct 30 18:07:25.050697 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:25.050687 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/764a8232-9c6d-48aa-b055-8a15bb2e352d-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7s88v\" (UID: \"764a8232-9c6d-48aa-b055-8a15bb2e352d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7s88v" Oct 30 18:07:25.050957 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:25.050710 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/764a8232-9c6d-48aa-b055-8a15bb2e352d-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7s88v\" (UID: \"764a8232-9c6d-48aa-b055-8a15bb2e352d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7s88v" Oct 30 18:07:25.050957 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:25.050740 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/764a8232-9c6d-48aa-b055-8a15bb2e352d-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7s88v\" (UID: \"764a8232-9c6d-48aa-b055-8a15bb2e352d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7s88v" Oct 30 18:07:25.050957 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:25.050763 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vftw\" (UniqueName: \"kubernetes.io/projected/764a8232-9c6d-48aa-b055-8a15bb2e352d-kube-api-access-2vftw\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7s88v\" (UID: \"764a8232-9c6d-48aa-b055-8a15bb2e352d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7s88v" Oct 30 18:07:25.051113 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:25.051076 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/764a8232-9c6d-48aa-b055-8a15bb2e352d-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7s88v\" (UID: \"764a8232-9c6d-48aa-b055-8a15bb2e352d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7s88v" Oct 30 18:07:25.051220 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:25.051178 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/764a8232-9c6d-48aa-b055-8a15bb2e352d-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7s88v\" (UID: \"764a8232-9c6d-48aa-b055-8a15bb2e352d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7s88v" Oct 30 18:07:25.051220 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:25.051201 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/764a8232-9c6d-48aa-b055-8a15bb2e352d-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7s88v\" (UID: \"764a8232-9c6d-48aa-b055-8a15bb2e352d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7s88v" Oct 30 18:07:25.053172 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:25.053149 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/764a8232-9c6d-48aa-b055-8a15bb2e352d-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7s88v\" (UID: \"764a8232-9c6d-48aa-b055-8a15bb2e352d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7s88v" Oct 30 18:07:25.053637 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:25.053613 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/764a8232-9c6d-48aa-b055-8a15bb2e352d-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7s88v\" (UID: \"764a8232-9c6d-48aa-b055-8a15bb2e352d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7s88v" Oct 30 18:07:25.081306 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:25.081275 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vftw\" (UniqueName: \"kubernetes.io/projected/764a8232-9c6d-48aa-b055-8a15bb2e352d-kube-api-access-2vftw\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7s88v\" (UID: \"764a8232-9c6d-48aa-b055-8a15bb2e352d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7s88v" Oct 30 18:07:25.146697 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:25.146666 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7s88v" Oct 30 18:07:25.310098 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:25.310024 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dctgm" Oct 30 18:07:25.441479 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:25.441458 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-7s88v"] Oct 30 18:07:25.444664 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:07:25.444632 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod764a8232_9c6d_48aa_b055_8a15bb2e352d.slice/crio-6390d59bec1220c5b1322271723c700828d8457d5a6ea5a74ffedccf53e6fbf1 WatchSource:0}: Error finding container 6390d59bec1220c5b1322271723c700828d8457d5a6ea5a74ffedccf53e6fbf1: Status 404 returned error can't find the container with id 6390d59bec1220c5b1322271723c700828d8457d5a6ea5a74ffedccf53e6fbf1 Oct 30 18:07:25.925798 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:25.925704 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7s88v" event={"ID":"764a8232-9c6d-48aa-b055-8a15bb2e352d","Type":"ContainerStarted","Data":"b6df7a0b758f7828242b4345eacfae23016a0e40f542540be16f6db063051e75"} Oct 30 18:07:25.925798 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:25.925774 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7s88v" event={"ID":"764a8232-9c6d-48aa-b055-8a15bb2e352d","Type":"ContainerStarted","Data":"6390d59bec1220c5b1322271723c700828d8457d5a6ea5a74ffedccf53e6fbf1"} Oct 30 18:07:25.967994 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:25.967940 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7s88v" podStartSLOduration=1.749181612 podStartE2EDuration="1.967924094s" totalImagesPullingTime="218.742482ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:07:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:07:25.966146295 +0000 UTC m=+543.056208626" watchObservedRunningTime="2026-10-30 18:07:25.967924094 +0000 UTC m=+543.057986433" Oct 30 18:07:26.147775 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:26.147743 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7s88v" Oct 30 18:07:30.292442 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:30.292411 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-qq7fh"] Oct 30 18:07:30.297617 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:30.297590 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-qq7fh" Oct 30 18:07:30.316080 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:30.316055 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-qq7fh"] Oct 30 18:07:30.324723 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:30.324703 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Oct 30 18:07:30.399755 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:30.399708 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31abe66d-4cdc-4258-afbc-39a2d5d05cbf-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-qq7fh\" (UID: \"31abe66d-4cdc-4258-afbc-39a2d5d05cbf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qq7fh" Oct 30 18:07:30.399910 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:30.399778 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31abe66d-4cdc-4258-afbc-39a2d5d05cbf-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-qq7fh\" (UID: \"31abe66d-4cdc-4258-afbc-39a2d5d05cbf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qq7fh" Oct 30 18:07:30.399910 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:30.399836 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31abe66d-4cdc-4258-afbc-39a2d5d05cbf-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-qq7fh\" (UID: \"31abe66d-4cdc-4258-afbc-39a2d5d05cbf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qq7fh" Oct 30 18:07:30.400017 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:30.399922 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31abe66d-4cdc-4258-afbc-39a2d5d05cbf-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-qq7fh\" (UID: \"31abe66d-4cdc-4258-afbc-39a2d5d05cbf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qq7fh" Oct 30 18:07:30.400017 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:30.399983 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chh55\" (UniqueName: \"kubernetes.io/projected/31abe66d-4cdc-4258-afbc-39a2d5d05cbf-kube-api-access-chh55\") pod \"echo-simulator-kserve-64b6f8f96c-qq7fh\" (UID: \"31abe66d-4cdc-4258-afbc-39a2d5d05cbf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qq7fh" Oct 30 18:07:30.400085 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:30.400037 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31abe66d-4cdc-4258-afbc-39a2d5d05cbf-home\") pod \"echo-simulator-kserve-64b6f8f96c-qq7fh\" (UID: \"31abe66d-4cdc-4258-afbc-39a2d5d05cbf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qq7fh" Oct 30 18:07:30.500530 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:30.500493 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31abe66d-4cdc-4258-afbc-39a2d5d05cbf-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-qq7fh\" (UID: \"31abe66d-4cdc-4258-afbc-39a2d5d05cbf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qq7fh" Oct 30 18:07:30.500764 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:30.500546 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31abe66d-4cdc-4258-afbc-39a2d5d05cbf-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-qq7fh\" (UID: \"31abe66d-4cdc-4258-afbc-39a2d5d05cbf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qq7fh" Oct 30 18:07:30.500764 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:30.500562 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31abe66d-4cdc-4258-afbc-39a2d5d05cbf-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-qq7fh\" (UID: \"31abe66d-4cdc-4258-afbc-39a2d5d05cbf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qq7fh" Oct 30 18:07:30.500764 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:30.500635 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31abe66d-4cdc-4258-afbc-39a2d5d05cbf-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-qq7fh\" (UID: \"31abe66d-4cdc-4258-afbc-39a2d5d05cbf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qq7fh" Oct 30 18:07:30.500764 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:30.500672 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chh55\" (UniqueName: \"kubernetes.io/projected/31abe66d-4cdc-4258-afbc-39a2d5d05cbf-kube-api-access-chh55\") pod \"echo-simulator-kserve-64b6f8f96c-qq7fh\" (UID: \"31abe66d-4cdc-4258-afbc-39a2d5d05cbf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qq7fh" Oct 30 18:07:30.501097 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:30.501016 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31abe66d-4cdc-4258-afbc-39a2d5d05cbf-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-qq7fh\" (UID: \"31abe66d-4cdc-4258-afbc-39a2d5d05cbf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qq7fh" Oct 30 18:07:30.501201 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:30.501184 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31abe66d-4cdc-4258-afbc-39a2d5d05cbf-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-qq7fh\" (UID: \"31abe66d-4cdc-4258-afbc-39a2d5d05cbf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qq7fh" Oct 30 18:07:30.501298 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:30.501276 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31abe66d-4cdc-4258-afbc-39a2d5d05cbf-home\") pod \"echo-simulator-kserve-64b6f8f96c-qq7fh\" (UID: \"31abe66d-4cdc-4258-afbc-39a2d5d05cbf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qq7fh" Oct 30 18:07:30.501602 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:30.501583 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31abe66d-4cdc-4258-afbc-39a2d5d05cbf-home\") pod \"echo-simulator-kserve-64b6f8f96c-qq7fh\" (UID: \"31abe66d-4cdc-4258-afbc-39a2d5d05cbf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qq7fh" Oct 30 18:07:30.503274 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:30.503246 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31abe66d-4cdc-4258-afbc-39a2d5d05cbf-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-qq7fh\" (UID: \"31abe66d-4cdc-4258-afbc-39a2d5d05cbf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qq7fh" Oct 30 18:07:30.503712 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:30.503674 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31abe66d-4cdc-4258-afbc-39a2d5d05cbf-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-qq7fh\" (UID: \"31abe66d-4cdc-4258-afbc-39a2d5d05cbf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qq7fh" Oct 30 18:07:30.520114 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:30.520090 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chh55\" (UniqueName: \"kubernetes.io/projected/31abe66d-4cdc-4258-afbc-39a2d5d05cbf-kube-api-access-chh55\") pod \"echo-simulator-kserve-64b6f8f96c-qq7fh\" (UID: \"31abe66d-4cdc-4258-afbc-39a2d5d05cbf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qq7fh" Oct 30 18:07:30.611377 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:30.611041 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-qq7fh" Oct 30 18:07:30.918758 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:30.918556 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-qq7fh"] Oct 30 18:07:30.921565 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:07:30.921530 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31abe66d_4cdc_4258_afbc_39a2d5d05cbf.slice/crio-52dbdbe05e8aa08503730c1c0a91973b1a9625df50d4286a72b44a5e5219c2f9 WatchSource:0}: Error finding container 52dbdbe05e8aa08503730c1c0a91973b1a9625df50d4286a72b44a5e5219c2f9: Status 404 returned error can't find the container with id 52dbdbe05e8aa08503730c1c0a91973b1a9625df50d4286a72b44a5e5219c2f9 Oct 30 18:07:30.957083 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:30.957051 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-qq7fh" event={"ID":"31abe66d-4cdc-4258-afbc-39a2d5d05cbf","Type":"ContainerStarted","Data":"52dbdbe05e8aa08503730c1c0a91973b1a9625df50d4286a72b44a5e5219c2f9"} Oct 30 18:07:32.970600 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:32.970562 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-qq7fh" event={"ID":"31abe66d-4cdc-4258-afbc-39a2d5d05cbf","Type":"ContainerStarted","Data":"798680e0b9aaaa37556038030864f59372bd393578de83ab3d89fb095eb1dcf0"} Oct 30 18:07:33.019917 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:33.019871 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-qq7fh" podStartSLOduration=1.331421671 podStartE2EDuration="3.019857081s" totalImagesPullingTime="1.68843541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:07:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:07:33.018362091 +0000 UTC m=+550.108424430" watchObservedRunningTime="2026-10-30 18:07:33.019857081 +0000 UTC m=+550.109919420" Oct 30 18:07:33.310845 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:33.310747 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dctgm" Oct 30 18:07:33.324327 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:33.324293 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dctgm" Oct 30 18:07:33.611887 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:33.611813 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-qq7fh" Oct 30 18:07:33.989598 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:33.989551 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dctgm" Oct 30 18:07:35.147005 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:35.146968 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7s88v" Oct 30 18:07:35.159301 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:35.159280 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7s88v" Oct 30 18:07:36.000721 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:36.000696 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7s88v" Oct 30 18:07:40.612210 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:40.612171 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-qq7fh" Oct 30 18:07:40.625310 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:40.625285 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-qq7fh" Oct 30 18:07:41.027363 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:41.027332 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-qq7fh" Oct 30 18:07:47.557993 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:07:47.557959 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:07:49.975302 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:49.975264 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c"] Oct 30 18:07:49.979603 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:49.979579 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" Oct 30 18:07:50.007652 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:50.007619 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c"] Oct 30 18:07:50.091742 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:50.091695 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dab58a66-301a-4e74-9375-95bd4964b644-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-vlm9c\" (UID: \"dab58a66-301a-4e74-9375-95bd4964b644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" Oct 30 18:07:50.091930 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:50.091749 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab58a66-301a-4e74-9375-95bd4964b644-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-vlm9c\" (UID: \"dab58a66-301a-4e74-9375-95bd4964b644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" Oct 30 18:07:50.091930 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:50.091876 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dab58a66-301a-4e74-9375-95bd4964b644-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-vlm9c\" (UID: \"dab58a66-301a-4e74-9375-95bd4964b644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" Oct 30 18:07:50.091930 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:50.091912 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab58a66-301a-4e74-9375-95bd4964b644-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-vlm9c\" (UID: \"dab58a66-301a-4e74-9375-95bd4964b644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" Oct 30 18:07:50.092050 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:50.091940 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dab58a66-301a-4e74-9375-95bd4964b644-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-vlm9c\" (UID: \"dab58a66-301a-4e74-9375-95bd4964b644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" Oct 30 18:07:50.092050 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:50.091971 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hgnf\" (UniqueName: \"kubernetes.io/projected/dab58a66-301a-4e74-9375-95bd4964b644-kube-api-access-8hgnf\") pod \"maas-default-gateway-openshift-default-76597d6855-vlm9c\" (UID: \"dab58a66-301a-4e74-9375-95bd4964b644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" Oct 30 18:07:50.092113 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:50.092065 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dab58a66-301a-4e74-9375-95bd4964b644-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-vlm9c\" (UID: \"dab58a66-301a-4e74-9375-95bd4964b644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" Oct 30 18:07:50.092113 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:50.092082 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dab58a66-301a-4e74-9375-95bd4964b644-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-vlm9c\" (UID: \"dab58a66-301a-4e74-9375-95bd4964b644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" Oct 30 18:07:50.092252 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:50.092117 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dab58a66-301a-4e74-9375-95bd4964b644-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-vlm9c\" (UID: \"dab58a66-301a-4e74-9375-95bd4964b644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" Oct 30 18:07:50.193037 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:50.193001 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dab58a66-301a-4e74-9375-95bd4964b644-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-vlm9c\" (UID: \"dab58a66-301a-4e74-9375-95bd4964b644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" Oct 30 18:07:50.193037 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:50.193041 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab58a66-301a-4e74-9375-95bd4964b644-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-vlm9c\" (UID: \"dab58a66-301a-4e74-9375-95bd4964b644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" Oct 30 18:07:50.193337 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:50.193087 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dab58a66-301a-4e74-9375-95bd4964b644-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-vlm9c\" (UID: \"dab58a66-301a-4e74-9375-95bd4964b644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" Oct 30 18:07:50.193337 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:50.193111 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab58a66-301a-4e74-9375-95bd4964b644-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-vlm9c\" (UID: \"dab58a66-301a-4e74-9375-95bd4964b644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" Oct 30 18:07:50.193337 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:50.193146 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dab58a66-301a-4e74-9375-95bd4964b644-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-vlm9c\" (UID: \"dab58a66-301a-4e74-9375-95bd4964b644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" Oct 30 18:07:50.193337 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:50.193171 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hgnf\" (UniqueName: \"kubernetes.io/projected/dab58a66-301a-4e74-9375-95bd4964b644-kube-api-access-8hgnf\") pod \"maas-default-gateway-openshift-default-76597d6855-vlm9c\" (UID: \"dab58a66-301a-4e74-9375-95bd4964b644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" Oct 30 18:07:50.193337 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:50.193232 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dab58a66-301a-4e74-9375-95bd4964b644-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-vlm9c\" (UID: \"dab58a66-301a-4e74-9375-95bd4964b644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" Oct 30 18:07:50.193337 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:50.193254 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dab58a66-301a-4e74-9375-95bd4964b644-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-vlm9c\" (UID: \"dab58a66-301a-4e74-9375-95bd4964b644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" Oct 30 18:07:50.193337 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:50.193303 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dab58a66-301a-4e74-9375-95bd4964b644-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-vlm9c\" (UID: \"dab58a66-301a-4e74-9375-95bd4964b644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" Oct 30 18:07:50.193739 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:50.193530 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dab58a66-301a-4e74-9375-95bd4964b644-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-vlm9c\" (UID: \"dab58a66-301a-4e74-9375-95bd4964b644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" Oct 30 18:07:50.193739 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:50.193596 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dab58a66-301a-4e74-9375-95bd4964b644-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-vlm9c\" (UID: \"dab58a66-301a-4e74-9375-95bd4964b644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" Oct 30 18:07:50.193739 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:50.193624 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab58a66-301a-4e74-9375-95bd4964b644-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-vlm9c\" (UID: \"dab58a66-301a-4e74-9375-95bd4964b644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" Oct 30 18:07:50.193916 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:50.193899 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab58a66-301a-4e74-9375-95bd4964b644-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-vlm9c\" (UID: \"dab58a66-301a-4e74-9375-95bd4964b644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" Oct 30 18:07:50.193981 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:50.193928 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dab58a66-301a-4e74-9375-95bd4964b644-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-vlm9c\" (UID: \"dab58a66-301a-4e74-9375-95bd4964b644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" Oct 30 18:07:50.195858 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:50.195835 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dab58a66-301a-4e74-9375-95bd4964b644-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-vlm9c\" (UID: \"dab58a66-301a-4e74-9375-95bd4964b644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" Oct 30 18:07:50.196081 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:50.196062 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dab58a66-301a-4e74-9375-95bd4964b644-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-vlm9c\" (UID: \"dab58a66-301a-4e74-9375-95bd4964b644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" Oct 30 18:07:50.215386 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:50.215358 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dab58a66-301a-4e74-9375-95bd4964b644-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-vlm9c\" (UID: \"dab58a66-301a-4e74-9375-95bd4964b644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" Oct 30 18:07:50.216219 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:50.216191 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hgnf\" (UniqueName: \"kubernetes.io/projected/dab58a66-301a-4e74-9375-95bd4964b644-kube-api-access-8hgnf\") pod \"maas-default-gateway-openshift-default-76597d6855-vlm9c\" (UID: \"dab58a66-301a-4e74-9375-95bd4964b644\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" Oct 30 18:07:50.307292 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:50.307199 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" Oct 30 18:07:50.626573 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:50.626543 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c"] Oct 30 18:07:50.630817 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:07:50.630784 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddab58a66_301a_4e74_9375_95bd4964b644.slice/crio-353a89117e65871c736bf6ddddea1fca29ea6e06a00020b571695a6846d5e300 WatchSource:0}: Error finding container 353a89117e65871c736bf6ddddea1fca29ea6e06a00020b571695a6846d5e300: Status 404 returned error can't find the container with id 353a89117e65871c736bf6ddddea1fca29ea6e06a00020b571695a6846d5e300 Oct 30 18:07:50.633083 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:50.633048 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:07:50.633192 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:50.633117 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:07:50.633192 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:50.633174 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:07:51.076242 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:51.075038 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" event={"ID":"dab58a66-301a-4e74-9375-95bd4964b644","Type":"ContainerStarted","Data":"9ed8fd43e99ee5f67663526a917e7077855e8895ca1e022354be284f255da881"} Oct 30 18:07:51.076242 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:51.075119 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" event={"ID":"dab58a66-301a-4e74-9375-95bd4964b644","Type":"ContainerStarted","Data":"353a89117e65871c736bf6ddddea1fca29ea6e06a00020b571695a6846d5e300"} Oct 30 18:07:51.119118 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:51.119066 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" podStartSLOduration=2.119052152 podStartE2EDuration="2.119052152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:07:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:07:51.118215795 +0000 UTC m=+568.208278136" watchObservedRunningTime="2026-10-30 18:07:51.119052152 +0000 UTC m=+568.209114491" Oct 30 18:07:51.308360 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:51.308319 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" Oct 30 18:07:51.310009 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:51.309985 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-vlm9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 18:07:51.310127 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:51.310057 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" podUID="dab58a66-301a-4e74-9375-95bd4964b644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 18:07:52.308349 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:52.308310 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-vlm9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 18:07:52.308731 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:52.308379 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" podUID="dab58a66-301a-4e74-9375-95bd4964b644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 18:07:53.307963 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:53.307933 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-vlm9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 18:07:53.308159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:53.307994 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" podUID="dab58a66-301a-4e74-9375-95bd4964b644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 18:07:54.308311 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:54.308273 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-vlm9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 18:07:54.308718 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:54.308330 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" podUID="dab58a66-301a-4e74-9375-95bd4964b644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 18:07:55.307704 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:55.307668 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-vlm9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 18:07:55.307885 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:55.307737 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" podUID="dab58a66-301a-4e74-9375-95bd4964b644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 18:07:56.307771 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:56.307735 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-vlm9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 18:07:56.308178 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:56.307788 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" podUID="dab58a66-301a-4e74-9375-95bd4964b644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 18:07:57.307638 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:57.307600 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-vlm9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 18:07:57.307872 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:57.307662 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" podUID="dab58a66-301a-4e74-9375-95bd4964b644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 18:07:58.850651 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:58.850609 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" Oct 30 18:07:58.851177 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:58.851039 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" Oct 30 18:07:58.852187 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:07:58.852169 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" Oct 30 18:08:05.035226 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:05.035147 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q"] Oct 30 18:08:05.042363 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:05.042336 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" Oct 30 18:08:05.076338 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:05.076300 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q"] Oct 30 18:08:05.152262 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:05.152226 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxs75\" (UniqueName: \"kubernetes.io/projected/50067d7b-521e-403e-a128-20223ce09044-kube-api-access-gxs75\") pod \"maas-default-gateway-openshift-default-76597d6855-f5f5q\" (UID: \"50067d7b-521e-403e-a128-20223ce09044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" Oct 30 18:08:05.152440 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:05.152283 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50067d7b-521e-403e-a128-20223ce09044-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-f5f5q\" (UID: \"50067d7b-521e-403e-a128-20223ce09044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" Oct 30 18:08:05.152440 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:05.152386 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50067d7b-521e-403e-a128-20223ce09044-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-f5f5q\" (UID: \"50067d7b-521e-403e-a128-20223ce09044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" Oct 30 18:08:05.152577 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:05.152466 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50067d7b-521e-403e-a128-20223ce09044-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-f5f5q\" (UID: \"50067d7b-521e-403e-a128-20223ce09044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" Oct 30 18:08:05.152577 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:05.152555 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50067d7b-521e-403e-a128-20223ce09044-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-f5f5q\" (UID: \"50067d7b-521e-403e-a128-20223ce09044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" Oct 30 18:08:05.152680 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:05.152603 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50067d7b-521e-403e-a128-20223ce09044-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-f5f5q\" (UID: \"50067d7b-521e-403e-a128-20223ce09044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" Oct 30 18:08:05.152680 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:05.152632 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50067d7b-521e-403e-a128-20223ce09044-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-f5f5q\" (UID: \"50067d7b-521e-403e-a128-20223ce09044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" Oct 30 18:08:05.152680 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:05.152669 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50067d7b-521e-403e-a128-20223ce09044-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-f5f5q\" (UID: \"50067d7b-521e-403e-a128-20223ce09044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" Oct 30 18:08:05.152833 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:05.152697 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50067d7b-521e-403e-a128-20223ce09044-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-f5f5q\" (UID: \"50067d7b-521e-403e-a128-20223ce09044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" Oct 30 18:08:05.253237 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:05.253204 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxs75\" (UniqueName: \"kubernetes.io/projected/50067d7b-521e-403e-a128-20223ce09044-kube-api-access-gxs75\") pod \"maas-default-gateway-openshift-default-76597d6855-f5f5q\" (UID: \"50067d7b-521e-403e-a128-20223ce09044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" Oct 30 18:08:05.253237 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:05.253244 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50067d7b-521e-403e-a128-20223ce09044-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-f5f5q\" (UID: \"50067d7b-521e-403e-a128-20223ce09044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" Oct 30 18:08:05.253477 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:05.253270 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50067d7b-521e-403e-a128-20223ce09044-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-f5f5q\" (UID: \"50067d7b-521e-403e-a128-20223ce09044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" Oct 30 18:08:05.253477 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:05.253319 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50067d7b-521e-403e-a128-20223ce09044-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-f5f5q\" (UID: \"50067d7b-521e-403e-a128-20223ce09044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" Oct 30 18:08:05.253477 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:05.253365 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50067d7b-521e-403e-a128-20223ce09044-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-f5f5q\" (UID: \"50067d7b-521e-403e-a128-20223ce09044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" Oct 30 18:08:05.253477 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:05.253402 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50067d7b-521e-403e-a128-20223ce09044-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-f5f5q\" (UID: \"50067d7b-521e-403e-a128-20223ce09044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" Oct 30 18:08:05.253693 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:05.253545 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50067d7b-521e-403e-a128-20223ce09044-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-f5f5q\" (UID: \"50067d7b-521e-403e-a128-20223ce09044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" Oct 30 18:08:05.253693 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:05.253590 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50067d7b-521e-403e-a128-20223ce09044-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-f5f5q\" (UID: \"50067d7b-521e-403e-a128-20223ce09044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" Oct 30 18:08:05.253693 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:05.253615 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50067d7b-521e-403e-a128-20223ce09044-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-f5f5q\" (UID: \"50067d7b-521e-403e-a128-20223ce09044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" Oct 30 18:08:05.254033 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:05.254012 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50067d7b-521e-403e-a128-20223ce09044-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-f5f5q\" (UID: \"50067d7b-521e-403e-a128-20223ce09044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" Oct 30 18:08:05.254254 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:05.254234 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50067d7b-521e-403e-a128-20223ce09044-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-f5f5q\" (UID: \"50067d7b-521e-403e-a128-20223ce09044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" Oct 30 18:08:05.254341 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:05.254290 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50067d7b-521e-403e-a128-20223ce09044-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-f5f5q\" (UID: \"50067d7b-521e-403e-a128-20223ce09044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" Oct 30 18:08:05.254420 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:05.254397 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50067d7b-521e-403e-a128-20223ce09044-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-f5f5q\" (UID: \"50067d7b-521e-403e-a128-20223ce09044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" Oct 30 18:08:05.254501 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:05.254479 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50067d7b-521e-403e-a128-20223ce09044-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-f5f5q\" (UID: \"50067d7b-521e-403e-a128-20223ce09044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" Oct 30 18:08:05.255781 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:05.255764 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50067d7b-521e-403e-a128-20223ce09044-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-f5f5q\" (UID: \"50067d7b-521e-403e-a128-20223ce09044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" Oct 30 18:08:05.256235 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:05.256218 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50067d7b-521e-403e-a128-20223ce09044-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-f5f5q\" (UID: \"50067d7b-521e-403e-a128-20223ce09044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" Oct 30 18:08:05.271234 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:05.271209 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50067d7b-521e-403e-a128-20223ce09044-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-f5f5q\" (UID: \"50067d7b-521e-403e-a128-20223ce09044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" Oct 30 18:08:05.271598 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:05.271579 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxs75\" (UniqueName: \"kubernetes.io/projected/50067d7b-521e-403e-a128-20223ce09044-kube-api-access-gxs75\") pod \"maas-default-gateway-openshift-default-76597d6855-f5f5q\" (UID: \"50067d7b-521e-403e-a128-20223ce09044\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" Oct 30 18:08:05.357228 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:05.357128 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" Oct 30 18:08:05.667694 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:05.667671 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q"] Oct 30 18:08:05.669539 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:08:05.669514 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50067d7b_521e_403e_a128_20223ce09044.slice/crio-738b393143f4f2758c9d9252bbe5ec07984d6daa33776bc1f6744af3f57b4b8a WatchSource:0}: Error finding container 738b393143f4f2758c9d9252bbe5ec07984d6daa33776bc1f6744af3f57b4b8a: Status 404 returned error can't find the container with id 738b393143f4f2758c9d9252bbe5ec07984d6daa33776bc1f6744af3f57b4b8a Oct 30 18:08:05.671911 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:05.671880 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:08:05.671992 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:05.671942 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:08:05.671992 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:05.671969 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:08:06.163804 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:06.163755 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" event={"ID":"50067d7b-521e-403e-a128-20223ce09044","Type":"ContainerStarted","Data":"634bbeed132f31a37362f2854acca358c5265126d0e02af5f853efa21ed39a90"} Oct 30 18:08:06.164301 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:06.163872 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" event={"ID":"50067d7b-521e-403e-a128-20223ce09044","Type":"ContainerStarted","Data":"738b393143f4f2758c9d9252bbe5ec07984d6daa33776bc1f6744af3f57b4b8a"} Oct 30 18:08:06.206993 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:06.206938 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" podStartSLOduration=1.20692469 podStartE2EDuration="1.20692469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:08:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:08:06.206093559 +0000 UTC m=+583.296155921" watchObservedRunningTime="2026-10-30 18:08:06.20692469 +0000 UTC m=+583.296987028" Oct 30 18:08:06.357663 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:06.357624 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" Oct 30 18:08:06.359301 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:06.359276 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-f5f5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 18:08:06.359418 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:06.359321 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" podUID="50067d7b-521e-403e-a128-20223ce09044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 18:08:07.358068 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:07.358034 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-f5f5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 18:08:07.358542 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:07.358088 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" podUID="50067d7b-521e-403e-a128-20223ce09044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 18:08:08.357648 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:08.357612 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-f5f5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 18:08:08.357810 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:08.357667 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" podUID="50067d7b-521e-403e-a128-20223ce09044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 18:08:09.357494 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:09.357459 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-f5f5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 18:08:09.357897 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:09.357518 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" podUID="50067d7b-521e-403e-a128-20223ce09044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 18:08:10.357676 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:10.357643 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-f5f5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 18:08:10.358099 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:10.357697 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" podUID="50067d7b-521e-403e-a128-20223ce09044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 18:08:11.358554 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:11.358516 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-f5f5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 18:08:11.358945 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:11.358585 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" podUID="50067d7b-521e-403e-a128-20223ce09044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 18:08:12.358331 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:12.358296 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-f5f5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 18:08:12.358514 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:12.358345 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" podUID="50067d7b-521e-403e-a128-20223ce09044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 18:08:13.910824 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:13.910792 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" Oct 30 18:08:13.911335 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:13.911157 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" Oct 30 18:08:13.911995 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:13.911978 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" Oct 30 18:08:17.558881 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:08:17.558850 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:08:23.124386 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:23.124356 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52rn6_516c6b6c-fc40-49cf-93b5-1c4471b02cf0/ovn-acl-logging/0.log" Oct 30 18:08:23.126266 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:08:23.126243 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52rn6_516c6b6c-fc40-49cf-93b5-1c4471b02cf0/ovn-acl-logging/0.log" Oct 30 18:08:47.559176 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:08:47.559127 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:09:15.458383 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:15.458345 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-57bf5177-b587ccc9-9q6rr"] Oct 30 18:09:15.461887 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:15.461870 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-57bf5177-b587ccc9-9q6rr" Oct 30 18:09:15.500257 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:15.500226 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-57bf5177-b587ccc9-9q6rr"] Oct 30 18:09:15.557231 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:15.557198 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-57bf5177-serving-cert\"" Oct 30 18:09:15.557231 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:15.557256 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 18:09:15.557470 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:15.557258 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wjb5t\"" Oct 30 18:09:15.635358 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:15.635320 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12e28d3d-06c0-4e7f-ade8-f8c062d97969-openshift-service-ca\") pod \"maas-api-e2e-migrate-57bf5177-b587ccc9-9q6rr\" (UID: \"12e28d3d-06c0-4e7f-ade8-f8c062d97969\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-57bf5177-b587ccc9-9q6rr" Oct 30 18:09:15.635567 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:15.635449 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qljg6\" (UniqueName: \"kubernetes.io/projected/12e28d3d-06c0-4e7f-ade8-f8c062d97969-kube-api-access-qljg6\") pod \"maas-api-e2e-migrate-57bf5177-b587ccc9-9q6rr\" (UID: \"12e28d3d-06c0-4e7f-ade8-f8c062d97969\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-57bf5177-b587ccc9-9q6rr" Oct 30 18:09:15.635567 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:15.635507 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12e28d3d-06c0-4e7f-ade8-f8c062d97969-metrics-certs\") pod \"maas-api-e2e-migrate-57bf5177-b587ccc9-9q6rr\" (UID: \"12e28d3d-06c0-4e7f-ade8-f8c062d97969\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-57bf5177-b587ccc9-9q6rr" Oct 30 18:09:15.635567 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:15.635549 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12e28d3d-06c0-4e7f-ade8-f8c062d97969-maas-api-tls\") pod \"maas-api-e2e-migrate-57bf5177-b587ccc9-9q6rr\" (UID: \"12e28d3d-06c0-4e7f-ade8-f8c062d97969\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-57bf5177-b587ccc9-9q6rr" Oct 30 18:09:15.736209 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:15.736112 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12e28d3d-06c0-4e7f-ade8-f8c062d97969-openshift-service-ca\") pod \"maas-api-e2e-migrate-57bf5177-b587ccc9-9q6rr\" (UID: \"12e28d3d-06c0-4e7f-ade8-f8c062d97969\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-57bf5177-b587ccc9-9q6rr" Oct 30 18:09:15.736380 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:15.736209 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qljg6\" (UniqueName: \"kubernetes.io/projected/12e28d3d-06c0-4e7f-ade8-f8c062d97969-kube-api-access-qljg6\") pod \"maas-api-e2e-migrate-57bf5177-b587ccc9-9q6rr\" (UID: \"12e28d3d-06c0-4e7f-ade8-f8c062d97969\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-57bf5177-b587ccc9-9q6rr" Oct 30 18:09:15.736380 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:15.736263 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12e28d3d-06c0-4e7f-ade8-f8c062d97969-metrics-certs\") pod \"maas-api-e2e-migrate-57bf5177-b587ccc9-9q6rr\" (UID: \"12e28d3d-06c0-4e7f-ade8-f8c062d97969\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-57bf5177-b587ccc9-9q6rr" Oct 30 18:09:15.736380 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:15.736320 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12e28d3d-06c0-4e7f-ade8-f8c062d97969-maas-api-tls\") pod \"maas-api-e2e-migrate-57bf5177-b587ccc9-9q6rr\" (UID: \"12e28d3d-06c0-4e7f-ade8-f8c062d97969\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-57bf5177-b587ccc9-9q6rr" Oct 30 18:09:15.736871 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:15.736850 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12e28d3d-06c0-4e7f-ade8-f8c062d97969-openshift-service-ca\") pod \"maas-api-e2e-migrate-57bf5177-b587ccc9-9q6rr\" (UID: \"12e28d3d-06c0-4e7f-ade8-f8c062d97969\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-57bf5177-b587ccc9-9q6rr" Oct 30 18:09:15.739493 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:15.739470 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12e28d3d-06c0-4e7f-ade8-f8c062d97969-maas-api-tls\") pod \"maas-api-e2e-migrate-57bf5177-b587ccc9-9q6rr\" (UID: \"12e28d3d-06c0-4e7f-ade8-f8c062d97969\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-57bf5177-b587ccc9-9q6rr" Oct 30 18:09:15.739606 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:15.739532 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12e28d3d-06c0-4e7f-ade8-f8c062d97969-metrics-certs\") pod \"maas-api-e2e-migrate-57bf5177-b587ccc9-9q6rr\" (UID: \"12e28d3d-06c0-4e7f-ade8-f8c062d97969\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-57bf5177-b587ccc9-9q6rr" Oct 30 18:09:15.754795 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:15.754772 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qljg6\" (UniqueName: \"kubernetes.io/projected/12e28d3d-06c0-4e7f-ade8-f8c062d97969-kube-api-access-qljg6\") pod \"maas-api-e2e-migrate-57bf5177-b587ccc9-9q6rr\" (UID: \"12e28d3d-06c0-4e7f-ade8-f8c062d97969\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-57bf5177-b587ccc9-9q6rr" Oct 30 18:09:15.775655 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:15.775625 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-57bf5177-b587ccc9-9q6rr" Oct 30 18:09:15.953850 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:15.953817 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-57bf5177-5f6f57f4f5-xw6rx"] Oct 30 18:09:15.957557 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:15.957533 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-57bf5177-5f6f57f4f5-xw6rx" Oct 30 18:09:15.981884 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:15.981854 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-57bf5177-5f6f57f4f5-xw6rx"] Oct 30 18:09:15.985523 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:15.985497 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-57bf5177-dockercfg-p6zrl\"" Oct 30 18:09:15.985694 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:15.985524 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-57bf5177\"" Oct 30 18:09:16.067721 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:16.067636 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-57bf5177-7879c68c96-896r4"] Oct 30 18:09:16.073738 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:16.073707 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-57bf5177-7879c68c96-896r4" Oct 30 18:09:16.094713 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:16.094687 4442 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:09:16.098184 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:16.098162 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-57bf5177-7879c68c96-896r4"] Oct 30 18:09:16.101972 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:16.100402 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-57bf5177-b587ccc9-9q6rr"] Oct 30 18:09:16.143094 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:16.143058 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72pkf\" (UniqueName: \"kubernetes.io/projected/f1d7323e-9efa-4c36-b781-5bc157008ed4-kube-api-access-72pkf\") pod \"payload-pre-processing-e2e-migrate-57bf5177-5f6f57f4f5-xw6rx\" (UID: \"f1d7323e-9efa-4c36-b781-5bc157008ed4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-57bf5177-5f6f57f4f5-xw6rx" Oct 30 18:09:16.143094 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:16.143097 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1d7323e-9efa-4c36-b781-5bc157008ed4-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-57bf5177-5f6f57f4f5-xw6rx\" (UID: \"f1d7323e-9efa-4c36-b781-5bc157008ed4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-57bf5177-5f6f57f4f5-xw6rx" Oct 30 18:09:16.244601 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:16.244564 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72pkf\" (UniqueName: \"kubernetes.io/projected/f1d7323e-9efa-4c36-b781-5bc157008ed4-kube-api-access-72pkf\") pod \"payload-pre-processing-e2e-migrate-57bf5177-5f6f57f4f5-xw6rx\" (UID: \"f1d7323e-9efa-4c36-b781-5bc157008ed4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-57bf5177-5f6f57f4f5-xw6rx" Oct 30 18:09:16.244791 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:16.244612 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1d7323e-9efa-4c36-b781-5bc157008ed4-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-57bf5177-5f6f57f4f5-xw6rx\" (UID: \"f1d7323e-9efa-4c36-b781-5bc157008ed4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-57bf5177-5f6f57f4f5-xw6rx" Oct 30 18:09:16.244791 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:16.244692 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr6lp\" (UniqueName: \"kubernetes.io/projected/db9706fd-5979-4cea-aae8-cffd27090481-kube-api-access-nr6lp\") pod \"payload-processing-e2e-migrate-57bf5177-7879c68c96-896r4\" (UID: \"db9706fd-5979-4cea-aae8-cffd27090481\") " pod="openshift-ingress/payload-processing-e2e-migrate-57bf5177-7879c68c96-896r4" Oct 30 18:09:16.244791 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:16.244722 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db9706fd-5979-4cea-aae8-cffd27090481-plugins-config-volume\") pod \"payload-processing-e2e-migrate-57bf5177-7879c68c96-896r4\" (UID: \"db9706fd-5979-4cea-aae8-cffd27090481\") " pod="openshift-ingress/payload-processing-e2e-migrate-57bf5177-7879c68c96-896r4" Oct 30 18:09:16.245419 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:16.245399 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1d7323e-9efa-4c36-b781-5bc157008ed4-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-57bf5177-5f6f57f4f5-xw6rx\" (UID: \"f1d7323e-9efa-4c36-b781-5bc157008ed4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-57bf5177-5f6f57f4f5-xw6rx" Oct 30 18:09:16.263277 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:16.263244 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72pkf\" (UniqueName: \"kubernetes.io/projected/f1d7323e-9efa-4c36-b781-5bc157008ed4-kube-api-access-72pkf\") pod \"payload-pre-processing-e2e-migrate-57bf5177-5f6f57f4f5-xw6rx\" (UID: \"f1d7323e-9efa-4c36-b781-5bc157008ed4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-57bf5177-5f6f57f4f5-xw6rx" Oct 30 18:09:16.271213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:16.271188 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-57bf5177-5f6f57f4f5-xw6rx" Oct 30 18:09:16.345799 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:16.345769 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr6lp\" (UniqueName: \"kubernetes.io/projected/db9706fd-5979-4cea-aae8-cffd27090481-kube-api-access-nr6lp\") pod \"payload-processing-e2e-migrate-57bf5177-7879c68c96-896r4\" (UID: \"db9706fd-5979-4cea-aae8-cffd27090481\") " pod="openshift-ingress/payload-processing-e2e-migrate-57bf5177-7879c68c96-896r4" Oct 30 18:09:16.345947 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:16.345807 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db9706fd-5979-4cea-aae8-cffd27090481-plugins-config-volume\") pod \"payload-processing-e2e-migrate-57bf5177-7879c68c96-896r4\" (UID: \"db9706fd-5979-4cea-aae8-cffd27090481\") " pod="openshift-ingress/payload-processing-e2e-migrate-57bf5177-7879c68c96-896r4" Oct 30 18:09:16.346599 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:16.346579 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db9706fd-5979-4cea-aae8-cffd27090481-plugins-config-volume\") pod \"payload-processing-e2e-migrate-57bf5177-7879c68c96-896r4\" (UID: \"db9706fd-5979-4cea-aae8-cffd27090481\") " pod="openshift-ingress/payload-processing-e2e-migrate-57bf5177-7879c68c96-896r4" Oct 30 18:09:16.365246 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:16.365216 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr6lp\" (UniqueName: \"kubernetes.io/projected/db9706fd-5979-4cea-aae8-cffd27090481-kube-api-access-nr6lp\") pod \"payload-processing-e2e-migrate-57bf5177-7879c68c96-896r4\" (UID: \"db9706fd-5979-4cea-aae8-cffd27090481\") " pod="openshift-ingress/payload-processing-e2e-migrate-57bf5177-7879c68c96-896r4" Oct 30 18:09:16.408282 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:16.408253 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-57bf5177-7879c68c96-896r4" Oct 30 18:09:16.546511 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:16.546477 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-57bf5177-b587ccc9-9q6rr" event={"ID":"12e28d3d-06c0-4e7f-ade8-f8c062d97969","Type":"ContainerStarted","Data":"efa2d69f371532d99e5ae2dd5c9f089722fa02586bbcc6a8e90e7a4dcaf7a286"} Oct 30 18:09:16.574841 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:16.574807 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-57bf5177-5f6f57f4f5-xw6rx"] Oct 30 18:09:16.577898 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:09:16.577876 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1d7323e_9efa_4c36_b781_5bc157008ed4.slice/crio-29d822aace56b274a0be2912e85a15dabbf3c20ebab851dc3c97d5033f85b82e WatchSource:0}: Error finding container 29d822aace56b274a0be2912e85a15dabbf3c20ebab851dc3c97d5033f85b82e: Status 404 returned error can't find the container with id 29d822aace56b274a0be2912e85a15dabbf3c20ebab851dc3c97d5033f85b82e Oct 30 18:09:16.715178 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:16.715150 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-57bf5177-7879c68c96-896r4"] Oct 30 18:09:16.718005 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:09:16.717977 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb9706fd_5979_4cea_aae8_cffd27090481.slice/crio-eb992d3567dece17b8a2280d8c6ced66f65135f5a3b88ccb1e0dbc2c2a08888e WatchSource:0}: Error finding container eb992d3567dece17b8a2280d8c6ced66f65135f5a3b88ccb1e0dbc2c2a08888e: Status 404 returned error can't find the container with id eb992d3567dece17b8a2280d8c6ced66f65135f5a3b88ccb1e0dbc2c2a08888e Oct 30 18:09:17.542885 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:17.542852 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-54599e13-84bddc8b55-8hngq"] Oct 30 18:09:17.547309 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:17.547283 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54599e13-84bddc8b55-8hngq" Oct 30 18:09:17.563380 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:17.563350 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-54599e13-serving-cert\"" Oct 30 18:09:17.585160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:17.575633 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-57bf5177-5f6f57f4f5-xw6rx" event={"ID":"f1d7323e-9efa-4c36-b781-5bc157008ed4","Type":"ContainerStarted","Data":"29d822aace56b274a0be2912e85a15dabbf3c20ebab851dc3c97d5033f85b82e"} Oct 30 18:09:17.585160 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:09:17.576796 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:09:17.585160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:17.578667 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-57bf5177-7879c68c96-896r4" event={"ID":"db9706fd-5979-4cea-aae8-cffd27090481","Type":"ContainerStarted","Data":"eb992d3567dece17b8a2280d8c6ced66f65135f5a3b88ccb1e0dbc2c2a08888e"} Oct 30 18:09:17.585160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:17.578727 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-54599e13-84bddc8b55-8hngq"] Oct 30 18:09:17.648640 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:17.648605 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-594f46d3-9c97b6f9-8f6d8"] Oct 30 18:09:17.661492 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:17.661363 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-594f46d3-9c97b6f9-8f6d8" Oct 30 18:09:17.673764 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:17.673723 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-594f46d3-9c97b6f9-8f6d8"] Oct 30 18:09:17.679582 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:17.679557 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-594f46d3\"" Oct 30 18:09:17.679861 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:17.679841 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-594f46d3-dockercfg-wpc8z\"" Oct 30 18:09:17.681050 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:17.681023 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a782e223-3075-4fb8-b38c-1c009b65de6e-openshift-service-ca\") pod \"maas-api-e2e-ait-54599e13-84bddc8b55-8hngq\" (UID: \"a782e223-3075-4fb8-b38c-1c009b65de6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54599e13-84bddc8b55-8hngq" Oct 30 18:09:17.682303 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:17.682277 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce043f5d-7b08-420c-84c9-ea0b405b8ba9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-594f46d3-9c97b6f9-8f6d8\" (UID: \"ce043f5d-7b08-420c-84c9-ea0b405b8ba9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-594f46d3-9c97b6f9-8f6d8" Oct 30 18:09:17.682486 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:17.682464 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a782e223-3075-4fb8-b38c-1c009b65de6e-metrics-certs\") pod \"maas-api-e2e-ait-54599e13-84bddc8b55-8hngq\" (UID: \"a782e223-3075-4fb8-b38c-1c009b65de6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54599e13-84bddc8b55-8hngq" Oct 30 18:09:17.682619 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:17.682604 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a782e223-3075-4fb8-b38c-1c009b65de6e-maas-api-tls\") pod \"maas-api-e2e-ait-54599e13-84bddc8b55-8hngq\" (UID: \"a782e223-3075-4fb8-b38c-1c009b65de6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54599e13-84bddc8b55-8hngq" Oct 30 18:09:17.682809 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:17.682790 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzqpc\" (UniqueName: \"kubernetes.io/projected/ce043f5d-7b08-420c-84c9-ea0b405b8ba9-kube-api-access-rzqpc\") pod \"payload-pre-processing-e2e-ait-594f46d3-9c97b6f9-8f6d8\" (UID: \"ce043f5d-7b08-420c-84c9-ea0b405b8ba9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-594f46d3-9c97b6f9-8f6d8" Oct 30 18:09:17.682961 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:17.682945 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q858\" (UniqueName: \"kubernetes.io/projected/a782e223-3075-4fb8-b38c-1c009b65de6e-kube-api-access-6q858\") pod \"maas-api-e2e-ait-54599e13-84bddc8b55-8hngq\" (UID: \"a782e223-3075-4fb8-b38c-1c009b65de6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54599e13-84bddc8b55-8hngq" Oct 30 18:09:17.729458 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:17.729419 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-594f46d3-86dd96cf4d-vpwfb"] Oct 30 18:09:17.734055 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:17.734033 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-594f46d3-86dd96cf4d-vpwfb" Oct 30 18:09:17.757173 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:17.757119 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-594f46d3-86dd96cf4d-vpwfb"] Oct 30 18:09:17.787632 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:17.783631 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac471711-23a8-405b-a77d-3e74378ba04d-plugins-config-volume\") pod \"payload-processing-e2e-ait-594f46d3-86dd96cf4d-vpwfb\" (UID: \"ac471711-23a8-405b-a77d-3e74378ba04d\") " pod="openshift-ingress/payload-processing-e2e-ait-594f46d3-86dd96cf4d-vpwfb" Oct 30 18:09:17.787632 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:17.783729 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a782e223-3075-4fb8-b38c-1c009b65de6e-openshift-service-ca\") pod \"maas-api-e2e-ait-54599e13-84bddc8b55-8hngq\" (UID: \"a782e223-3075-4fb8-b38c-1c009b65de6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54599e13-84bddc8b55-8hngq" Oct 30 18:09:17.787632 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:17.783786 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mg8w\" (UniqueName: \"kubernetes.io/projected/ac471711-23a8-405b-a77d-3e74378ba04d-kube-api-access-4mg8w\") pod \"payload-processing-e2e-ait-594f46d3-86dd96cf4d-vpwfb\" (UID: \"ac471711-23a8-405b-a77d-3e74378ba04d\") " pod="openshift-ingress/payload-processing-e2e-ait-594f46d3-86dd96cf4d-vpwfb" Oct 30 18:09:17.787632 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:17.783816 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce043f5d-7b08-420c-84c9-ea0b405b8ba9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-594f46d3-9c97b6f9-8f6d8\" (UID: \"ce043f5d-7b08-420c-84c9-ea0b405b8ba9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-594f46d3-9c97b6f9-8f6d8" Oct 30 18:09:17.787632 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:17.783855 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a782e223-3075-4fb8-b38c-1c009b65de6e-metrics-certs\") pod \"maas-api-e2e-ait-54599e13-84bddc8b55-8hngq\" (UID: \"a782e223-3075-4fb8-b38c-1c009b65de6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54599e13-84bddc8b55-8hngq" Oct 30 18:09:17.787632 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:17.783883 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a782e223-3075-4fb8-b38c-1c009b65de6e-maas-api-tls\") pod \"maas-api-e2e-ait-54599e13-84bddc8b55-8hngq\" (UID: \"a782e223-3075-4fb8-b38c-1c009b65de6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54599e13-84bddc8b55-8hngq" Oct 30 18:09:17.787632 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:17.783971 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzqpc\" (UniqueName: \"kubernetes.io/projected/ce043f5d-7b08-420c-84c9-ea0b405b8ba9-kube-api-access-rzqpc\") pod \"payload-pre-processing-e2e-ait-594f46d3-9c97b6f9-8f6d8\" (UID: \"ce043f5d-7b08-420c-84c9-ea0b405b8ba9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-594f46d3-9c97b6f9-8f6d8" Oct 30 18:09:17.787632 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:17.784008 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q858\" (UniqueName: \"kubernetes.io/projected/a782e223-3075-4fb8-b38c-1c009b65de6e-kube-api-access-6q858\") pod \"maas-api-e2e-ait-54599e13-84bddc8b55-8hngq\" (UID: \"a782e223-3075-4fb8-b38c-1c009b65de6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54599e13-84bddc8b55-8hngq" Oct 30 18:09:17.787632 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:17.785040 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce043f5d-7b08-420c-84c9-ea0b405b8ba9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-594f46d3-9c97b6f9-8f6d8\" (UID: \"ce043f5d-7b08-420c-84c9-ea0b405b8ba9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-594f46d3-9c97b6f9-8f6d8" Oct 30 18:09:17.787632 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:17.785217 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a782e223-3075-4fb8-b38c-1c009b65de6e-openshift-service-ca\") pod \"maas-api-e2e-ait-54599e13-84bddc8b55-8hngq\" (UID: \"a782e223-3075-4fb8-b38c-1c009b65de6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54599e13-84bddc8b55-8hngq" Oct 30 18:09:17.788322 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:17.788254 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a782e223-3075-4fb8-b38c-1c009b65de6e-metrics-certs\") pod \"maas-api-e2e-ait-54599e13-84bddc8b55-8hngq\" (UID: \"a782e223-3075-4fb8-b38c-1c009b65de6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54599e13-84bddc8b55-8hngq" Oct 30 18:09:17.800300 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:17.800185 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a782e223-3075-4fb8-b38c-1c009b65de6e-maas-api-tls\") pod \"maas-api-e2e-ait-54599e13-84bddc8b55-8hngq\" (UID: \"a782e223-3075-4fb8-b38c-1c009b65de6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54599e13-84bddc8b55-8hngq" Oct 30 18:09:17.820881 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:17.820847 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzqpc\" (UniqueName: \"kubernetes.io/projected/ce043f5d-7b08-420c-84c9-ea0b405b8ba9-kube-api-access-rzqpc\") pod \"payload-pre-processing-e2e-ait-594f46d3-9c97b6f9-8f6d8\" (UID: \"ce043f5d-7b08-420c-84c9-ea0b405b8ba9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-594f46d3-9c97b6f9-8f6d8" Oct 30 18:09:17.830777 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:17.826879 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q858\" (UniqueName: \"kubernetes.io/projected/a782e223-3075-4fb8-b38c-1c009b65de6e-kube-api-access-6q858\") pod \"maas-api-e2e-ait-54599e13-84bddc8b55-8hngq\" (UID: \"a782e223-3075-4fb8-b38c-1c009b65de6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54599e13-84bddc8b55-8hngq" Oct 30 18:09:17.870653 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:17.870341 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54599e13-84bddc8b55-8hngq" Oct 30 18:09:17.887353 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:17.886695 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac471711-23a8-405b-a77d-3e74378ba04d-plugins-config-volume\") pod \"payload-processing-e2e-ait-594f46d3-86dd96cf4d-vpwfb\" (UID: \"ac471711-23a8-405b-a77d-3e74378ba04d\") " pod="openshift-ingress/payload-processing-e2e-ait-594f46d3-86dd96cf4d-vpwfb" Oct 30 18:09:17.887353 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:17.886784 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mg8w\" (UniqueName: \"kubernetes.io/projected/ac471711-23a8-405b-a77d-3e74378ba04d-kube-api-access-4mg8w\") pod \"payload-processing-e2e-ait-594f46d3-86dd96cf4d-vpwfb\" (UID: \"ac471711-23a8-405b-a77d-3e74378ba04d\") " pod="openshift-ingress/payload-processing-e2e-ait-594f46d3-86dd96cf4d-vpwfb" Oct 30 18:09:17.888945 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:17.888060 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac471711-23a8-405b-a77d-3e74378ba04d-plugins-config-volume\") pod \"payload-processing-e2e-ait-594f46d3-86dd96cf4d-vpwfb\" (UID: \"ac471711-23a8-405b-a77d-3e74378ba04d\") " pod="openshift-ingress/payload-processing-e2e-ait-594f46d3-86dd96cf4d-vpwfb" Oct 30 18:09:17.906254 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:17.906225 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mg8w\" (UniqueName: \"kubernetes.io/projected/ac471711-23a8-405b-a77d-3e74378ba04d-kube-api-access-4mg8w\") pod \"payload-processing-e2e-ait-594f46d3-86dd96cf4d-vpwfb\" (UID: \"ac471711-23a8-405b-a77d-3e74378ba04d\") " pod="openshift-ingress/payload-processing-e2e-ait-594f46d3-86dd96cf4d-vpwfb" Oct 30 18:09:17.988552 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:17.988042 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-594f46d3-9c97b6f9-8f6d8" Oct 30 18:09:18.057665 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:18.057578 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-594f46d3-86dd96cf4d-vpwfb" Oct 30 18:09:18.218374 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:18.218342 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-54599e13-84bddc8b55-8hngq"] Oct 30 18:09:18.224954 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:09:18.224909 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda782e223_3075_4fb8_b38c_1c009b65de6e.slice/crio-dde375ab584d5760ea27b5c7e9620e470570130f6d33f00c020a1cefc1851819 WatchSource:0}: Error finding container dde375ab584d5760ea27b5c7e9620e470570130f6d33f00c020a1cefc1851819: Status 404 returned error can't find the container with id dde375ab584d5760ea27b5c7e9620e470570130f6d33f00c020a1cefc1851819 Oct 30 18:09:18.315889 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:18.315235 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-594f46d3-9c97b6f9-8f6d8"] Oct 30 18:09:18.318686 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:09:18.318626 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce043f5d_7b08_420c_84c9_ea0b405b8ba9.slice/crio-b8516cf11a92fb7502f8de7ac66226aa7ce79b4ad2e221513703768e7233af17 WatchSource:0}: Error finding container b8516cf11a92fb7502f8de7ac66226aa7ce79b4ad2e221513703768e7233af17: Status 404 returned error can't find the container with id b8516cf11a92fb7502f8de7ac66226aa7ce79b4ad2e221513703768e7233af17 Oct 30 18:09:18.412391 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:18.412362 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-594f46d3-86dd96cf4d-vpwfb"] Oct 30 18:09:18.416639 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:09:18.416609 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac471711_23a8_405b_a77d_3e74378ba04d.slice/crio-11a26e422818a8d7913a1ebe2e67ed2c6c29a4c2d64ff97a564984e3777938c0 WatchSource:0}: Error finding container 11a26e422818a8d7913a1ebe2e67ed2c6c29a4c2d64ff97a564984e3777938c0: Status 404 returned error can't find the container with id 11a26e422818a8d7913a1ebe2e67ed2c6c29a4c2d64ff97a564984e3777938c0 Oct 30 18:09:18.591915 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:18.591829 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-54599e13-9fdd4c687-9dncf"] Oct 30 18:09:18.597992 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:18.597966 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-594f46d3-86dd96cf4d-vpwfb" event={"ID":"ac471711-23a8-405b-a77d-3e74378ba04d","Type":"ContainerStarted","Data":"11a26e422818a8d7913a1ebe2e67ed2c6c29a4c2d64ff97a564984e3777938c0"} Oct 30 18:09:18.598165 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:18.598000 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54599e13-84bddc8b55-8hngq" event={"ID":"a782e223-3075-4fb8-b38c-1c009b65de6e","Type":"ContainerStarted","Data":"dde375ab584d5760ea27b5c7e9620e470570130f6d33f00c020a1cefc1851819"} Oct 30 18:09:18.598165 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:18.598019 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-594f46d3-9c97b6f9-8f6d8" event={"ID":"ce043f5d-7b08-420c-84c9-ea0b405b8ba9","Type":"ContainerStarted","Data":"b8516cf11a92fb7502f8de7ac66226aa7ce79b4ad2e221513703768e7233af17"} Oct 30 18:09:18.598165 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:18.598078 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-54599e13-9fdd4c687-9dncf" Oct 30 18:09:18.613898 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:18.613758 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-54599e13-dockercfg-qqwtz\"" Oct 30 18:09:18.613898 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:18.613822 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-54599e13\"" Oct 30 18:09:18.616010 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:18.615984 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-54599e13-9fdd4c687-9dncf"] Oct 30 18:09:18.680154 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:18.680113 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-54599e13-69cdd8dfb-fnppd"] Oct 30 18:09:18.684292 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:18.684264 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-54599e13-69cdd8dfb-fnppd" Oct 30 18:09:18.705935 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:18.705910 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-54599e13-69cdd8dfb-fnppd"] Oct 30 18:09:18.812218 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:18.798979 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vxmx\" (UniqueName: \"kubernetes.io/projected/13cfe51c-22d4-49f0-827b-b10bac35f0c1-kube-api-access-4vxmx\") pod \"payload-processing-e2e-ait-54599e13-69cdd8dfb-fnppd\" (UID: \"13cfe51c-22d4-49f0-827b-b10bac35f0c1\") " pod="openshift-ingress/payload-processing-e2e-ait-54599e13-69cdd8dfb-fnppd" Oct 30 18:09:18.812218 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:18.799099 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13cfe51c-22d4-49f0-827b-b10bac35f0c1-plugins-config-volume\") pod \"payload-processing-e2e-ait-54599e13-69cdd8dfb-fnppd\" (UID: \"13cfe51c-22d4-49f0-827b-b10bac35f0c1\") " pod="openshift-ingress/payload-processing-e2e-ait-54599e13-69cdd8dfb-fnppd" Oct 30 18:09:18.812218 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:18.799126 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8bcefce-471d-44ab-93c7-e5917c7e6231-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-54599e13-9fdd4c687-9dncf\" (UID: \"d8bcefce-471d-44ab-93c7-e5917c7e6231\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-54599e13-9fdd4c687-9dncf" Oct 30 18:09:18.812218 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:18.799256 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrkxw\" (UniqueName: \"kubernetes.io/projected/d8bcefce-471d-44ab-93c7-e5917c7e6231-kube-api-access-zrkxw\") pod \"payload-pre-processing-e2e-ait-54599e13-9fdd4c687-9dncf\" (UID: \"d8bcefce-471d-44ab-93c7-e5917c7e6231\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-54599e13-9fdd4c687-9dncf" Oct 30 18:09:18.900059 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:18.900010 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrkxw\" (UniqueName: \"kubernetes.io/projected/d8bcefce-471d-44ab-93c7-e5917c7e6231-kube-api-access-zrkxw\") pod \"payload-pre-processing-e2e-ait-54599e13-9fdd4c687-9dncf\" (UID: \"d8bcefce-471d-44ab-93c7-e5917c7e6231\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-54599e13-9fdd4c687-9dncf" Oct 30 18:09:18.900270 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:18.900095 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vxmx\" (UniqueName: \"kubernetes.io/projected/13cfe51c-22d4-49f0-827b-b10bac35f0c1-kube-api-access-4vxmx\") pod \"payload-processing-e2e-ait-54599e13-69cdd8dfb-fnppd\" (UID: \"13cfe51c-22d4-49f0-827b-b10bac35f0c1\") " pod="openshift-ingress/payload-processing-e2e-ait-54599e13-69cdd8dfb-fnppd" Oct 30 18:09:18.900270 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:18.900167 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13cfe51c-22d4-49f0-827b-b10bac35f0c1-plugins-config-volume\") pod \"payload-processing-e2e-ait-54599e13-69cdd8dfb-fnppd\" (UID: \"13cfe51c-22d4-49f0-827b-b10bac35f0c1\") " pod="openshift-ingress/payload-processing-e2e-ait-54599e13-69cdd8dfb-fnppd" Oct 30 18:09:18.900270 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:18.900197 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8bcefce-471d-44ab-93c7-e5917c7e6231-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-54599e13-9fdd4c687-9dncf\" (UID: \"d8bcefce-471d-44ab-93c7-e5917c7e6231\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-54599e13-9fdd4c687-9dncf" Oct 30 18:09:18.901012 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:18.900985 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8bcefce-471d-44ab-93c7-e5917c7e6231-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-54599e13-9fdd4c687-9dncf\" (UID: \"d8bcefce-471d-44ab-93c7-e5917c7e6231\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-54599e13-9fdd4c687-9dncf" Oct 30 18:09:18.902022 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:18.901998 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13cfe51c-22d4-49f0-827b-b10bac35f0c1-plugins-config-volume\") pod \"payload-processing-e2e-ait-54599e13-69cdd8dfb-fnppd\" (UID: \"13cfe51c-22d4-49f0-827b-b10bac35f0c1\") " pod="openshift-ingress/payload-processing-e2e-ait-54599e13-69cdd8dfb-fnppd" Oct 30 18:09:18.921213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:18.921173 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrkxw\" (UniqueName: \"kubernetes.io/projected/d8bcefce-471d-44ab-93c7-e5917c7e6231-kube-api-access-zrkxw\") pod \"payload-pre-processing-e2e-ait-54599e13-9fdd4c687-9dncf\" (UID: \"d8bcefce-471d-44ab-93c7-e5917c7e6231\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-54599e13-9fdd4c687-9dncf" Oct 30 18:09:18.922430 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:18.922105 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vxmx\" (UniqueName: \"kubernetes.io/projected/13cfe51c-22d4-49f0-827b-b10bac35f0c1-kube-api-access-4vxmx\") pod \"payload-processing-e2e-ait-54599e13-69cdd8dfb-fnppd\" (UID: \"13cfe51c-22d4-49f0-827b-b10bac35f0c1\") " pod="openshift-ingress/payload-processing-e2e-ait-54599e13-69cdd8dfb-fnppd" Oct 30 18:09:19.013650 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:19.013617 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-54599e13-69cdd8dfb-fnppd" Oct 30 18:09:19.220429 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:19.220331 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-54599e13-9fdd4c687-9dncf" Oct 30 18:09:19.452834 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:19.452761 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a521cf49-7f9d8dc77f-m9gg7"] Oct 30 18:09:19.465664 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:19.465597 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-54599e13-69cdd8dfb-fnppd"] Oct 30 18:09:19.466472 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:19.465908 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a521cf49-7f9d8dc77f-m9gg7" Oct 30 18:09:19.470229 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:09:19.470159 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13cfe51c_22d4_49f0_827b_b10bac35f0c1.slice/crio-8361ec365888f46eb1030adad70e564751c5f723285af294a9432f22ed8238f1 WatchSource:0}: Error finding container 8361ec365888f46eb1030adad70e564751c5f723285af294a9432f22ed8238f1: Status 404 returned error can't find the container with id 8361ec365888f46eb1030adad70e564751c5f723285af294a9432f22ed8238f1 Oct 30 18:09:19.486287 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:19.486046 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a521cf49-7f9d8dc77f-m9gg7"] Oct 30 18:09:19.492612 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:19.491789 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a521cf49-dockercfg-jm5dp\"" Oct 30 18:09:19.576055 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:19.541635 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a521cf49\"" Oct 30 18:09:19.610742 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:19.610366 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdr2z\" (UniqueName: \"kubernetes.io/projected/4627d314-3100-4129-8681-af9e2c7d13be-kube-api-access-gdr2z\") pod \"payload-pre-processing-e2e-ait-a521cf49-7f9d8dc77f-m9gg7\" (UID: \"4627d314-3100-4129-8681-af9e2c7d13be\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a521cf49-7f9d8dc77f-m9gg7" Oct 30 18:09:19.610742 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:19.610550 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4627d314-3100-4129-8681-af9e2c7d13be-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a521cf49-7f9d8dc77f-m9gg7\" (UID: \"4627d314-3100-4129-8681-af9e2c7d13be\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a521cf49-7f9d8dc77f-m9gg7" Oct 30 18:09:19.633237 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:19.633121 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-54599e13-69cdd8dfb-fnppd" event={"ID":"13cfe51c-22d4-49f0-827b-b10bac35f0c1","Type":"ContainerStarted","Data":"8361ec365888f46eb1030adad70e564751c5f723285af294a9432f22ed8238f1"} Oct 30 18:09:19.641103 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:19.641075 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-57bf5177-b587ccc9-9q6rr" event={"ID":"12e28d3d-06c0-4e7f-ade8-f8c062d97969","Type":"ContainerStarted","Data":"6b24b8e2ec4eccd14c0e3bb7a4134147325a480d504e7f325de21b3173162489"} Oct 30 18:09:19.642077 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:19.642055 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-57bf5177-b587ccc9-9q6rr" Oct 30 18:09:19.644178 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:19.644153 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-54599e13-9fdd4c687-9dncf"] Oct 30 18:09:19.685187 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:19.685089 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-57bf5177-b587ccc9-9q6rr" podStartSLOduration=1.488664013 podStartE2EDuration="4.685070046s" totalImagesPullingTime="3.196406033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:19.680878638 +0000 UTC m=+656.770940980" watchObservedRunningTime="2026-10-30 18:09:19.685070046 +0000 UTC m=+656.775132386" Oct 30 18:09:19.713666 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:19.712261 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdr2z\" (UniqueName: \"kubernetes.io/projected/4627d314-3100-4129-8681-af9e2c7d13be-kube-api-access-gdr2z\") pod \"payload-pre-processing-e2e-ait-a521cf49-7f9d8dc77f-m9gg7\" (UID: \"4627d314-3100-4129-8681-af9e2c7d13be\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a521cf49-7f9d8dc77f-m9gg7" Oct 30 18:09:19.713666 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:19.712352 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4627d314-3100-4129-8681-af9e2c7d13be-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a521cf49-7f9d8dc77f-m9gg7\" (UID: \"4627d314-3100-4129-8681-af9e2c7d13be\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a521cf49-7f9d8dc77f-m9gg7" Oct 30 18:09:19.713666 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:19.713266 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4627d314-3100-4129-8681-af9e2c7d13be-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a521cf49-7f9d8dc77f-m9gg7\" (UID: \"4627d314-3100-4129-8681-af9e2c7d13be\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a521cf49-7f9d8dc77f-m9gg7" Oct 30 18:09:19.742169 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:19.742042 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdr2z\" (UniqueName: \"kubernetes.io/projected/4627d314-3100-4129-8681-af9e2c7d13be-kube-api-access-gdr2z\") pod \"payload-pre-processing-e2e-ait-a521cf49-7f9d8dc77f-m9gg7\" (UID: \"4627d314-3100-4129-8681-af9e2c7d13be\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a521cf49-7f9d8dc77f-m9gg7" Oct 30 18:09:19.803749 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:19.797370 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a521cf49-7f9d8dc77f-m9gg7" Oct 30 18:09:20.176115 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:20.176078 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a521cf49-7f9d8dc77f-m9gg7"] Oct 30 18:09:20.669081 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:20.668989 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-54599e13-9fdd4c687-9dncf" event={"ID":"d8bcefce-471d-44ab-93c7-e5917c7e6231","Type":"ContainerStarted","Data":"4572ea1f698d54ab263a90b285839245c8f678d5e4eb7d650ec6c5f78cfa3820"} Oct 30 18:09:20.674345 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:20.674268 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a521cf49-7f9d8dc77f-m9gg7" event={"ID":"4627d314-3100-4129-8681-af9e2c7d13be","Type":"ContainerStarted","Data":"6a44b71fd194d93f5e87aa35c8b7b8aef52ac7631a160dfa099c5fbcd70a40c8"} Oct 30 18:09:21.695379 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:21.695342 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54599e13-84bddc8b55-8hngq" event={"ID":"a782e223-3075-4fb8-b38c-1c009b65de6e","Type":"ContainerStarted","Data":"5c9c84fb3cdde6d399216a670d9d50d51d904f6f1183dd0df74435945ad22a04"} Oct 30 18:09:21.695830 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:21.695433 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54599e13-84bddc8b55-8hngq" Oct 30 18:09:22.703641 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:22.703548 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-57bf5177-5f6f57f4f5-xw6rx" event={"ID":"f1d7323e-9efa-4c36-b781-5bc157008ed4","Type":"ContainerStarted","Data":"cf1a0005b53231788629b8693364caab26dd596053b63e991d14c1f1bcdc13d3"} Oct 30 18:09:22.704103 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:22.703713 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-57bf5177-5f6f57f4f5-xw6rx" Oct 30 18:09:22.705195 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:22.705168 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-54599e13-69cdd8dfb-fnppd" event={"ID":"13cfe51c-22d4-49f0-827b-b10bac35f0c1","Type":"ContainerStarted","Data":"bd4ef26a937e95fb0d094202344d83c2234f013ce5a4207b9e7513421ecb5ffd"} Oct 30 18:09:22.705332 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:22.705267 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-54599e13-69cdd8dfb-fnppd" Oct 30 18:09:22.706938 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:22.706912 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-594f46d3-86dd96cf4d-vpwfb" event={"ID":"ac471711-23a8-405b-a77d-3e74378ba04d","Type":"ContainerStarted","Data":"8eda477e537cee3c06a3d67eb7228e398a2a9c09b6fb9d6162e4281d38659c3b"} Oct 30 18:09:22.707051 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:22.706961 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-594f46d3-86dd96cf4d-vpwfb" Oct 30 18:09:22.708750 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:22.708724 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-57bf5177-7879c68c96-896r4" event={"ID":"db9706fd-5979-4cea-aae8-cffd27090481","Type":"ContainerStarted","Data":"16bbd7b0a48094436b32f6a7e083dcb961842ca93404c167c7ea34f36d6a1280"} Oct 30 18:09:22.708880 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:22.708793 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-57bf5177-7879c68c96-896r4" Oct 30 18:09:22.710800 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:22.710384 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-54599e13-9fdd4c687-9dncf" event={"ID":"d8bcefce-471d-44ab-93c7-e5917c7e6231","Type":"ContainerStarted","Data":"2db9bc43604c2819fb2a969160d82fa7d44ad5ef7234b1b89a838506d7f36a4c"} Oct 30 18:09:22.744422 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:22.744279 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-57bf5177-5f6f57f4f5-xw6rx" podStartSLOduration=2.084271297 podStartE2EDuration="7.744257832s" totalImagesPullingTime="5.659986535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:22.741491002 +0000 UTC m=+659.831553342" watchObservedRunningTime="2026-10-30 18:09:22.744257832 +0000 UTC m=+659.834320171" Oct 30 18:09:22.744669 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:22.744647 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54599e13-84bddc8b55-8hngq" podStartSLOduration=3.116072306 podStartE2EDuration="5.744643229s" totalImagesPullingTime="2.628570923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:21.741057047 +0000 UTC m=+658.831119387" watchObservedRunningTime="2026-10-30 18:09:22.744643229 +0000 UTC m=+659.834705570" Oct 30 18:09:22.783327 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:22.783271 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-57bf5177-7879c68c96-896r4" podStartSLOduration=1.269246183 podStartE2EDuration="6.783245632s" totalImagesPullingTime="5.513999449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:22.779905977 +0000 UTC m=+659.869968319" watchObservedRunningTime="2026-10-30 18:09:22.783245632 +0000 UTC m=+659.873307973" Oct 30 18:09:22.821243 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:22.821188 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-54599e13-69cdd8dfb-fnppd" podStartSLOduration=1.797057573 podStartE2EDuration="4.821172168s" totalImagesPullingTime="3.024114595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:22.818761226 +0000 UTC m=+659.908823577" watchObservedRunningTime="2026-10-30 18:09:22.821172168 +0000 UTC m=+659.911234506" Oct 30 18:09:22.860980 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:22.860927 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-594f46d3-86dd96cf4d-vpwfb" podStartSLOduration=2.112819517 podStartE2EDuration="5.860906771s" totalImagesPullingTime="3.748087254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:22.857548355 +0000 UTC m=+659.947610707" watchObservedRunningTime="2026-10-30 18:09:22.860906771 +0000 UTC m=+659.950969110" Oct 30 18:09:22.902411 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:22.902359 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-54599e13-9fdd4c687-9dncf" podStartSLOduration=2.140419827 podStartE2EDuration="4.902342246s" totalImagesPullingTime="2.761922419s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:22.899437421 +0000 UTC m=+659.989499764" watchObservedRunningTime="2026-10-30 18:09:22.902342246 +0000 UTC m=+659.992404586" Oct 30 18:09:23.718752 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:23.718709 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-594f46d3-9c97b6f9-8f6d8" event={"ID":"ce043f5d-7b08-420c-84c9-ea0b405b8ba9","Type":"ContainerStarted","Data":"0887e38453a57276c1d4e02464cb083eb905ebe73103c3473b1ea9110bc42d52"} Oct 30 18:09:23.719175 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:23.718774 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-594f46d3-9c97b6f9-8f6d8" Oct 30 18:09:23.720543 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:23.720518 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a521cf49-7f9d8dc77f-m9gg7" event={"ID":"4627d314-3100-4129-8681-af9e2c7d13be","Type":"ContainerStarted","Data":"fc8f37f71c350c01fcfabbaf7def744f4d76b852d4c1cc1f7aa817757eedfe92"} Oct 30 18:09:23.720697 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:23.720678 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a521cf49-7f9d8dc77f-m9gg7" Oct 30 18:09:23.720750 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:23.720735 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-54599e13-9fdd4c687-9dncf" Oct 30 18:09:23.757370 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:23.757320 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-594f46d3-9c97b6f9-8f6d8" podStartSLOduration=2.059924069 podStartE2EDuration="6.757306387s" totalImagesPullingTime="4.697382318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:23.756738246 +0000 UTC m=+660.846800585" watchObservedRunningTime="2026-10-30 18:09:23.757306387 +0000 UTC m=+660.847368775" Oct 30 18:09:23.796585 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:23.796510 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a521cf49-7f9d8dc77f-m9gg7" podStartSLOduration=2.236246066 podStartE2EDuration="4.796493432s" totalImagesPullingTime="2.560247366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:23.794888122 +0000 UTC m=+660.884950464" watchObservedRunningTime="2026-10-30 18:09:23.796493432 +0000 UTC m=+660.886555772" Oct 30 18:09:26.700887 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:26.700854 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-57bf5177-b587ccc9-9q6rr" Oct 30 18:09:28.757159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:28.756314 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54599e13-84bddc8b55-8hngq" Oct 30 18:09:33.724437 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:33.724402 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-594f46d3-86dd96cf4d-vpwfb" Oct 30 18:09:33.725072 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:33.724495 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-54599e13-69cdd8dfb-fnppd" Oct 30 18:09:33.725072 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:33.724996 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-57bf5177-5f6f57f4f5-xw6rx" Oct 30 18:09:33.725660 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:33.725628 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-57bf5177-7879c68c96-896r4" Oct 30 18:09:33.819459 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:33.819428 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ba56e909-c84888fc5-mm82n"] Oct 30 18:09:33.824724 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:33.824699 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ba56e909-c84888fc5-mm82n" Oct 30 18:09:33.843939 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:33.843909 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ba56e909-c84888fc5-mm82n"] Oct 30 18:09:33.844920 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:33.844898 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-ba56e909\"" Oct 30 18:09:33.845063 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:33.844926 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-ba56e909-dockercfg-z5c6f\"" Oct 30 18:09:33.911723 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:33.911686 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ba56e909-6f5476fdc4-vmz9q"] Oct 30 18:09:33.915686 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:33.915653 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ba56e909-6f5476fdc4-vmz9q" Oct 30 18:09:33.936417 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:33.930424 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpkxm\" (UniqueName: \"kubernetes.io/projected/b8edb1cd-d37b-4dd8-8600-729318034c2a-kube-api-access-vpkxm\") pod \"payload-pre-processing-e2e-derive-ba56e909-c84888fc5-mm82n\" (UID: \"b8edb1cd-d37b-4dd8-8600-729318034c2a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ba56e909-c84888fc5-mm82n" Oct 30 18:09:33.936417 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:33.930469 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8edb1cd-d37b-4dd8-8600-729318034c2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ba56e909-c84888fc5-mm82n\" (UID: \"b8edb1cd-d37b-4dd8-8600-729318034c2a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ba56e909-c84888fc5-mm82n" Oct 30 18:09:33.942905 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:33.942872 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ba56e909-6f5476fdc4-vmz9q"] Oct 30 18:09:34.032243 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:34.032090 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/383feced-9ea0-4524-a76e-84112d54fa10-plugins-config-volume\") pod \"payload-processing-e2e-derive-ba56e909-6f5476fdc4-vmz9q\" (UID: \"383feced-9ea0-4524-a76e-84112d54fa10\") " pod="openshift-ingress/payload-processing-e2e-derive-ba56e909-6f5476fdc4-vmz9q" Oct 30 18:09:34.032820 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:34.032795 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpkxm\" (UniqueName: \"kubernetes.io/projected/b8edb1cd-d37b-4dd8-8600-729318034c2a-kube-api-access-vpkxm\") pod \"payload-pre-processing-e2e-derive-ba56e909-c84888fc5-mm82n\" (UID: \"b8edb1cd-d37b-4dd8-8600-729318034c2a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ba56e909-c84888fc5-mm82n" Oct 30 18:09:34.035983 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:34.035928 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8edb1cd-d37b-4dd8-8600-729318034c2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ba56e909-c84888fc5-mm82n\" (UID: \"b8edb1cd-d37b-4dd8-8600-729318034c2a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ba56e909-c84888fc5-mm82n" Oct 30 18:09:34.037662 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:34.037643 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fphl8\" (UniqueName: \"kubernetes.io/projected/383feced-9ea0-4524-a76e-84112d54fa10-kube-api-access-fphl8\") pod \"payload-processing-e2e-derive-ba56e909-6f5476fdc4-vmz9q\" (UID: \"383feced-9ea0-4524-a76e-84112d54fa10\") " pod="openshift-ingress/payload-processing-e2e-derive-ba56e909-6f5476fdc4-vmz9q" Oct 30 18:09:34.037957 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:34.037581 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8edb1cd-d37b-4dd8-8600-729318034c2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ba56e909-c84888fc5-mm82n\" (UID: \"b8edb1cd-d37b-4dd8-8600-729318034c2a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ba56e909-c84888fc5-mm82n" Oct 30 18:09:34.067035 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:34.067005 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpkxm\" (UniqueName: \"kubernetes.io/projected/b8edb1cd-d37b-4dd8-8600-729318034c2a-kube-api-access-vpkxm\") pod \"payload-pre-processing-e2e-derive-ba56e909-c84888fc5-mm82n\" (UID: \"b8edb1cd-d37b-4dd8-8600-729318034c2a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ba56e909-c84888fc5-mm82n" Oct 30 18:09:34.138854 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:34.138816 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fphl8\" (UniqueName: \"kubernetes.io/projected/383feced-9ea0-4524-a76e-84112d54fa10-kube-api-access-fphl8\") pod \"payload-processing-e2e-derive-ba56e909-6f5476fdc4-vmz9q\" (UID: \"383feced-9ea0-4524-a76e-84112d54fa10\") " pod="openshift-ingress/payload-processing-e2e-derive-ba56e909-6f5476fdc4-vmz9q" Oct 30 18:09:34.139040 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:34.138898 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/383feced-9ea0-4524-a76e-84112d54fa10-plugins-config-volume\") pod \"payload-processing-e2e-derive-ba56e909-6f5476fdc4-vmz9q\" (UID: \"383feced-9ea0-4524-a76e-84112d54fa10\") " pod="openshift-ingress/payload-processing-e2e-derive-ba56e909-6f5476fdc4-vmz9q" Oct 30 18:09:34.139549 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:34.139530 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/383feced-9ea0-4524-a76e-84112d54fa10-plugins-config-volume\") pod \"payload-processing-e2e-derive-ba56e909-6f5476fdc4-vmz9q\" (UID: \"383feced-9ea0-4524-a76e-84112d54fa10\") " pod="openshift-ingress/payload-processing-e2e-derive-ba56e909-6f5476fdc4-vmz9q" Oct 30 18:09:34.140679 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:34.140657 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ba56e909-c84888fc5-mm82n" Oct 30 18:09:34.161360 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:34.160322 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fphl8\" (UniqueName: \"kubernetes.io/projected/383feced-9ea0-4524-a76e-84112d54fa10-kube-api-access-fphl8\") pod \"payload-processing-e2e-derive-ba56e909-6f5476fdc4-vmz9q\" (UID: \"383feced-9ea0-4524-a76e-84112d54fa10\") " pod="openshift-ingress/payload-processing-e2e-derive-ba56e909-6f5476fdc4-vmz9q" Oct 30 18:09:34.238325 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:34.238294 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ba56e909-6f5476fdc4-vmz9q" Oct 30 18:09:34.448537 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:34.448507 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ba56e909-c84888fc5-mm82n"] Oct 30 18:09:34.451807 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:09:34.451779 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8edb1cd_d37b_4dd8_8600_729318034c2a.slice/crio-16748c27a9248eae0319ef031db9f65d9b8f0a3059802506f7d587b056033b52 WatchSource:0}: Error finding container 16748c27a9248eae0319ef031db9f65d9b8f0a3059802506f7d587b056033b52: Status 404 returned error can't find the container with id 16748c27a9248eae0319ef031db9f65d9b8f0a3059802506f7d587b056033b52 Oct 30 18:09:34.547823 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:34.547766 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ba56e909-6f5476fdc4-vmz9q"] Oct 30 18:09:34.557480 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:09:34.557398 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod383feced_9ea0_4524_a76e_84112d54fa10.slice/crio-813d722277a6454f3413041b8f669deccec73c3638c1e403fd701955b3a14bb5 WatchSource:0}: Error finding container 813d722277a6454f3413041b8f669deccec73c3638c1e403fd701955b3a14bb5: Status 404 returned error can't find the container with id 813d722277a6454f3413041b8f669deccec73c3638c1e403fd701955b3a14bb5 Oct 30 18:09:34.729358 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:34.729330 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-54599e13-9fdd4c687-9dncf" Oct 30 18:09:34.729754 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:34.729398 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-594f46d3-9c97b6f9-8f6d8" Oct 30 18:09:34.729754 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:34.729438 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a521cf49-7f9d8dc77f-m9gg7" Oct 30 18:09:34.804232 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:34.804195 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ba56e909-6f5476fdc4-vmz9q" event={"ID":"383feced-9ea0-4524-a76e-84112d54fa10","Type":"ContainerStarted","Data":"4e47fcd638b51d08685ad7f6736e10850b4899d90eb62138cef21a1beb7201d9"} Oct 30 18:09:34.804420 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:34.804280 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-ba56e909-6f5476fdc4-vmz9q" Oct 30 18:09:34.804851 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:34.804303 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ba56e909-6f5476fdc4-vmz9q" event={"ID":"383feced-9ea0-4524-a76e-84112d54fa10","Type":"ContainerStarted","Data":"813d722277a6454f3413041b8f669deccec73c3638c1e403fd701955b3a14bb5"} Oct 30 18:09:34.807197 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:34.807164 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ba56e909-c84888fc5-mm82n" event={"ID":"b8edb1cd-d37b-4dd8-8600-729318034c2a","Type":"ContainerStarted","Data":"27bc1c83d1a22b5e49e3bfe66a08944accef9b477281fd5814f3f5ecbb22f0a2"} Oct 30 18:09:34.807328 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:34.807218 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ba56e909-c84888fc5-mm82n" event={"ID":"b8edb1cd-d37b-4dd8-8600-729318034c2a","Type":"ContainerStarted","Data":"16748c27a9248eae0319ef031db9f65d9b8f0a3059802506f7d587b056033b52"} Oct 30 18:09:34.807328 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:34.807246 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-ba56e909-c84888fc5-mm82n" Oct 30 18:09:34.890907 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:34.890855 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-ba56e909-c84888fc5-mm82n" podStartSLOduration=1.8908405510000001 podStartE2EDuration="1.890840551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:34.887058243 +0000 UTC m=+671.977120582" watchObservedRunningTime="2026-10-30 18:09:34.890840551 +0000 UTC m=+671.980902889" Oct 30 18:09:34.927266 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:34.927204 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-ba56e909-6f5476fdc4-vmz9q" podStartSLOduration=1.927185175 podStartE2EDuration="1.927185175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:34.924957473 +0000 UTC m=+672.015019814" watchObservedRunningTime="2026-10-30 18:09:34.927185175 +0000 UTC m=+672.017247514" Oct 30 18:09:38.755478 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:38.755444 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm"] Oct 30 18:09:38.759022 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:38.758997 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm" Oct 30 18:09:38.779596 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:38.779572 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-ba56e909-serving-cert\"" Oct 30 18:09:38.790608 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:38.790579 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm"] Oct 30 18:09:38.899540 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:38.899492 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa665c80-1db7-41b2-b3f7-5259248ccc05-maas-api-tls\") pod \"maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm\" (UID: \"fa665c80-1db7-41b2-b3f7-5259248ccc05\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm" Oct 30 18:09:38.899724 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:38.899575 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa665c80-1db7-41b2-b3f7-5259248ccc05-openshift-service-ca\") pod \"maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm\" (UID: \"fa665c80-1db7-41b2-b3f7-5259248ccc05\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm" Oct 30 18:09:38.899724 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:38.899700 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa665c80-1db7-41b2-b3f7-5259248ccc05-metrics-certs\") pod \"maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm\" (UID: \"fa665c80-1db7-41b2-b3f7-5259248ccc05\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm" Oct 30 18:09:38.899818 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:38.899800 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbvk9\" (UniqueName: \"kubernetes.io/projected/fa665c80-1db7-41b2-b3f7-5259248ccc05-kube-api-access-mbvk9\") pod \"maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm\" (UID: \"fa665c80-1db7-41b2-b3f7-5259248ccc05\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm" Oct 30 18:09:39.001709 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:39.001677 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa665c80-1db7-41b2-b3f7-5259248ccc05-openshift-service-ca\") pod \"maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm\" (UID: \"fa665c80-1db7-41b2-b3f7-5259248ccc05\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm" Oct 30 18:09:39.001869 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:39.001738 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa665c80-1db7-41b2-b3f7-5259248ccc05-metrics-certs\") pod \"maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm\" (UID: \"fa665c80-1db7-41b2-b3f7-5259248ccc05\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm" Oct 30 18:09:39.001869 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:39.001813 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbvk9\" (UniqueName: \"kubernetes.io/projected/fa665c80-1db7-41b2-b3f7-5259248ccc05-kube-api-access-mbvk9\") pod \"maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm\" (UID: \"fa665c80-1db7-41b2-b3f7-5259248ccc05\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm" Oct 30 18:09:39.002000 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:39.001868 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa665c80-1db7-41b2-b3f7-5259248ccc05-maas-api-tls\") pod \"maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm\" (UID: \"fa665c80-1db7-41b2-b3f7-5259248ccc05\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm" Oct 30 18:09:39.002055 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:09:39.002006 4442 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-ba56e909-serving-cert: secret "maas-api-e2e-derive-ba56e909-serving-cert" not found Oct 30 18:09:39.002109 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:09:39.002078 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa665c80-1db7-41b2-b3f7-5259248ccc05-maas-api-tls podName:fa665c80-1db7-41b2-b3f7-5259248ccc05 nodeName:}" failed. No retries permitted until 2026-10-30 18:09:39.502057557 +0000 UTC m=+676.592119879 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fa665c80-1db7-41b2-b3f7-5259248ccc05-maas-api-tls") pod "maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm" (UID: "fa665c80-1db7-41b2-b3f7-5259248ccc05") : secret "maas-api-e2e-derive-ba56e909-serving-cert" not found Oct 30 18:09:39.002457 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:39.002437 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa665c80-1db7-41b2-b3f7-5259248ccc05-openshift-service-ca\") pod \"maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm\" (UID: \"fa665c80-1db7-41b2-b3f7-5259248ccc05\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm" Oct 30 18:09:39.004419 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:39.004399 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa665c80-1db7-41b2-b3f7-5259248ccc05-metrics-certs\") pod \"maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm\" (UID: \"fa665c80-1db7-41b2-b3f7-5259248ccc05\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm" Oct 30 18:09:39.022931 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:39.022874 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbvk9\" (UniqueName: \"kubernetes.io/projected/fa665c80-1db7-41b2-b3f7-5259248ccc05-kube-api-access-mbvk9\") pod \"maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm\" (UID: \"fa665c80-1db7-41b2-b3f7-5259248ccc05\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm" Oct 30 18:09:39.506261 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:39.506226 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa665c80-1db7-41b2-b3f7-5259248ccc05-maas-api-tls\") pod \"maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm\" (UID: \"fa665c80-1db7-41b2-b3f7-5259248ccc05\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm" Oct 30 18:09:39.506444 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:09:39.506380 4442 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-ba56e909-serving-cert: secret "maas-api-e2e-derive-ba56e909-serving-cert" not found Oct 30 18:09:39.506487 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:09:39.506445 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa665c80-1db7-41b2-b3f7-5259248ccc05-maas-api-tls podName:fa665c80-1db7-41b2-b3f7-5259248ccc05 nodeName:}" failed. No retries permitted until 2026-10-30 18:09:40.506430377 +0000 UTC m=+677.596492694 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fa665c80-1db7-41b2-b3f7-5259248ccc05-maas-api-tls") pod "maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm" (UID: "fa665c80-1db7-41b2-b3f7-5259248ccc05") : secret "maas-api-e2e-derive-ba56e909-serving-cert" not found Oct 30 18:09:40.519387 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:40.519355 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa665c80-1db7-41b2-b3f7-5259248ccc05-maas-api-tls\") pod \"maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm\" (UID: \"fa665c80-1db7-41b2-b3f7-5259248ccc05\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm" Oct 30 18:09:40.521960 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:40.521932 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa665c80-1db7-41b2-b3f7-5259248ccc05-maas-api-tls\") pod \"maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm\" (UID: \"fa665c80-1db7-41b2-b3f7-5259248ccc05\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm" Oct 30 18:09:40.572729 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:40.572690 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm" Oct 30 18:09:40.880715 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:40.880690 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm"] Oct 30 18:09:40.883756 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:09:40.883726 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa665c80_1db7_41b2_b3f7_5259248ccc05.slice/crio-737fe68196f681fb9a1da462a15a5bb0cabe9f634e9cdafb579b6e981a18bc21 WatchSource:0}: Error finding container 737fe68196f681fb9a1da462a15a5bb0cabe9f634e9cdafb579b6e981a18bc21: Status 404 returned error can't find the container with id 737fe68196f681fb9a1da462a15a5bb0cabe9f634e9cdafb579b6e981a18bc21 Oct 30 18:09:41.875637 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:41.875597 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm" event={"ID":"fa665c80-1db7-41b2-b3f7-5259248ccc05","Type":"ContainerStarted","Data":"303ac54d76f25b3b09ec2379eead8731efa5d59272b564793cb504082055c699"} Oct 30 18:09:41.876185 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:41.875655 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm" event={"ID":"fa665c80-1db7-41b2-b3f7-5259248ccc05","Type":"ContainerStarted","Data":"737fe68196f681fb9a1da462a15a5bb0cabe9f634e9cdafb579b6e981a18bc21"} Oct 30 18:09:41.876185 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:41.875689 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm" Oct 30 18:09:41.920301 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:41.920248 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm" podStartSLOduration=3.516188261 podStartE2EDuration="3.920226924s" totalImagesPullingTime="404.038663ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:41.918306178 +0000 UTC m=+679.008368556" watchObservedRunningTime="2026-10-30 18:09:41.920226924 +0000 UTC m=+679.010289264" Oct 30 18:09:45.820608 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:45.820579 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-ba56e909-c84888fc5-mm82n" Oct 30 18:09:46.845785 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:46.845755 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-ba56e909-6f5476fdc4-vmz9q" Oct 30 18:09:47.560616 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:09:47.560580 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:09:47.736872 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:47.736834 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a521cf49-7f9d8dc77f-m9gg7"] Oct 30 18:09:47.893980 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:47.893952 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm" Oct 30 18:09:50.543758 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:50.543718 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-594f46d3-86dd96cf4d-vpwfb"] Oct 30 18:09:50.579735 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:50.579696 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-594f46d3-9c97b6f9-8f6d8"] Oct 30 18:09:52.737839 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:52.737792 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a521cf49-7f9d8dc77f-m9gg7" podUID="4627d314-3100-4129-8681-af9e2c7d13be" containerName="payload-pre-processing" containerID="cri-o://fc8f37f71c350c01fcfabbaf7def744f4d76b852d4c1cc1f7aa817757eedfe92" gracePeriod=25 Oct 30 18:09:52.962446 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:52.962304 4442 generic.go:356] "Generic (PLEG): container finished" podID="4627d314-3100-4129-8681-af9e2c7d13be" containerID="fc8f37f71c350c01fcfabbaf7def744f4d76b852d4c1cc1f7aa817757eedfe92" exitCode=0 Oct 30 18:09:52.962731 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:52.962466 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a521cf49-7f9d8dc77f-m9gg7" event={"ID":"4627d314-3100-4129-8681-af9e2c7d13be","Type":"ContainerDied","Data":"fc8f37f71c350c01fcfabbaf7def744f4d76b852d4c1cc1f7aa817757eedfe92"} Oct 30 18:09:53.035402 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:53.035376 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a521cf49-7f9d8dc77f-m9gg7" Oct 30 18:09:53.085346 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:53.085319 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4627d314-3100-4129-8681-af9e2c7d13be-kube-api-access-gdr2z\" (UniqueName: \"kubernetes.io/projected/4627d314-3100-4129-8681-af9e2c7d13be-kube-api-access-gdr2z\") pod \"4627d314-3100-4129-8681-af9e2c7d13be\" (UID: \"4627d314-3100-4129-8681-af9e2c7d13be\") " Oct 30 18:09:53.085638 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:53.085396 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4627d314-3100-4129-8681-af9e2c7d13be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4627d314-3100-4129-8681-af9e2c7d13be-plugins-config-volume\") pod \"4627d314-3100-4129-8681-af9e2c7d13be\" (UID: \"4627d314-3100-4129-8681-af9e2c7d13be\") " Oct 30 18:09:53.091735 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:53.091698 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4627d314-3100-4129-8681-af9e2c7d13be-plugins-config-volume" pod "4627d314-3100-4129-8681-af9e2c7d13be" (UID: "4627d314-3100-4129-8681-af9e2c7d13be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:09:53.092313 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:53.092287 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4627d314-3100-4129-8681-af9e2c7d13be-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:09:53.104171 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:53.104111 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4627d314-3100-4129-8681-af9e2c7d13be-kube-api-access-gdr2z" pod "4627d314-3100-4129-8681-af9e2c7d13be" (UID: "4627d314-3100-4129-8681-af9e2c7d13be"). InnerVolumeSpecName "kube-api-access-gdr2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:09:53.193451 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:53.193408 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdr2z\" (UniqueName: \"kubernetes.io/projected/4627d314-3100-4129-8681-af9e2c7d13be-kube-api-access-gdr2z\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:09:53.970804 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:53.970769 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a521cf49-7f9d8dc77f-m9gg7" Oct 30 18:09:53.970804 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:53.970782 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a521cf49-7f9d8dc77f-m9gg7" event={"ID":"4627d314-3100-4129-8681-af9e2c7d13be","Type":"ContainerDied","Data":"6a44b71fd194d93f5e87aa35c8b7b8aef52ac7631a160dfa099c5fbcd70a40c8"} Oct 30 18:09:53.971390 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:53.970877 4442 scope.go:122] "RemoveContainer" containerID="fc8f37f71c350c01fcfabbaf7def744f4d76b852d4c1cc1f7aa817757eedfe92" Oct 30 18:09:54.024483 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:54.024450 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a521cf49-7f9d8dc77f-m9gg7"] Oct 30 18:09:54.029106 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:54.029076 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a521cf49-7f9d8dc77f-m9gg7"] Oct 30 18:09:55.063525 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:55.063488 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4627d314-3100-4129-8681-af9e2c7d13be" path="/var/lib/kubelet/pods/4627d314-3100-4129-8681-af9e2c7d13be/volumes" Oct 30 18:09:55.441258 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:55.441224 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-57bf5177-7879c68c96-896r4"] Oct 30 18:09:55.481125 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:55.481096 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-57bf5177-5f6f57f4f5-xw6rx"] Oct 30 18:09:55.544628 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:55.544585 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-594f46d3-86dd96cf4d-vpwfb" podUID="ac471711-23a8-405b-a77d-3e74378ba04d" containerName="payload-processing" containerID="cri-o://8eda477e537cee3c06a3d67eb7228e398a2a9c09b6fb9d6162e4281d38659c3b" gracePeriod=25 Oct 30 18:09:55.580181 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:55.580104 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-594f46d3-9c97b6f9-8f6d8" podUID="ce043f5d-7b08-420c-84c9-ea0b405b8ba9" containerName="payload-pre-processing" containerID="cri-o://0887e38453a57276c1d4e02464cb083eb905ebe73103c3473b1ea9110bc42d52" gracePeriod=25 Oct 30 18:09:55.897903 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:55.897868 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-594f46d3-9c97b6f9-8f6d8" Oct 30 18:09:55.900835 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:55.900818 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-594f46d3-86dd96cf4d-vpwfb" Oct 30 18:09:55.988199 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:55.988085 4442 generic.go:356] "Generic (PLEG): container finished" podID="ac471711-23a8-405b-a77d-3e74378ba04d" containerID="8eda477e537cee3c06a3d67eb7228e398a2a9c09b6fb9d6162e4281d38659c3b" exitCode=0 Oct 30 18:09:55.988199 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:55.988175 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-594f46d3-86dd96cf4d-vpwfb" Oct 30 18:09:55.988199 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:55.988175 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-594f46d3-86dd96cf4d-vpwfb" event={"ID":"ac471711-23a8-405b-a77d-3e74378ba04d","Type":"ContainerDied","Data":"8eda477e537cee3c06a3d67eb7228e398a2a9c09b6fb9d6162e4281d38659c3b"} Oct 30 18:09:55.988449 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:55.988248 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-594f46d3-86dd96cf4d-vpwfb" event={"ID":"ac471711-23a8-405b-a77d-3e74378ba04d","Type":"ContainerDied","Data":"11a26e422818a8d7913a1ebe2e67ed2c6c29a4c2d64ff97a564984e3777938c0"} Oct 30 18:09:55.988449 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:55.988300 4442 scope.go:122] "RemoveContainer" containerID="8eda477e537cee3c06a3d67eb7228e398a2a9c09b6fb9d6162e4281d38659c3b" Oct 30 18:09:55.994529 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:55.994505 4442 generic.go:356] "Generic (PLEG): container finished" podID="ce043f5d-7b08-420c-84c9-ea0b405b8ba9" containerID="0887e38453a57276c1d4e02464cb083eb905ebe73103c3473b1ea9110bc42d52" exitCode=0 Oct 30 18:09:55.994650 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:55.994548 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-594f46d3-9c97b6f9-8f6d8" event={"ID":"ce043f5d-7b08-420c-84c9-ea0b405b8ba9","Type":"ContainerDied","Data":"0887e38453a57276c1d4e02464cb083eb905ebe73103c3473b1ea9110bc42d52"} Oct 30 18:09:55.994650 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:55.994591 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-594f46d3-9c97b6f9-8f6d8" event={"ID":"ce043f5d-7b08-420c-84c9-ea0b405b8ba9","Type":"ContainerDied","Data":"b8516cf11a92fb7502f8de7ac66226aa7ce79b4ad2e221513703768e7233af17"} Oct 30 18:09:55.994786 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:55.994660 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-594f46d3-9c97b6f9-8f6d8" Oct 30 18:09:55.999603 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:55.999584 4442 scope.go:122] "RemoveContainer" containerID="8eda477e537cee3c06a3d67eb7228e398a2a9c09b6fb9d6162e4281d38659c3b" Oct 30 18:09:55.999877 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:09:55.999859 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eda477e537cee3c06a3d67eb7228e398a2a9c09b6fb9d6162e4281d38659c3b\": container with ID starting with 8eda477e537cee3c06a3d67eb7228e398a2a9c09b6fb9d6162e4281d38659c3b not found: ID does not exist" containerID="8eda477e537cee3c06a3d67eb7228e398a2a9c09b6fb9d6162e4281d38659c3b" Oct 30 18:09:55.999967 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:55.999888 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eda477e537cee3c06a3d67eb7228e398a2a9c09b6fb9d6162e4281d38659c3b"} err="failed to get container status \"8eda477e537cee3c06a3d67eb7228e398a2a9c09b6fb9d6162e4281d38659c3b\": rpc error: code = NotFound desc = could not find container \"8eda477e537cee3c06a3d67eb7228e398a2a9c09b6fb9d6162e4281d38659c3b\": container with ID starting with 8eda477e537cee3c06a3d67eb7228e398a2a9c09b6fb9d6162e4281d38659c3b not found: ID does not exist" Oct 30 18:09:55.999967 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:55.999913 4442 scope.go:122] "RemoveContainer" containerID="0887e38453a57276c1d4e02464cb083eb905ebe73103c3473b1ea9110bc42d52" Oct 30 18:09:56.011220 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:56.011198 4442 scope.go:122] "RemoveContainer" containerID="0887e38453a57276c1d4e02464cb083eb905ebe73103c3473b1ea9110bc42d52" Oct 30 18:09:56.011552 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:09:56.011526 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0887e38453a57276c1d4e02464cb083eb905ebe73103c3473b1ea9110bc42d52\": container with ID starting with 0887e38453a57276c1d4e02464cb083eb905ebe73103c3473b1ea9110bc42d52 not found: ID does not exist" containerID="0887e38453a57276c1d4e02464cb083eb905ebe73103c3473b1ea9110bc42d52" Oct 30 18:09:56.011620 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:56.011562 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0887e38453a57276c1d4e02464cb083eb905ebe73103c3473b1ea9110bc42d52"} err="failed to get container status \"0887e38453a57276c1d4e02464cb083eb905ebe73103c3473b1ea9110bc42d52\": rpc error: code = NotFound desc = could not find container \"0887e38453a57276c1d4e02464cb083eb905ebe73103c3473b1ea9110bc42d52\": container with ID starting with 0887e38453a57276c1d4e02464cb083eb905ebe73103c3473b1ea9110bc42d52 not found: ID does not exist" Oct 30 18:09:56.054353 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:56.054321 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce043f5d-7b08-420c-84c9-ea0b405b8ba9-kube-api-access-rzqpc\" (UniqueName: \"kubernetes.io/projected/ce043f5d-7b08-420c-84c9-ea0b405b8ba9-kube-api-access-rzqpc\") pod \"ce043f5d-7b08-420c-84c9-ea0b405b8ba9\" (UID: \"ce043f5d-7b08-420c-84c9-ea0b405b8ba9\") " Oct 30 18:09:56.054507 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:56.054376 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac471711-23a8-405b-a77d-3e74378ba04d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac471711-23a8-405b-a77d-3e74378ba04d-plugins-config-volume\") pod \"ac471711-23a8-405b-a77d-3e74378ba04d\" (UID: \"ac471711-23a8-405b-a77d-3e74378ba04d\") " Oct 30 18:09:56.054507 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:56.054492 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce043f5d-7b08-420c-84c9-ea0b405b8ba9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce043f5d-7b08-420c-84c9-ea0b405b8ba9-plugins-config-volume\") pod \"ce043f5d-7b08-420c-84c9-ea0b405b8ba9\" (UID: \"ce043f5d-7b08-420c-84c9-ea0b405b8ba9\") " Oct 30 18:09:56.054633 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:56.054525 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac471711-23a8-405b-a77d-3e74378ba04d-kube-api-access-4mg8w\" (UniqueName: \"kubernetes.io/projected/ac471711-23a8-405b-a77d-3e74378ba04d-kube-api-access-4mg8w\") pod \"ac471711-23a8-405b-a77d-3e74378ba04d\" (UID: \"ac471711-23a8-405b-a77d-3e74378ba04d\") " Oct 30 18:09:56.054778 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:56.054756 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac471711-23a8-405b-a77d-3e74378ba04d-plugins-config-volume" pod "ac471711-23a8-405b-a77d-3e74378ba04d" (UID: "ac471711-23a8-405b-a77d-3e74378ba04d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:09:56.054778 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:56.054826 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce043f5d-7b08-420c-84c9-ea0b405b8ba9-plugins-config-volume" pod "ce043f5d-7b08-420c-84c9-ea0b405b8ba9" (UID: "ce043f5d-7b08-420c-84c9-ea0b405b8ba9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:09:56.054966 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:56.054879 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac471711-23a8-405b-a77d-3e74378ba04d-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:09:56.056654 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:56.056632 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce043f5d-7b08-420c-84c9-ea0b405b8ba9-kube-api-access-rzqpc" pod "ce043f5d-7b08-420c-84c9-ea0b405b8ba9" (UID: "ce043f5d-7b08-420c-84c9-ea0b405b8ba9"). InnerVolumeSpecName "kube-api-access-rzqpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:09:56.056741 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:56.056691 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac471711-23a8-405b-a77d-3e74378ba04d-kube-api-access-4mg8w" pod "ac471711-23a8-405b-a77d-3e74378ba04d" (UID: "ac471711-23a8-405b-a77d-3e74378ba04d"). InnerVolumeSpecName "kube-api-access-4mg8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:09:56.156902 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:56.156351 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce043f5d-7b08-420c-84c9-ea0b405b8ba9-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:09:56.156902 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:56.156388 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mg8w\" (UniqueName: \"kubernetes.io/projected/ac471711-23a8-405b-a77d-3e74378ba04d-kube-api-access-4mg8w\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:09:56.156902 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:56.156402 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzqpc\" (UniqueName: \"kubernetes.io/projected/ce043f5d-7b08-420c-84c9-ea0b405b8ba9-kube-api-access-rzqpc\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:09:56.347011 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:56.346978 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-594f46d3-86dd96cf4d-vpwfb"] Oct 30 18:09:56.349889 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:56.349862 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-594f46d3-86dd96cf4d-vpwfb"] Oct 30 18:09:56.390852 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:56.390819 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-594f46d3-9c97b6f9-8f6d8"] Oct 30 18:09:56.394349 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:56.394321 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-594f46d3-9c97b6f9-8f6d8"] Oct 30 18:09:57.064878 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:57.064830 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac471711-23a8-405b-a77d-3e74378ba04d" path="/var/lib/kubelet/pods/ac471711-23a8-405b-a77d-3e74378ba04d/volumes" Oct 30 18:09:57.065301 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:57.065286 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce043f5d-7b08-420c-84c9-ea0b405b8ba9" path="/var/lib/kubelet/pods/ce043f5d-7b08-420c-84c9-ea0b405b8ba9/volumes" Oct 30 18:09:58.420939 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:58.420905 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-57bf5177-b587ccc9-9q6rr"] Oct 30 18:09:58.421662 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:58.421629 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-57bf5177-b587ccc9-9q6rr" podUID="12e28d3d-06c0-4e7f-ade8-f8c062d97969" containerName="maas-api" containerID="cri-o://6b24b8e2ec4eccd14c0e3bb7a4134147325a480d504e7f325de21b3173162489" gracePeriod=30 Oct 30 18:09:58.512121 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:58.512090 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-921450-7f64fb4c79-xs76b"] Oct 30 18:09:58.512584 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:58.512571 4442 cpu_manager.go:414] "containerMap: removing container" podUID="ac471711-23a8-405b-a77d-3e74378ba04d" containerName="payload-processing" Oct 30 18:09:58.512626 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:58.512586 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac471711-23a8-405b-a77d-3e74378ba04d" containerName="payload-processing" Oct 30 18:09:58.512626 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:58.512611 4442 cpu_manager.go:414] "containerMap: removing container" podUID="ce043f5d-7b08-420c-84c9-ea0b405b8ba9" containerName="payload-pre-processing" Oct 30 18:09:58.512626 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:58.512616 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce043f5d-7b08-420c-84c9-ea0b405b8ba9" containerName="payload-pre-processing" Oct 30 18:09:58.512725 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:58.512630 4442 cpu_manager.go:414] "containerMap: removing container" podUID="4627d314-3100-4129-8681-af9e2c7d13be" containerName="payload-pre-processing" Oct 30 18:09:58.512725 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:58.512636 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4627d314-3100-4129-8681-af9e2c7d13be" containerName="payload-pre-processing" Oct 30 18:09:58.512725 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:58.512708 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce043f5d-7b08-420c-84c9-ea0b405b8ba9" containerName="payload-pre-processing" Oct 30 18:09:58.512725 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:58.512716 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="4627d314-3100-4129-8681-af9e2c7d13be" containerName="payload-pre-processing" Oct 30 18:09:58.512834 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:58.512727 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac471711-23a8-405b-a77d-3e74378ba04d" containerName="payload-processing" Oct 30 18:09:58.515902 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:58.515882 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-921450-7f64fb4c79-xs76b" Oct 30 18:09:58.531267 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:58.531240 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-921450\"" Oct 30 18:09:58.531420 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:58.531297 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-921450-dockercfg-fnzvz\"" Oct 30 18:09:58.535831 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:58.535800 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-921450-7f64fb4c79-xs76b"] Oct 30 18:09:58.583595 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:58.583541 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02cb3b89-6926-459a-a7ed-de16a0d30d6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-921450-7f64fb4c79-xs76b\" (UID: \"02cb3b89-6926-459a-a7ed-de16a0d30d6f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-921450-7f64fb4c79-xs76b" Oct 30 18:09:58.583923 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:58.583744 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjrgq\" (UniqueName: \"kubernetes.io/projected/02cb3b89-6926-459a-a7ed-de16a0d30d6f-kube-api-access-wjrgq\") pod \"payload-pre-processing-e2e-worker-w0-921450-7f64fb4c79-xs76b\" (UID: \"02cb3b89-6926-459a-a7ed-de16a0d30d6f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-921450-7f64fb4c79-xs76b" Oct 30 18:09:58.645122 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:58.644801 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-921450-8d6546c9b-5pxxc"] Oct 30 18:09:58.651912 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:58.651325 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-921450-8d6546c9b-5pxxc" Oct 30 18:09:58.653965 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:58.653901 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-921450-8d6546c9b-5pxxc"] Oct 30 18:09:58.685400 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:58.685284 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjrgq\" (UniqueName: \"kubernetes.io/projected/02cb3b89-6926-459a-a7ed-de16a0d30d6f-kube-api-access-wjrgq\") pod \"payload-pre-processing-e2e-worker-w0-921450-7f64fb4c79-xs76b\" (UID: \"02cb3b89-6926-459a-a7ed-de16a0d30d6f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-921450-7f64fb4c79-xs76b" Oct 30 18:09:58.685560 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:58.685420 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02cb3b89-6926-459a-a7ed-de16a0d30d6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-921450-7f64fb4c79-xs76b\" (UID: \"02cb3b89-6926-459a-a7ed-de16a0d30d6f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-921450-7f64fb4c79-xs76b" Oct 30 18:09:58.686289 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:58.686257 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02cb3b89-6926-459a-a7ed-de16a0d30d6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-921450-7f64fb4c79-xs76b\" (UID: \"02cb3b89-6926-459a-a7ed-de16a0d30d6f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-921450-7f64fb4c79-xs76b" Oct 30 18:09:58.708314 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:58.708124 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjrgq\" (UniqueName: \"kubernetes.io/projected/02cb3b89-6926-459a-a7ed-de16a0d30d6f-kube-api-access-wjrgq\") pod \"payload-pre-processing-e2e-worker-w0-921450-7f64fb4c79-xs76b\" (UID: \"02cb3b89-6926-459a-a7ed-de16a0d30d6f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-921450-7f64fb4c79-xs76b" Oct 30 18:09:58.786112 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:58.786079 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5pg8\" (UniqueName: \"kubernetes.io/projected/17bb49e3-80a9-4216-bc19-20b1679782a9-kube-api-access-t5pg8\") pod \"payload-processing-e2e-worker-w0-921450-8d6546c9b-5pxxc\" (UID: \"17bb49e3-80a9-4216-bc19-20b1679782a9\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-921450-8d6546c9b-5pxxc" Oct 30 18:09:58.786692 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:58.786665 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17bb49e3-80a9-4216-bc19-20b1679782a9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-921450-8d6546c9b-5pxxc\" (UID: \"17bb49e3-80a9-4216-bc19-20b1679782a9\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-921450-8d6546c9b-5pxxc" Oct 30 18:09:58.791676 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:58.791645 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-57bf5177-b587ccc9-9q6rr" Oct 30 18:09:58.830713 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:58.830687 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-921450-7f64fb4c79-xs76b" Oct 30 18:09:58.889155 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:58.887537 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12e28d3d-06c0-4e7f-ade8-f8c062d97969-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12e28d3d-06c0-4e7f-ade8-f8c062d97969-maas-api-tls\") pod \"12e28d3d-06c0-4e7f-ade8-f8c062d97969\" (UID: \"12e28d3d-06c0-4e7f-ade8-f8c062d97969\") " Oct 30 18:09:58.889155 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:58.887601 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12e28d3d-06c0-4e7f-ade8-f8c062d97969-kube-api-access-qljg6\" (UniqueName: \"kubernetes.io/projected/12e28d3d-06c0-4e7f-ade8-f8c062d97969-kube-api-access-qljg6\") pod \"12e28d3d-06c0-4e7f-ade8-f8c062d97969\" (UID: \"12e28d3d-06c0-4e7f-ade8-f8c062d97969\") " Oct 30 18:09:58.889155 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:58.887747 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12e28d3d-06c0-4e7f-ade8-f8c062d97969-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12e28d3d-06c0-4e7f-ade8-f8c062d97969-openshift-service-ca\") pod \"12e28d3d-06c0-4e7f-ade8-f8c062d97969\" (UID: \"12e28d3d-06c0-4e7f-ade8-f8c062d97969\") " Oct 30 18:09:58.889155 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:58.887769 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12e28d3d-06c0-4e7f-ade8-f8c062d97969-metrics-certs\" (UniqueName: \"kubernetes.io/secret/12e28d3d-06c0-4e7f-ade8-f8c062d97969-metrics-certs\") pod \"12e28d3d-06c0-4e7f-ade8-f8c062d97969\" (UID: \"12e28d3d-06c0-4e7f-ade8-f8c062d97969\") " Oct 30 18:09:58.889155 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:58.887863 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5pg8\" (UniqueName: \"kubernetes.io/projected/17bb49e3-80a9-4216-bc19-20b1679782a9-kube-api-access-t5pg8\") pod \"payload-processing-e2e-worker-w0-921450-8d6546c9b-5pxxc\" (UID: \"17bb49e3-80a9-4216-bc19-20b1679782a9\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-921450-8d6546c9b-5pxxc" Oct 30 18:09:58.889155 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:58.887952 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17bb49e3-80a9-4216-bc19-20b1679782a9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-921450-8d6546c9b-5pxxc\" (UID: \"17bb49e3-80a9-4216-bc19-20b1679782a9\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-921450-8d6546c9b-5pxxc" Oct 30 18:09:58.889155 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:58.888680 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17bb49e3-80a9-4216-bc19-20b1679782a9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-921450-8d6546c9b-5pxxc\" (UID: \"17bb49e3-80a9-4216-bc19-20b1679782a9\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-921450-8d6546c9b-5pxxc" Oct 30 18:09:58.889616 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:58.889189 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12e28d3d-06c0-4e7f-ade8-f8c062d97969-openshift-service-ca" pod "12e28d3d-06c0-4e7f-ade8-f8c062d97969" (UID: "12e28d3d-06c0-4e7f-ade8-f8c062d97969"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:09:58.892398 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:58.892366 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12e28d3d-06c0-4e7f-ade8-f8c062d97969-kube-api-access-qljg6" pod "12e28d3d-06c0-4e7f-ade8-f8c062d97969" (UID: "12e28d3d-06c0-4e7f-ade8-f8c062d97969"). InnerVolumeSpecName "kube-api-access-qljg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:09:58.896575 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:58.896543 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12e28d3d-06c0-4e7f-ade8-f8c062d97969-maas-api-tls" pod "12e28d3d-06c0-4e7f-ade8-f8c062d97969" (UID: "12e28d3d-06c0-4e7f-ade8-f8c062d97969"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:09:58.897938 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:58.897908 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12e28d3d-06c0-4e7f-ade8-f8c062d97969-metrics-certs" pod "12e28d3d-06c0-4e7f-ade8-f8c062d97969" (UID: "12e28d3d-06c0-4e7f-ade8-f8c062d97969"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:09:58.913290 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:58.913226 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5pg8\" (UniqueName: \"kubernetes.io/projected/17bb49e3-80a9-4216-bc19-20b1679782a9-kube-api-access-t5pg8\") pod \"payload-processing-e2e-worker-w0-921450-8d6546c9b-5pxxc\" (UID: \"17bb49e3-80a9-4216-bc19-20b1679782a9\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-921450-8d6546c9b-5pxxc" Oct 30 18:09:58.971806 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:58.971720 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-921450-8d6546c9b-5pxxc" Oct 30 18:09:58.989099 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:58.989070 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12e28d3d-06c0-4e7f-ade8-f8c062d97969-openshift-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:09:58.989207 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:58.989103 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12e28d3d-06c0-4e7f-ade8-f8c062d97969-metrics-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:09:58.989207 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:58.989119 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12e28d3d-06c0-4e7f-ade8-f8c062d97969-maas-api-tls\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:09:58.989207 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:58.989147 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qljg6\" (UniqueName: \"kubernetes.io/projected/12e28d3d-06c0-4e7f-ade8-f8c062d97969-kube-api-access-qljg6\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:09:59.034216 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:59.033304 4442 generic.go:356] "Generic (PLEG): container finished" podID="12e28d3d-06c0-4e7f-ade8-f8c062d97969" containerID="6b24b8e2ec4eccd14c0e3bb7a4134147325a480d504e7f325de21b3173162489" exitCode=0 Oct 30 18:09:59.034216 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:59.033410 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-57bf5177-b587ccc9-9q6rr" event={"ID":"12e28d3d-06c0-4e7f-ade8-f8c062d97969","Type":"ContainerDied","Data":"6b24b8e2ec4eccd14c0e3bb7a4134147325a480d504e7f325de21b3173162489"} Oct 30 18:09:59.034216 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:59.033461 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-57bf5177-b587ccc9-9q6rr" event={"ID":"12e28d3d-06c0-4e7f-ade8-f8c062d97969","Type":"ContainerDied","Data":"efa2d69f371532d99e5ae2dd5c9f089722fa02586bbcc6a8e90e7a4dcaf7a286"} Oct 30 18:09:59.034216 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:59.033487 4442 scope.go:122] "RemoveContainer" containerID="6b24b8e2ec4eccd14c0e3bb7a4134147325a480d504e7f325de21b3173162489" Oct 30 18:09:59.034216 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:59.033642 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-57bf5177-b587ccc9-9q6rr" Oct 30 18:09:59.053442 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:59.053424 4442 scope.go:122] "RemoveContainer" containerID="6b24b8e2ec4eccd14c0e3bb7a4134147325a480d504e7f325de21b3173162489" Oct 30 18:09:59.053765 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:09:59.053741 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b24b8e2ec4eccd14c0e3bb7a4134147325a480d504e7f325de21b3173162489\": container with ID starting with 6b24b8e2ec4eccd14c0e3bb7a4134147325a480d504e7f325de21b3173162489 not found: ID does not exist" containerID="6b24b8e2ec4eccd14c0e3bb7a4134147325a480d504e7f325de21b3173162489" Oct 30 18:09:59.053848 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:59.053773 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b24b8e2ec4eccd14c0e3bb7a4134147325a480d504e7f325de21b3173162489"} err="failed to get container status \"6b24b8e2ec4eccd14c0e3bb7a4134147325a480d504e7f325de21b3173162489\": rpc error: code = NotFound desc = could not find container \"6b24b8e2ec4eccd14c0e3bb7a4134147325a480d504e7f325de21b3173162489\": container with ID starting with 6b24b8e2ec4eccd14c0e3bb7a4134147325a480d504e7f325de21b3173162489 not found: ID does not exist" Oct 30 18:09:59.095957 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:59.095931 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-57bf5177-b587ccc9-9q6rr"] Oct 30 18:09:59.102874 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:59.102849 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-57bf5177-b587ccc9-9q6rr"] Oct 30 18:09:59.149591 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:59.149564 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-921450-7f64fb4c79-xs76b"] Oct 30 18:09:59.152113 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:09:59.152089 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02cb3b89_6926_459a_a7ed_de16a0d30d6f.slice/crio-732b1ed0477e53371eeea9c251899c55c10e72832362b69bdb9da0616e84afca WatchSource:0}: Error finding container 732b1ed0477e53371eeea9c251899c55c10e72832362b69bdb9da0616e84afca: Status 404 returned error can't find the container with id 732b1ed0477e53371eeea9c251899c55c10e72832362b69bdb9da0616e84afca Oct 30 18:09:59.279903 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:09:59.279871 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-921450-8d6546c9b-5pxxc"] Oct 30 18:09:59.282938 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:09:59.282902 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17bb49e3_80a9_4216_bc19_20b1679782a9.slice/crio-0727b8630a7383cac66c687913d3be6802b9c5904e1298d9df601624b4504fde WatchSource:0}: Error finding container 0727b8630a7383cac66c687913d3be6802b9c5904e1298d9df601624b4504fde: Status 404 returned error can't find the container with id 0727b8630a7383cac66c687913d3be6802b9c5904e1298d9df601624b4504fde Oct 30 18:10:00.043502 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.043461 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-921450-7f64fb4c79-xs76b" event={"ID":"02cb3b89-6926-459a-a7ed-de16a0d30d6f","Type":"ContainerStarted","Data":"944ac059d0d326575fa0bd96694de71513b880a1af9e5757fc04490b5715dcbb"} Oct 30 18:10:00.043952 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.043536 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-921450-7f64fb4c79-xs76b" Oct 30 18:10:00.043952 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.043562 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-921450-7f64fb4c79-xs76b" event={"ID":"02cb3b89-6926-459a-a7ed-de16a0d30d6f","Type":"ContainerStarted","Data":"732b1ed0477e53371eeea9c251899c55c10e72832362b69bdb9da0616e84afca"} Oct 30 18:10:00.045221 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.045196 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-921450-8d6546c9b-5pxxc" event={"ID":"17bb49e3-80a9-4216-bc19-20b1679782a9","Type":"ContainerStarted","Data":"6162680b62ce5d82dcccecc4dfd922c110a617f7fedee8a8d3a750c98f15febb"} Oct 30 18:10:00.045342 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.045241 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-921450-8d6546c9b-5pxxc" Oct 30 18:10:00.045342 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.045265 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-921450-8d6546c9b-5pxxc" event={"ID":"17bb49e3-80a9-4216-bc19-20b1679782a9","Type":"ContainerStarted","Data":"0727b8630a7383cac66c687913d3be6802b9c5904e1298d9df601624b4504fde"} Oct 30 18:10:00.083892 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.083834 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-921450-7f64fb4c79-xs76b" podStartSLOduration=2.08382039 podStartE2EDuration="2.08382039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:00.083434833 +0000 UTC m=+697.173497184" watchObservedRunningTime="2026-10-30 18:10:00.08382039 +0000 UTC m=+697.173882728" Oct 30 18:10:00.121617 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.121569 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-921450-8d6546c9b-5pxxc" podStartSLOduration=2.121550283 podStartE2EDuration="2.121550283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:00.120584071 +0000 UTC m=+697.210646410" watchObservedRunningTime="2026-10-30 18:10:00.121550283 +0000 UTC m=+697.211612622" Oct 30 18:10:00.441875 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.441832 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-57bf5177-7879c68c96-896r4" podUID="db9706fd-5979-4cea-aae8-cffd27090481" containerName="payload-processing" containerID="cri-o://16bbd7b0a48094436b32f6a7e083dcb961842ca93404c167c7ea34f36d6a1280" gracePeriod=25 Oct 30 18:10:00.470194 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.470166 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-45dc34-d7f945cff-rqxz2"] Oct 30 18:10:00.470631 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.470616 4442 cpu_manager.go:414] "containerMap: removing container" podUID="12e28d3d-06c0-4e7f-ade8-f8c062d97969" containerName="maas-api" Oct 30 18:10:00.470631 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.470631 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12e28d3d-06c0-4e7f-ade8-f8c062d97969" containerName="maas-api" Oct 30 18:10:00.470750 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.470707 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="12e28d3d-06c0-4e7f-ade8-f8c062d97969" containerName="maas-api" Oct 30 18:10:00.473940 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.473921 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-45dc34-d7f945cff-rqxz2" Oct 30 18:10:00.481834 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.481782 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-57bf5177-5f6f57f4f5-xw6rx" podUID="f1d7323e-9efa-4c36-b781-5bc157008ed4" containerName="payload-pre-processing" containerID="cri-o://cf1a0005b53231788629b8693364caab26dd596053b63e991d14c1f1bcdc13d3" gracePeriod=25 Oct 30 18:10:00.491039 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.490630 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-45dc34\"" Oct 30 18:10:00.491039 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.490719 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-45dc34-dockercfg-fcdl6\"" Oct 30 18:10:00.494781 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.494723 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-45dc34-d7f945cff-rqxz2"] Oct 30 18:10:00.508634 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.508611 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96cf77ba-eb58-4e52-bab4-2a4e362dc95c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-45dc34-d7f945cff-rqxz2\" (UID: \"96cf77ba-eb58-4e52-bab4-2a4e362dc95c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-45dc34-d7f945cff-rqxz2" Oct 30 18:10:00.508850 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.508679 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8zcz\" (UniqueName: \"kubernetes.io/projected/96cf77ba-eb58-4e52-bab4-2a4e362dc95c-kube-api-access-q8zcz\") pod \"payload-pre-processing-e2e-worker-w2-45dc34-d7f945cff-rqxz2\" (UID: \"96cf77ba-eb58-4e52-bab4-2a4e362dc95c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-45dc34-d7f945cff-rqxz2" Oct 30 18:10:00.596512 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.593993 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-45dc34-79f9556b56-st7qr"] Oct 30 18:10:00.598640 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.598612 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-45dc34-79f9556b56-st7qr" Oct 30 18:10:00.614075 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.609464 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96cf77ba-eb58-4e52-bab4-2a4e362dc95c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-45dc34-d7f945cff-rqxz2\" (UID: \"96cf77ba-eb58-4e52-bab4-2a4e362dc95c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-45dc34-d7f945cff-rqxz2" Oct 30 18:10:00.614075 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.609525 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8zcz\" (UniqueName: \"kubernetes.io/projected/96cf77ba-eb58-4e52-bab4-2a4e362dc95c-kube-api-access-q8zcz\") pod \"payload-pre-processing-e2e-worker-w2-45dc34-d7f945cff-rqxz2\" (UID: \"96cf77ba-eb58-4e52-bab4-2a4e362dc95c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-45dc34-d7f945cff-rqxz2" Oct 30 18:10:00.614075 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.610606 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96cf77ba-eb58-4e52-bab4-2a4e362dc95c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-45dc34-d7f945cff-rqxz2\" (UID: \"96cf77ba-eb58-4e52-bab4-2a4e362dc95c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-45dc34-d7f945cff-rqxz2" Oct 30 18:10:00.614075 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.611162 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-45dc34-79f9556b56-st7qr"] Oct 30 18:10:00.650888 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.650817 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8zcz\" (UniqueName: \"kubernetes.io/projected/96cf77ba-eb58-4e52-bab4-2a4e362dc95c-kube-api-access-q8zcz\") pod \"payload-pre-processing-e2e-worker-w2-45dc34-d7f945cff-rqxz2\" (UID: \"96cf77ba-eb58-4e52-bab4-2a4e362dc95c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-45dc34-d7f945cff-rqxz2" Oct 30 18:10:00.711855 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.711028 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/269e36c9-21e8-412e-b49e-e446b7d57def-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-45dc34-79f9556b56-st7qr\" (UID: \"269e36c9-21e8-412e-b49e-e446b7d57def\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-45dc34-79f9556b56-st7qr" Oct 30 18:10:00.711855 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.711089 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnt22\" (UniqueName: \"kubernetes.io/projected/269e36c9-21e8-412e-b49e-e446b7d57def-kube-api-access-vnt22\") pod \"payload-processing-e2e-worker-w2-45dc34-79f9556b56-st7qr\" (UID: \"269e36c9-21e8-412e-b49e-e446b7d57def\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-45dc34-79f9556b56-st7qr" Oct 30 18:10:00.787062 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.787033 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-57bf5177-7879c68c96-896r4" Oct 30 18:10:00.812926 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.812108 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db9706fd-5979-4cea-aae8-cffd27090481-kube-api-access-nr6lp\" (UniqueName: \"kubernetes.io/projected/db9706fd-5979-4cea-aae8-cffd27090481-kube-api-access-nr6lp\") pod \"db9706fd-5979-4cea-aae8-cffd27090481\" (UID: \"db9706fd-5979-4cea-aae8-cffd27090481\") " Oct 30 18:10:00.812926 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.812200 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db9706fd-5979-4cea-aae8-cffd27090481-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db9706fd-5979-4cea-aae8-cffd27090481-plugins-config-volume\") pod \"db9706fd-5979-4cea-aae8-cffd27090481\" (UID: \"db9706fd-5979-4cea-aae8-cffd27090481\") " Oct 30 18:10:00.812926 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.812408 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/269e36c9-21e8-412e-b49e-e446b7d57def-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-45dc34-79f9556b56-st7qr\" (UID: \"269e36c9-21e8-412e-b49e-e446b7d57def\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-45dc34-79f9556b56-st7qr" Oct 30 18:10:00.812926 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.812440 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnt22\" (UniqueName: \"kubernetes.io/projected/269e36c9-21e8-412e-b49e-e446b7d57def-kube-api-access-vnt22\") pod \"payload-processing-e2e-worker-w2-45dc34-79f9556b56-st7qr\" (UID: \"269e36c9-21e8-412e-b49e-e446b7d57def\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-45dc34-79f9556b56-st7qr" Oct 30 18:10:00.813364 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.813253 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db9706fd-5979-4cea-aae8-cffd27090481-plugins-config-volume" pod "db9706fd-5979-4cea-aae8-cffd27090481" (UID: "db9706fd-5979-4cea-aae8-cffd27090481"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:10:00.813751 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.813725 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/269e36c9-21e8-412e-b49e-e446b7d57def-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-45dc34-79f9556b56-st7qr\" (UID: \"269e36c9-21e8-412e-b49e-e446b7d57def\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-45dc34-79f9556b56-st7qr" Oct 30 18:10:00.815516 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.815492 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db9706fd-5979-4cea-aae8-cffd27090481-kube-api-access-nr6lp" pod "db9706fd-5979-4cea-aae8-cffd27090481" (UID: "db9706fd-5979-4cea-aae8-cffd27090481"). InnerVolumeSpecName "kube-api-access-nr6lp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:10:00.823987 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.823961 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-57bf5177-5f6f57f4f5-xw6rx" Oct 30 18:10:00.834979 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.834942 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnt22\" (UniqueName: \"kubernetes.io/projected/269e36c9-21e8-412e-b49e-e446b7d57def-kube-api-access-vnt22\") pod \"payload-processing-e2e-worker-w2-45dc34-79f9556b56-st7qr\" (UID: \"269e36c9-21e8-412e-b49e-e446b7d57def\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-45dc34-79f9556b56-st7qr" Oct 30 18:10:00.900464 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.900432 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-45dc34-d7f945cff-rqxz2" Oct 30 18:10:00.913520 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.913429 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1d7323e-9efa-4c36-b781-5bc157008ed4-kube-api-access-72pkf\" (UniqueName: \"kubernetes.io/projected/f1d7323e-9efa-4c36-b781-5bc157008ed4-kube-api-access-72pkf\") pod \"f1d7323e-9efa-4c36-b781-5bc157008ed4\" (UID: \"f1d7323e-9efa-4c36-b781-5bc157008ed4\") " Oct 30 18:10:00.913520 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.913486 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1d7323e-9efa-4c36-b781-5bc157008ed4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1d7323e-9efa-4c36-b781-5bc157008ed4-plugins-config-volume\") pod \"f1d7323e-9efa-4c36-b781-5bc157008ed4\" (UID: \"f1d7323e-9efa-4c36-b781-5bc157008ed4\") " Oct 30 18:10:00.913833 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.913797 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr6lp\" (UniqueName: \"kubernetes.io/projected/db9706fd-5979-4cea-aae8-cffd27090481-kube-api-access-nr6lp\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:10:00.913833 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.913815 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db9706fd-5979-4cea-aae8-cffd27090481-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:10:00.915223 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.915187 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1d7323e-9efa-4c36-b781-5bc157008ed4-plugins-config-volume" pod "f1d7323e-9efa-4c36-b781-5bc157008ed4" (UID: "f1d7323e-9efa-4c36-b781-5bc157008ed4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:10:00.916009 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.915985 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1d7323e-9efa-4c36-b781-5bc157008ed4-kube-api-access-72pkf" pod "f1d7323e-9efa-4c36-b781-5bc157008ed4" (UID: "f1d7323e-9efa-4c36-b781-5bc157008ed4"). InnerVolumeSpecName "kube-api-access-72pkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:10:00.963560 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:00.963214 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-45dc34-79f9556b56-st7qr" Oct 30 18:10:01.014929 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:01.014813 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72pkf\" (UniqueName: \"kubernetes.io/projected/f1d7323e-9efa-4c36-b781-5bc157008ed4-kube-api-access-72pkf\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:10:01.014929 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:01.014853 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1d7323e-9efa-4c36-b781-5bc157008ed4-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:10:01.054520 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:01.054485 4442 generic.go:356] "Generic (PLEG): container finished" podID="db9706fd-5979-4cea-aae8-cffd27090481" containerID="16bbd7b0a48094436b32f6a7e083dcb961842ca93404c167c7ea34f36d6a1280" exitCode=0 Oct 30 18:10:01.054975 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:01.054687 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-57bf5177-7879c68c96-896r4" event={"ID":"db9706fd-5979-4cea-aae8-cffd27090481","Type":"ContainerDied","Data":"16bbd7b0a48094436b32f6a7e083dcb961842ca93404c167c7ea34f36d6a1280"} Oct 30 18:10:01.054975 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:01.054734 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-57bf5177-7879c68c96-896r4" event={"ID":"db9706fd-5979-4cea-aae8-cffd27090481","Type":"ContainerDied","Data":"eb992d3567dece17b8a2280d8c6ced66f65135f5a3b88ccb1e0dbc2c2a08888e"} Oct 30 18:10:01.054975 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:01.054761 4442 scope.go:122] "RemoveContainer" containerID="16bbd7b0a48094436b32f6a7e083dcb961842ca93404c167c7ea34f36d6a1280" Oct 30 18:10:01.054975 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:01.054900 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-57bf5177-7879c68c96-896r4" Oct 30 18:10:01.065155 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:01.061449 4442 generic.go:356] "Generic (PLEG): container finished" podID="f1d7323e-9efa-4c36-b781-5bc157008ed4" containerID="cf1a0005b53231788629b8693364caab26dd596053b63e991d14c1f1bcdc13d3" exitCode=0 Oct 30 18:10:01.065155 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:01.061792 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-57bf5177-5f6f57f4f5-xw6rx" Oct 30 18:10:01.142226 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:01.142189 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12e28d3d-06c0-4e7f-ade8-f8c062d97969" path="/var/lib/kubelet/pods/12e28d3d-06c0-4e7f-ade8-f8c062d97969/volumes" Oct 30 18:10:01.142831 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:01.142805 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-57bf5177-5f6f57f4f5-xw6rx" event={"ID":"f1d7323e-9efa-4c36-b781-5bc157008ed4","Type":"ContainerDied","Data":"cf1a0005b53231788629b8693364caab26dd596053b63e991d14c1f1bcdc13d3"} Oct 30 18:10:01.142930 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:01.142910 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-57bf5177-5f6f57f4f5-xw6rx" event={"ID":"f1d7323e-9efa-4c36-b781-5bc157008ed4","Type":"ContainerDied","Data":"29d822aace56b274a0be2912e85a15dabbf3c20ebab851dc3c97d5033f85b82e"} Oct 30 18:10:01.148505 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:01.148369 4442 scope.go:122] "RemoveContainer" containerID="16bbd7b0a48094436b32f6a7e083dcb961842ca93404c167c7ea34f36d6a1280" Oct 30 18:10:01.149226 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:10:01.149048 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16bbd7b0a48094436b32f6a7e083dcb961842ca93404c167c7ea34f36d6a1280\": container with ID starting with 16bbd7b0a48094436b32f6a7e083dcb961842ca93404c167c7ea34f36d6a1280 not found: ID does not exist" containerID="16bbd7b0a48094436b32f6a7e083dcb961842ca93404c167c7ea34f36d6a1280" Oct 30 18:10:01.149226 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:01.149084 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16bbd7b0a48094436b32f6a7e083dcb961842ca93404c167c7ea34f36d6a1280"} err="failed to get container status \"16bbd7b0a48094436b32f6a7e083dcb961842ca93404c167c7ea34f36d6a1280\": rpc error: code = NotFound desc = could not find container \"16bbd7b0a48094436b32f6a7e083dcb961842ca93404c167c7ea34f36d6a1280\": container with ID starting with 16bbd7b0a48094436b32f6a7e083dcb961842ca93404c167c7ea34f36d6a1280 not found: ID does not exist" Oct 30 18:10:01.149226 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:01.149109 4442 scope.go:122] "RemoveContainer" containerID="cf1a0005b53231788629b8693364caab26dd596053b63e991d14c1f1bcdc13d3" Oct 30 18:10:01.162665 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:01.162641 4442 scope.go:122] "RemoveContainer" containerID="cf1a0005b53231788629b8693364caab26dd596053b63e991d14c1f1bcdc13d3" Oct 30 18:10:01.162981 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:10:01.162957 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf1a0005b53231788629b8693364caab26dd596053b63e991d14c1f1bcdc13d3\": container with ID starting with cf1a0005b53231788629b8693364caab26dd596053b63e991d14c1f1bcdc13d3 not found: ID does not exist" containerID="cf1a0005b53231788629b8693364caab26dd596053b63e991d14c1f1bcdc13d3" Oct 30 18:10:01.163053 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:01.162998 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf1a0005b53231788629b8693364caab26dd596053b63e991d14c1f1bcdc13d3"} err="failed to get container status \"cf1a0005b53231788629b8693364caab26dd596053b63e991d14c1f1bcdc13d3\": rpc error: code = NotFound desc = could not find container \"cf1a0005b53231788629b8693364caab26dd596053b63e991d14c1f1bcdc13d3\": container with ID starting with cf1a0005b53231788629b8693364caab26dd596053b63e991d14c1f1bcdc13d3 not found: ID does not exist" Oct 30 18:10:01.175349 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:01.175313 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-57bf5177-7879c68c96-896r4"] Oct 30 18:10:01.178542 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:01.178516 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-57bf5177-7879c68c96-896r4"] Oct 30 18:10:01.210476 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:01.210447 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-57bf5177-5f6f57f4f5-xw6rx"] Oct 30 18:10:01.213021 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:01.212995 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-57bf5177-5f6f57f4f5-xw6rx"] Oct 30 18:10:01.215743 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:01.215687 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-45dc34-d7f945cff-rqxz2"] Oct 30 18:10:01.225277 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:10:01.225252 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96cf77ba_eb58_4e52_bab4_2a4e362dc95c.slice/crio-c6c7dcbf52efb355859309265f24afd2132105bfac3857ed89c9ce7728f019a2 WatchSource:0}: Error finding container c6c7dcbf52efb355859309265f24afd2132105bfac3857ed89c9ce7728f019a2: Status 404 returned error can't find the container with id c6c7dcbf52efb355859309265f24afd2132105bfac3857ed89c9ce7728f019a2 Oct 30 18:10:01.280006 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:01.279973 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-45dc34-79f9556b56-st7qr"] Oct 30 18:10:01.287312 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:10:01.287281 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod269e36c9_21e8_412e_b49e_e446b7d57def.slice/crio-24bed1eb47bb4996d82c11a955960da28c84491b267b843ffb486148ad258eeb WatchSource:0}: Error finding container 24bed1eb47bb4996d82c11a955960da28c84491b267b843ffb486148ad258eeb: Status 404 returned error can't find the container with id 24bed1eb47bb4996d82c11a955960da28c84491b267b843ffb486148ad258eeb Oct 30 18:10:01.431356 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:01.431314 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ba56e909-6f5476fdc4-vmz9q"] Oct 30 18:10:01.469204 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:01.469115 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ba56e909-c84888fc5-mm82n"] Oct 30 18:10:02.077698 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:02.077659 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-45dc34-79f9556b56-st7qr" event={"ID":"269e36c9-21e8-412e-b49e-e446b7d57def","Type":"ContainerStarted","Data":"a5556d595f6870a40a95db09538a26423c8457392bad43c25bd378fa6f31d6fc"} Oct 30 18:10:02.078569 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:02.077768 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-45dc34-79f9556b56-st7qr" Oct 30 18:10:02.078569 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:02.077797 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-45dc34-79f9556b56-st7qr" event={"ID":"269e36c9-21e8-412e-b49e-e446b7d57def","Type":"ContainerStarted","Data":"24bed1eb47bb4996d82c11a955960da28c84491b267b843ffb486148ad258eeb"} Oct 30 18:10:02.081810 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:02.081779 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-45dc34-d7f945cff-rqxz2" event={"ID":"96cf77ba-eb58-4e52-bab4-2a4e362dc95c","Type":"ContainerStarted","Data":"b7c2a09ef070bb42908aea672d1b7ce7404283fe14a3f92baf2f8456ea0580ac"} Oct 30 18:10:02.081962 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:02.081834 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-45dc34-d7f945cff-rqxz2" Oct 30 18:10:02.081962 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:02.081853 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-45dc34-d7f945cff-rqxz2" event={"ID":"96cf77ba-eb58-4e52-bab4-2a4e362dc95c","Type":"ContainerStarted","Data":"c6c7dcbf52efb355859309265f24afd2132105bfac3857ed89c9ce7728f019a2"} Oct 30 18:10:02.162428 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:02.162322 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-45dc34-d7f945cff-rqxz2" podStartSLOduration=2.162303983 podStartE2EDuration="2.162303983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:02.161318829 +0000 UTC m=+699.251381169" watchObservedRunningTime="2026-10-30 18:10:02.162303983 +0000 UTC m=+699.252366324" Oct 30 18:10:02.164372 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:02.164324 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-45dc34-79f9556b56-st7qr" podStartSLOduration=2.164311524 podStartE2EDuration="2.164311524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:02.123853991 +0000 UTC m=+699.213916333" watchObservedRunningTime="2026-10-30 18:10:02.164311524 +0000 UTC m=+699.254373866" Oct 30 18:10:02.203918 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:02.203879 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-54599e13-69cdd8dfb-fnppd"] Oct 30 18:10:02.238726 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:02.238688 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-54599e13-9fdd4c687-9dncf"] Oct 30 18:10:02.955569 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:02.955532 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-921450-7cc76dbcd5-vwjkc"] Oct 30 18:10:02.956303 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:02.956274 4442 cpu_manager.go:414] "containerMap: removing container" podUID="db9706fd-5979-4cea-aae8-cffd27090481" containerName="payload-processing" Oct 30 18:10:02.956303 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:02.956302 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db9706fd-5979-4cea-aae8-cffd27090481" containerName="payload-processing" Oct 30 18:10:02.956532 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:02.956341 4442 cpu_manager.go:414] "containerMap: removing container" podUID="f1d7323e-9efa-4c36-b781-5bc157008ed4" containerName="payload-pre-processing" Oct 30 18:10:02.956532 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:02.956349 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1d7323e-9efa-4c36-b781-5bc157008ed4" containerName="payload-pre-processing" Oct 30 18:10:02.956532 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:02.956490 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="db9706fd-5979-4cea-aae8-cffd27090481" containerName="payload-processing" Oct 30 18:10:02.956532 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:02.956508 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1d7323e-9efa-4c36-b781-5bc157008ed4" containerName="payload-pre-processing" Oct 30 18:10:02.960006 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:02.959983 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-921450-7cc76dbcd5-vwjkc" Oct 30 18:10:02.972871 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:02.972845 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bc83ed4-302b-466c-9599-b022b3f0afb1-maas-api-tls\") pod \"maas-api-e2e-worker-w0-921450-7cc76dbcd5-vwjkc\" (UID: \"6bc83ed4-302b-466c-9599-b022b3f0afb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-921450-7cc76dbcd5-vwjkc" Oct 30 18:10:02.973013 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:02.972881 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j2k2\" (UniqueName: \"kubernetes.io/projected/6bc83ed4-302b-466c-9599-b022b3f0afb1-kube-api-access-6j2k2\") pod \"maas-api-e2e-worker-w0-921450-7cc76dbcd5-vwjkc\" (UID: \"6bc83ed4-302b-466c-9599-b022b3f0afb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-921450-7cc76dbcd5-vwjkc" Oct 30 18:10:02.973013 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:02.972928 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bc83ed4-302b-466c-9599-b022b3f0afb1-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-921450-7cc76dbcd5-vwjkc\" (UID: \"6bc83ed4-302b-466c-9599-b022b3f0afb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-921450-7cc76dbcd5-vwjkc" Oct 30 18:10:02.973187 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:02.973055 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bc83ed4-302b-466c-9599-b022b3f0afb1-metrics-certs\") pod \"maas-api-e2e-worker-w0-921450-7cc76dbcd5-vwjkc\" (UID: \"6bc83ed4-302b-466c-9599-b022b3f0afb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-921450-7cc76dbcd5-vwjkc" Oct 30 18:10:02.991251 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:02.990907 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-921450-serving-cert\"" Oct 30 18:10:02.994205 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:02.994179 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-921450-7cc76dbcd5-vwjkc"] Oct 30 18:10:03.063701 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:03.063665 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db9706fd-5979-4cea-aae8-cffd27090481" path="/var/lib/kubelet/pods/db9706fd-5979-4cea-aae8-cffd27090481/volumes" Oct 30 18:10:03.064047 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:03.064034 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1d7323e-9efa-4c36-b781-5bc157008ed4" path="/var/lib/kubelet/pods/f1d7323e-9efa-4c36-b781-5bc157008ed4/volumes" Oct 30 18:10:03.073739 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:03.073715 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bc83ed4-302b-466c-9599-b022b3f0afb1-maas-api-tls\") pod \"maas-api-e2e-worker-w0-921450-7cc76dbcd5-vwjkc\" (UID: \"6bc83ed4-302b-466c-9599-b022b3f0afb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-921450-7cc76dbcd5-vwjkc" Oct 30 18:10:03.073837 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:03.073748 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j2k2\" (UniqueName: \"kubernetes.io/projected/6bc83ed4-302b-466c-9599-b022b3f0afb1-kube-api-access-6j2k2\") pod \"maas-api-e2e-worker-w0-921450-7cc76dbcd5-vwjkc\" (UID: \"6bc83ed4-302b-466c-9599-b022b3f0afb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-921450-7cc76dbcd5-vwjkc" Oct 30 18:10:03.073837 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:03.073783 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bc83ed4-302b-466c-9599-b022b3f0afb1-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-921450-7cc76dbcd5-vwjkc\" (UID: \"6bc83ed4-302b-466c-9599-b022b3f0afb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-921450-7cc76dbcd5-vwjkc" Oct 30 18:10:03.073837 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:03.073801 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bc83ed4-302b-466c-9599-b022b3f0afb1-metrics-certs\") pod \"maas-api-e2e-worker-w0-921450-7cc76dbcd5-vwjkc\" (UID: \"6bc83ed4-302b-466c-9599-b022b3f0afb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-921450-7cc76dbcd5-vwjkc" Oct 30 18:10:03.073936 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:10:03.073840 4442 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-921450-serving-cert: secret "maas-api-e2e-worker-w0-921450-serving-cert" not found Oct 30 18:10:03.073936 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:10:03.073912 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bc83ed4-302b-466c-9599-b022b3f0afb1-maas-api-tls podName:6bc83ed4-302b-466c-9599-b022b3f0afb1 nodeName:}" failed. No retries permitted until 2026-10-30 18:10:03.573891306 +0000 UTC m=+700.663953643 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6bc83ed4-302b-466c-9599-b022b3f0afb1-maas-api-tls") pod "maas-api-e2e-worker-w0-921450-7cc76dbcd5-vwjkc" (UID: "6bc83ed4-302b-466c-9599-b022b3f0afb1") : secret "maas-api-e2e-worker-w0-921450-serving-cert" not found Oct 30 18:10:03.074497 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:03.074474 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bc83ed4-302b-466c-9599-b022b3f0afb1-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-921450-7cc76dbcd5-vwjkc\" (UID: \"6bc83ed4-302b-466c-9599-b022b3f0afb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-921450-7cc76dbcd5-vwjkc" Oct 30 18:10:03.076549 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:03.076527 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bc83ed4-302b-466c-9599-b022b3f0afb1-metrics-certs\") pod \"maas-api-e2e-worker-w0-921450-7cc76dbcd5-vwjkc\" (UID: \"6bc83ed4-302b-466c-9599-b022b3f0afb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-921450-7cc76dbcd5-vwjkc" Oct 30 18:10:03.096036 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:03.096002 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j2k2\" (UniqueName: \"kubernetes.io/projected/6bc83ed4-302b-466c-9599-b022b3f0afb1-kube-api-access-6j2k2\") pod \"maas-api-e2e-worker-w0-921450-7cc76dbcd5-vwjkc\" (UID: \"6bc83ed4-302b-466c-9599-b022b3f0afb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-921450-7cc76dbcd5-vwjkc" Oct 30 18:10:03.578419 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:03.578383 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bc83ed4-302b-466c-9599-b022b3f0afb1-maas-api-tls\") pod \"maas-api-e2e-worker-w0-921450-7cc76dbcd5-vwjkc\" (UID: \"6bc83ed4-302b-466c-9599-b022b3f0afb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-921450-7cc76dbcd5-vwjkc" Oct 30 18:10:03.581228 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:03.581197 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bc83ed4-302b-466c-9599-b022b3f0afb1-maas-api-tls\") pod \"maas-api-e2e-worker-w0-921450-7cc76dbcd5-vwjkc\" (UID: \"6bc83ed4-302b-466c-9599-b022b3f0afb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-921450-7cc76dbcd5-vwjkc" Oct 30 18:10:03.873879 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:03.873842 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-921450-7cc76dbcd5-vwjkc" Oct 30 18:10:04.211886 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:04.211840 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-921450-7cc76dbcd5-vwjkc"] Oct 30 18:10:04.221769 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:10:04.220493 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bc83ed4_302b_466c_9599_b022b3f0afb1.slice/crio-fe05c01fd517ebdaaa02280655346eb447646cb1cb99d73a213eee0bca75a3f4 WatchSource:0}: Error finding container fe05c01fd517ebdaaa02280655346eb447646cb1cb99d73a213eee0bca75a3f4: Status 404 returned error can't find the container with id fe05c01fd517ebdaaa02280655346eb447646cb1cb99d73a213eee0bca75a3f4 Oct 30 18:10:04.710020 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:04.709985 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm"] Oct 30 18:10:04.710276 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:04.710250 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm" podUID="fa665c80-1db7-41b2-b3f7-5259248ccc05" containerName="maas-api" containerID="cri-o://303ac54d76f25b3b09ec2379eead8731efa5d59272b564793cb504082055c699" gracePeriod=30 Oct 30 18:10:05.010343 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:05.010316 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm" Oct 30 18:10:05.098255 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:05.098161 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa665c80-1db7-41b2-b3f7-5259248ccc05-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa665c80-1db7-41b2-b3f7-5259248ccc05-metrics-certs\") pod \"fa665c80-1db7-41b2-b3f7-5259248ccc05\" (UID: \"fa665c80-1db7-41b2-b3f7-5259248ccc05\") " Oct 30 18:10:05.098421 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:05.098292 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa665c80-1db7-41b2-b3f7-5259248ccc05-kube-api-access-mbvk9\" (UniqueName: \"kubernetes.io/projected/fa665c80-1db7-41b2-b3f7-5259248ccc05-kube-api-access-mbvk9\") pod \"fa665c80-1db7-41b2-b3f7-5259248ccc05\" (UID: \"fa665c80-1db7-41b2-b3f7-5259248ccc05\") " Oct 30 18:10:05.098421 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:05.098345 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa665c80-1db7-41b2-b3f7-5259248ccc05-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa665c80-1db7-41b2-b3f7-5259248ccc05-maas-api-tls\") pod \"fa665c80-1db7-41b2-b3f7-5259248ccc05\" (UID: \"fa665c80-1db7-41b2-b3f7-5259248ccc05\") " Oct 30 18:10:05.098421 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:05.098376 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa665c80-1db7-41b2-b3f7-5259248ccc05-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa665c80-1db7-41b2-b3f7-5259248ccc05-openshift-service-ca\") pod \"fa665c80-1db7-41b2-b3f7-5259248ccc05\" (UID: \"fa665c80-1db7-41b2-b3f7-5259248ccc05\") " Oct 30 18:10:05.098819 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:05.098787 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa665c80-1db7-41b2-b3f7-5259248ccc05-openshift-service-ca" pod "fa665c80-1db7-41b2-b3f7-5259248ccc05" (UID: "fa665c80-1db7-41b2-b3f7-5259248ccc05"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:10:05.100636 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:05.100608 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa665c80-1db7-41b2-b3f7-5259248ccc05-metrics-certs" pod "fa665c80-1db7-41b2-b3f7-5259248ccc05" (UID: "fa665c80-1db7-41b2-b3f7-5259248ccc05"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:10:05.100743 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:05.100617 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa665c80-1db7-41b2-b3f7-5259248ccc05-kube-api-access-mbvk9" pod "fa665c80-1db7-41b2-b3f7-5259248ccc05" (UID: "fa665c80-1db7-41b2-b3f7-5259248ccc05"). InnerVolumeSpecName "kube-api-access-mbvk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:10:05.100743 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:05.100640 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa665c80-1db7-41b2-b3f7-5259248ccc05-maas-api-tls" pod "fa665c80-1db7-41b2-b3f7-5259248ccc05" (UID: "fa665c80-1db7-41b2-b3f7-5259248ccc05"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:10:05.105398 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:05.105372 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-921450-7cc76dbcd5-vwjkc" event={"ID":"6bc83ed4-302b-466c-9599-b022b3f0afb1","Type":"ContainerStarted","Data":"e439cf17531bbf157d05ce8d4d2a1e30ea03799edb4583965f6cf29acea39b5c"} Oct 30 18:10:05.105524 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:05.105429 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-921450-7cc76dbcd5-vwjkc" event={"ID":"6bc83ed4-302b-466c-9599-b022b3f0afb1","Type":"ContainerStarted","Data":"fe05c01fd517ebdaaa02280655346eb447646cb1cb99d73a213eee0bca75a3f4"} Oct 30 18:10:05.105587 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:05.105540 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-921450-7cc76dbcd5-vwjkc" Oct 30 18:10:05.106894 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:05.106872 4442 generic.go:356] "Generic (PLEG): container finished" podID="fa665c80-1db7-41b2-b3f7-5259248ccc05" containerID="303ac54d76f25b3b09ec2379eead8731efa5d59272b564793cb504082055c699" exitCode=0 Oct 30 18:10:05.106996 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:05.106918 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm" Oct 30 18:10:05.106996 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:05.106957 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm" event={"ID":"fa665c80-1db7-41b2-b3f7-5259248ccc05","Type":"ContainerDied","Data":"303ac54d76f25b3b09ec2379eead8731efa5d59272b564793cb504082055c699"} Oct 30 18:10:05.107111 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:05.107003 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm" event={"ID":"fa665c80-1db7-41b2-b3f7-5259248ccc05","Type":"ContainerDied","Data":"737fe68196f681fb9a1da462a15a5bb0cabe9f634e9cdafb579b6e981a18bc21"} Oct 30 18:10:05.107111 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:05.107027 4442 scope.go:122] "RemoveContainer" containerID="303ac54d76f25b3b09ec2379eead8731efa5d59272b564793cb504082055c699" Oct 30 18:10:05.117636 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:05.117617 4442 scope.go:122] "RemoveContainer" containerID="303ac54d76f25b3b09ec2379eead8731efa5d59272b564793cb504082055c699" Oct 30 18:10:05.117968 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:10:05.117950 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"303ac54d76f25b3b09ec2379eead8731efa5d59272b564793cb504082055c699\": container with ID starting with 303ac54d76f25b3b09ec2379eead8731efa5d59272b564793cb504082055c699 not found: ID does not exist" containerID="303ac54d76f25b3b09ec2379eead8731efa5d59272b564793cb504082055c699" Oct 30 18:10:05.118028 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:05.117979 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"303ac54d76f25b3b09ec2379eead8731efa5d59272b564793cb504082055c699"} err="failed to get container status \"303ac54d76f25b3b09ec2379eead8731efa5d59272b564793cb504082055c699\": rpc error: code = NotFound desc = could not find container \"303ac54d76f25b3b09ec2379eead8731efa5d59272b564793cb504082055c699\": container with ID starting with 303ac54d76f25b3b09ec2379eead8731efa5d59272b564793cb504082055c699 not found: ID does not exist" Oct 30 18:10:05.146517 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:05.146448 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-921450-7cc76dbcd5-vwjkc" podStartSLOduration=2.8440336840000002 podStartE2EDuration="3.146422183s" totalImagesPullingTime="302.388499ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:05.144871506 +0000 UTC m=+702.234933845" watchObservedRunningTime="2026-10-30 18:10:05.146422183 +0000 UTC m=+702.236484523" Oct 30 18:10:05.183448 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:05.183376 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm"] Oct 30 18:10:05.187430 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:05.187402 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ba56e909-7f8cf4cbf4-fjdgm"] Oct 30 18:10:05.200183 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:05.200116 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa665c80-1db7-41b2-b3f7-5259248ccc05-metrics-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:10:05.200183 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:05.200177 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbvk9\" (UniqueName: \"kubernetes.io/projected/fa665c80-1db7-41b2-b3f7-5259248ccc05-kube-api-access-mbvk9\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:10:05.200440 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:05.200197 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa665c80-1db7-41b2-b3f7-5259248ccc05-maas-api-tls\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:10:05.200440 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:05.200214 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa665c80-1db7-41b2-b3f7-5259248ccc05-openshift-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:10:05.478881 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:05.478812 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-45dc34-6cc4b7467c-8fgqt"] Oct 30 18:10:05.480320 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:05.480292 4442 cpu_manager.go:414] "containerMap: removing container" podUID="fa665c80-1db7-41b2-b3f7-5259248ccc05" containerName="maas-api" Oct 30 18:10:05.480320 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:05.480320 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa665c80-1db7-41b2-b3f7-5259248ccc05" containerName="maas-api" Oct 30 18:10:05.480659 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:05.480644 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa665c80-1db7-41b2-b3f7-5259248ccc05" containerName="maas-api" Oct 30 18:10:05.521539 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:05.521477 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-45dc34-6cc4b7467c-8fgqt"] Oct 30 18:10:05.521723 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:05.521616 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-45dc34-6cc4b7467c-8fgqt" Oct 30 18:10:05.535988 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:05.535948 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-45dc34-serving-cert\"" Oct 30 18:10:05.706611 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:05.706576 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d41d70-3283-4135-9f4e-3ff0c53cec22-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-45dc34-6cc4b7467c-8fgqt\" (UID: \"13d41d70-3283-4135-9f4e-3ff0c53cec22\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-45dc34-6cc4b7467c-8fgqt" Oct 30 18:10:05.706796 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:05.706659 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d41d70-3283-4135-9f4e-3ff0c53cec22-metrics-certs\") pod \"maas-api-e2e-worker-w2-45dc34-6cc4b7467c-8fgqt\" (UID: \"13d41d70-3283-4135-9f4e-3ff0c53cec22\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-45dc34-6cc4b7467c-8fgqt" Oct 30 18:10:05.706843 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:05.706795 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgtww\" (UniqueName: \"kubernetes.io/projected/13d41d70-3283-4135-9f4e-3ff0c53cec22-kube-api-access-kgtww\") pod \"maas-api-e2e-worker-w2-45dc34-6cc4b7467c-8fgqt\" (UID: \"13d41d70-3283-4135-9f4e-3ff0c53cec22\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-45dc34-6cc4b7467c-8fgqt" Oct 30 18:10:05.706843 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:05.706832 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13d41d70-3283-4135-9f4e-3ff0c53cec22-maas-api-tls\") pod \"maas-api-e2e-worker-w2-45dc34-6cc4b7467c-8fgqt\" (UID: \"13d41d70-3283-4135-9f4e-3ff0c53cec22\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-45dc34-6cc4b7467c-8fgqt" Oct 30 18:10:05.811931 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:05.811800 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgtww\" (UniqueName: \"kubernetes.io/projected/13d41d70-3283-4135-9f4e-3ff0c53cec22-kube-api-access-kgtww\") pod \"maas-api-e2e-worker-w2-45dc34-6cc4b7467c-8fgqt\" (UID: \"13d41d70-3283-4135-9f4e-3ff0c53cec22\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-45dc34-6cc4b7467c-8fgqt" Oct 30 18:10:05.811931 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:05.811863 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13d41d70-3283-4135-9f4e-3ff0c53cec22-maas-api-tls\") pod \"maas-api-e2e-worker-w2-45dc34-6cc4b7467c-8fgqt\" (UID: \"13d41d70-3283-4135-9f4e-3ff0c53cec22\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-45dc34-6cc4b7467c-8fgqt" Oct 30 18:10:05.811931 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:05.811906 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d41d70-3283-4135-9f4e-3ff0c53cec22-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-45dc34-6cc4b7467c-8fgqt\" (UID: \"13d41d70-3283-4135-9f4e-3ff0c53cec22\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-45dc34-6cc4b7467c-8fgqt" Oct 30 18:10:05.812295 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:05.812017 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d41d70-3283-4135-9f4e-3ff0c53cec22-metrics-certs\") pod \"maas-api-e2e-worker-w2-45dc34-6cc4b7467c-8fgqt\" (UID: \"13d41d70-3283-4135-9f4e-3ff0c53cec22\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-45dc34-6cc4b7467c-8fgqt" Oct 30 18:10:05.813644 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:05.813613 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d41d70-3283-4135-9f4e-3ff0c53cec22-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-45dc34-6cc4b7467c-8fgqt\" (UID: \"13d41d70-3283-4135-9f4e-3ff0c53cec22\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-45dc34-6cc4b7467c-8fgqt" Oct 30 18:10:05.815440 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:05.815413 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d41d70-3283-4135-9f4e-3ff0c53cec22-metrics-certs\") pod \"maas-api-e2e-worker-w2-45dc34-6cc4b7467c-8fgqt\" (UID: \"13d41d70-3283-4135-9f4e-3ff0c53cec22\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-45dc34-6cc4b7467c-8fgqt" Oct 30 18:10:05.815570 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:05.815447 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13d41d70-3283-4135-9f4e-3ff0c53cec22-maas-api-tls\") pod \"maas-api-e2e-worker-w2-45dc34-6cc4b7467c-8fgqt\" (UID: \"13d41d70-3283-4135-9f4e-3ff0c53cec22\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-45dc34-6cc4b7467c-8fgqt" Oct 30 18:10:05.832471 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:05.832434 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgtww\" (UniqueName: \"kubernetes.io/projected/13d41d70-3283-4135-9f4e-3ff0c53cec22-kube-api-access-kgtww\") pod \"maas-api-e2e-worker-w2-45dc34-6cc4b7467c-8fgqt\" (UID: \"13d41d70-3283-4135-9f4e-3ff0c53cec22\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-45dc34-6cc4b7467c-8fgqt" Oct 30 18:10:05.840237 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:05.840190 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-45dc34-6cc4b7467c-8fgqt" Oct 30 18:10:06.145538 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:06.145503 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-45dc34-6cc4b7467c-8fgqt"] Oct 30 18:10:06.149666 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:10:06.149637 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13d41d70_3283_4135_9f4e_3ff0c53cec22.slice/crio-96cc79e782238181c753d0e5b7498121979219d235c9f98eb79b81c435cbc1ff WatchSource:0}: Error finding container 96cc79e782238181c753d0e5b7498121979219d235c9f98eb79b81c435cbc1ff: Status 404 returned error can't find the container with id 96cc79e782238181c753d0e5b7498121979219d235c9f98eb79b81c435cbc1ff Oct 30 18:10:06.431958 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:06.431859 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-ba56e909-6f5476fdc4-vmz9q" podUID="383feced-9ea0-4524-a76e-84112d54fa10" containerName="payload-processing" containerID="cri-o://4e47fcd638b51d08685ad7f6736e10850b4899d90eb62138cef21a1beb7201d9" gracePeriod=25 Oct 30 18:10:06.469666 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:06.469631 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-ba56e909-c84888fc5-mm82n" podUID="b8edb1cd-d37b-4dd8-8600-729318034c2a" containerName="payload-pre-processing" containerID="cri-o://27bc1c83d1a22b5e49e3bfe66a08944accef9b477281fd5814f3f5ecbb22f0a2" gracePeriod=25 Oct 30 18:10:06.764316 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:06.764282 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ba56e909-6f5476fdc4-vmz9q" Oct 30 18:10:06.784166 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:06.784121 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ba56e909-c84888fc5-mm82n" Oct 30 18:10:06.866834 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:06.866794 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-de7e3a-6768cfb995-99nqx"] Oct 30 18:10:06.867304 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:06.867287 4442 cpu_manager.go:414] "containerMap: removing container" podUID="b8edb1cd-d37b-4dd8-8600-729318034c2a" containerName="payload-pre-processing" Oct 30 18:10:06.867304 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:06.867302 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8edb1cd-d37b-4dd8-8600-729318034c2a" containerName="payload-pre-processing" Oct 30 18:10:06.867473 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:06.867330 4442 cpu_manager.go:414] "containerMap: removing container" podUID="383feced-9ea0-4524-a76e-84112d54fa10" containerName="payload-processing" Oct 30 18:10:06.867473 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:06.867336 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="383feced-9ea0-4524-a76e-84112d54fa10" containerName="payload-processing" Oct 30 18:10:06.867473 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:06.867402 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="383feced-9ea0-4524-a76e-84112d54fa10" containerName="payload-processing" Oct 30 18:10:06.867473 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:06.867412 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8edb1cd-d37b-4dd8-8600-729318034c2a" containerName="payload-pre-processing" Oct 30 18:10:06.870797 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:06.870774 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-de7e3a-6768cfb995-99nqx" Oct 30 18:10:06.888747 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:06.888704 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-de7e3a\"" Oct 30 18:10:06.888913 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:06.888782 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-de7e3a-dockercfg-6qsgh\"" Oct 30 18:10:06.890770 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:06.890740 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-de7e3a-6768cfb995-99nqx"] Oct 30 18:10:06.924638 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:06.924594 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8edb1cd-d37b-4dd8-8600-729318034c2a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8edb1cd-d37b-4dd8-8600-729318034c2a-plugins-config-volume\") pod \"b8edb1cd-d37b-4dd8-8600-729318034c2a\" (UID: \"b8edb1cd-d37b-4dd8-8600-729318034c2a\") " Oct 30 18:10:06.924777 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:06.924692 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8edb1cd-d37b-4dd8-8600-729318034c2a-kube-api-access-vpkxm\" (UniqueName: \"kubernetes.io/projected/b8edb1cd-d37b-4dd8-8600-729318034c2a-kube-api-access-vpkxm\") pod \"b8edb1cd-d37b-4dd8-8600-729318034c2a\" (UID: \"b8edb1cd-d37b-4dd8-8600-729318034c2a\") " Oct 30 18:10:06.924879 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:06.924862 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/383feced-9ea0-4524-a76e-84112d54fa10-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/383feced-9ea0-4524-a76e-84112d54fa10-plugins-config-volume\") pod \"383feced-9ea0-4524-a76e-84112d54fa10\" (UID: \"383feced-9ea0-4524-a76e-84112d54fa10\") " Oct 30 18:10:06.924913 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:06.924896 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/383feced-9ea0-4524-a76e-84112d54fa10-kube-api-access-fphl8\" (UniqueName: \"kubernetes.io/projected/383feced-9ea0-4524-a76e-84112d54fa10-kube-api-access-fphl8\") pod \"383feced-9ea0-4524-a76e-84112d54fa10\" (UID: \"383feced-9ea0-4524-a76e-84112d54fa10\") " Oct 30 18:10:06.924913 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:06.924899 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8edb1cd-d37b-4dd8-8600-729318034c2a-plugins-config-volume" pod "b8edb1cd-d37b-4dd8-8600-729318034c2a" (UID: "b8edb1cd-d37b-4dd8-8600-729318034c2a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:10:06.925288 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:06.925262 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/383feced-9ea0-4524-a76e-84112d54fa10-plugins-config-volume" pod "383feced-9ea0-4524-a76e-84112d54fa10" (UID: "383feced-9ea0-4524-a76e-84112d54fa10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:10:06.925497 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:06.925453 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8edb1cd-d37b-4dd8-8600-729318034c2a-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:10:06.925497 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:06.925474 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/383feced-9ea0-4524-a76e-84112d54fa10-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:10:06.927079 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:06.927053 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8edb1cd-d37b-4dd8-8600-729318034c2a-kube-api-access-vpkxm" pod "b8edb1cd-d37b-4dd8-8600-729318034c2a" (UID: "b8edb1cd-d37b-4dd8-8600-729318034c2a"). InnerVolumeSpecName "kube-api-access-vpkxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:10:06.927564 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:06.927535 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/383feced-9ea0-4524-a76e-84112d54fa10-kube-api-access-fphl8" pod "383feced-9ea0-4524-a76e-84112d54fa10" (UID: "383feced-9ea0-4524-a76e-84112d54fa10"). InnerVolumeSpecName "kube-api-access-fphl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:10:07.016846 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.016759 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-de7e3a-7b788bbfd6-mlkk9"] Oct 30 18:10:07.021334 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.021311 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-de7e3a-7b788bbfd6-mlkk9" Oct 30 18:10:07.026855 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.026825 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgsk9\" (UniqueName: \"kubernetes.io/projected/c4f54adc-5fd4-45b0-acb5-761613564059-kube-api-access-lgsk9\") pod \"payload-pre-processing-e2e-worker-w3-de7e3a-6768cfb995-99nqx\" (UID: \"c4f54adc-5fd4-45b0-acb5-761613564059\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-de7e3a-6768cfb995-99nqx" Oct 30 18:10:07.026959 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.026930 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4f54adc-5fd4-45b0-acb5-761613564059-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-de7e3a-6768cfb995-99nqx\" (UID: \"c4f54adc-5fd4-45b0-acb5-761613564059\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-de7e3a-6768cfb995-99nqx" Oct 30 18:10:07.027030 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.027018 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpkxm\" (UniqueName: \"kubernetes.io/projected/b8edb1cd-d37b-4dd8-8600-729318034c2a-kube-api-access-vpkxm\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:10:07.027078 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.027035 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fphl8\" (UniqueName: \"kubernetes.io/projected/383feced-9ea0-4524-a76e-84112d54fa10-kube-api-access-fphl8\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:10:07.034109 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.034080 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-de7e3a-7b788bbfd6-mlkk9"] Oct 30 18:10:07.065355 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.065325 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa665c80-1db7-41b2-b3f7-5259248ccc05" path="/var/lib/kubelet/pods/fa665c80-1db7-41b2-b3f7-5259248ccc05/volumes" Oct 30 18:10:07.118843 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.118812 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-54599e13-84bddc8b55-8hngq"] Oct 30 18:10:07.119029 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.119006 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54599e13-84bddc8b55-8hngq" podUID="a782e223-3075-4fb8-b38c-1c009b65de6e" containerName="maas-api" containerID="cri-o://5c9c84fb3cdde6d399216a670d9d50d51d904f6f1183dd0df74435945ad22a04" gracePeriod=30 Oct 30 18:10:07.121872 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.121844 4442 generic.go:356] "Generic (PLEG): container finished" podID="383feced-9ea0-4524-a76e-84112d54fa10" containerID="4e47fcd638b51d08685ad7f6736e10850b4899d90eb62138cef21a1beb7201d9" exitCode=0 Oct 30 18:10:07.122038 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.121976 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ba56e909-6f5476fdc4-vmz9q" event={"ID":"383feced-9ea0-4524-a76e-84112d54fa10","Type":"ContainerDied","Data":"4e47fcd638b51d08685ad7f6736e10850b4899d90eb62138cef21a1beb7201d9"} Oct 30 18:10:07.122038 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.122021 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ba56e909-6f5476fdc4-vmz9q" event={"ID":"383feced-9ea0-4524-a76e-84112d54fa10","Type":"ContainerDied","Data":"813d722277a6454f3413041b8f669deccec73c3638c1e403fd701955b3a14bb5"} Oct 30 18:10:07.122229 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.122064 4442 scope.go:122] "RemoveContainer" containerID="4e47fcd638b51d08685ad7f6736e10850b4899d90eb62138cef21a1beb7201d9" Oct 30 18:10:07.124750 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.124730 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ba56e909-6f5476fdc4-vmz9q" Oct 30 18:10:07.127936 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.127604 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc4a9a79-c6c6-4660-ac88-16920ac96298-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-de7e3a-7b788bbfd6-mlkk9\" (UID: \"fc4a9a79-c6c6-4660-ac88-16920ac96298\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-de7e3a-7b788bbfd6-mlkk9" Oct 30 18:10:07.127936 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.127641 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgsk9\" (UniqueName: \"kubernetes.io/projected/c4f54adc-5fd4-45b0-acb5-761613564059-kube-api-access-lgsk9\") pod \"payload-pre-processing-e2e-worker-w3-de7e3a-6768cfb995-99nqx\" (UID: \"c4f54adc-5fd4-45b0-acb5-761613564059\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-de7e3a-6768cfb995-99nqx" Oct 30 18:10:07.127936 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.127785 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4f54adc-5fd4-45b0-acb5-761613564059-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-de7e3a-6768cfb995-99nqx\" (UID: \"c4f54adc-5fd4-45b0-acb5-761613564059\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-de7e3a-6768cfb995-99nqx" Oct 30 18:10:07.127936 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.127843 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5gk5\" (UniqueName: \"kubernetes.io/projected/fc4a9a79-c6c6-4660-ac88-16920ac96298-kube-api-access-r5gk5\") pod \"payload-processing-e2e-worker-w3-de7e3a-7b788bbfd6-mlkk9\" (UID: \"fc4a9a79-c6c6-4660-ac88-16920ac96298\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-de7e3a-7b788bbfd6-mlkk9" Oct 30 18:10:07.128960 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.128935 4442 generic.go:356] "Generic (PLEG): container finished" podID="b8edb1cd-d37b-4dd8-8600-729318034c2a" containerID="27bc1c83d1a22b5e49e3bfe66a08944accef9b477281fd5814f3f5ecbb22f0a2" exitCode=0 Oct 30 18:10:07.129186 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.129161 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ba56e909-c84888fc5-mm82n" event={"ID":"b8edb1cd-d37b-4dd8-8600-729318034c2a","Type":"ContainerDied","Data":"27bc1c83d1a22b5e49e3bfe66a08944accef9b477281fd5814f3f5ecbb22f0a2"} Oct 30 18:10:07.129303 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.129187 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4f54adc-5fd4-45b0-acb5-761613564059-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-de7e3a-6768cfb995-99nqx\" (UID: \"c4f54adc-5fd4-45b0-acb5-761613564059\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-de7e3a-6768cfb995-99nqx" Oct 30 18:10:07.129303 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.129213 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ba56e909-c84888fc5-mm82n" event={"ID":"b8edb1cd-d37b-4dd8-8600-729318034c2a","Type":"ContainerDied","Data":"16748c27a9248eae0319ef031db9f65d9b8f0a3059802506f7d587b056033b52"} Oct 30 18:10:07.129674 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.129395 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ba56e909-c84888fc5-mm82n" Oct 30 18:10:07.135304 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.135277 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-45dc34-6cc4b7467c-8fgqt" event={"ID":"13d41d70-3283-4135-9f4e-3ff0c53cec22","Type":"ContainerStarted","Data":"23ccb882d0e307887df5c75c094b0293b215e86956e50fb35da1a3ac28b78d40"} Oct 30 18:10:07.135421 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.135340 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-45dc34-6cc4b7467c-8fgqt" Oct 30 18:10:07.135421 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.135365 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-45dc34-6cc4b7467c-8fgqt" event={"ID":"13d41d70-3283-4135-9f4e-3ff0c53cec22","Type":"ContainerStarted","Data":"96cc79e782238181c753d0e5b7498121979219d235c9f98eb79b81c435cbc1ff"} Oct 30 18:10:07.162240 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.158465 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgsk9\" (UniqueName: \"kubernetes.io/projected/c4f54adc-5fd4-45b0-acb5-761613564059-kube-api-access-lgsk9\") pod \"payload-pre-processing-e2e-worker-w3-de7e3a-6768cfb995-99nqx\" (UID: \"c4f54adc-5fd4-45b0-acb5-761613564059\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-de7e3a-6768cfb995-99nqx" Oct 30 18:10:07.185229 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.185059 4442 scope.go:122] "RemoveContainer" containerID="4e47fcd638b51d08685ad7f6736e10850b4899d90eb62138cef21a1beb7201d9" Oct 30 18:10:07.185483 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:10:07.185461 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e47fcd638b51d08685ad7f6736e10850b4899d90eb62138cef21a1beb7201d9\": container with ID starting with 4e47fcd638b51d08685ad7f6736e10850b4899d90eb62138cef21a1beb7201d9 not found: ID does not exist" containerID="4e47fcd638b51d08685ad7f6736e10850b4899d90eb62138cef21a1beb7201d9" Oct 30 18:10:07.185564 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.185498 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e47fcd638b51d08685ad7f6736e10850b4899d90eb62138cef21a1beb7201d9"} err="failed to get container status \"4e47fcd638b51d08685ad7f6736e10850b4899d90eb62138cef21a1beb7201d9\": rpc error: code = NotFound desc = could not find container \"4e47fcd638b51d08685ad7f6736e10850b4899d90eb62138cef21a1beb7201d9\": container with ID starting with 4e47fcd638b51d08685ad7f6736e10850b4899d90eb62138cef21a1beb7201d9 not found: ID does not exist" Oct 30 18:10:07.185564 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.185522 4442 scope.go:122] "RemoveContainer" containerID="27bc1c83d1a22b5e49e3bfe66a08944accef9b477281fd5814f3f5ecbb22f0a2" Oct 30 18:10:07.187721 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.187705 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-de7e3a-6768cfb995-99nqx" Oct 30 18:10:07.204154 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.204107 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-54599e13-69cdd8dfb-fnppd" podUID="13cfe51c-22d4-49f0-827b-b10bac35f0c1" containerName="payload-processing" containerID="cri-o://bd4ef26a937e95fb0d094202344d83c2234f013ce5a4207b9e7513421ecb5ffd" gracePeriod=25 Oct 30 18:10:07.233695 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.231836 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5gk5\" (UniqueName: \"kubernetes.io/projected/fc4a9a79-c6c6-4660-ac88-16920ac96298-kube-api-access-r5gk5\") pod \"payload-processing-e2e-worker-w3-de7e3a-7b788bbfd6-mlkk9\" (UID: \"fc4a9a79-c6c6-4660-ac88-16920ac96298\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-de7e3a-7b788bbfd6-mlkk9" Oct 30 18:10:07.233695 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.231937 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc4a9a79-c6c6-4660-ac88-16920ac96298-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-de7e3a-7b788bbfd6-mlkk9\" (UID: \"fc4a9a79-c6c6-4660-ac88-16920ac96298\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-de7e3a-7b788bbfd6-mlkk9" Oct 30 18:10:07.234479 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.234452 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc4a9a79-c6c6-4660-ac88-16920ac96298-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-de7e3a-7b788bbfd6-mlkk9\" (UID: \"fc4a9a79-c6c6-4660-ac88-16920ac96298\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-de7e3a-7b788bbfd6-mlkk9" Oct 30 18:10:07.239008 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.238977 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-54599e13-9fdd4c687-9dncf" podUID="d8bcefce-471d-44ab-93c7-e5917c7e6231" containerName="payload-pre-processing" containerID="cri-o://2db9bc43604c2819fb2a969160d82fa7d44ad5ef7234b1b89a838506d7f36a4c" gracePeriod=25 Oct 30 18:10:07.256642 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.256608 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5gk5\" (UniqueName: \"kubernetes.io/projected/fc4a9a79-c6c6-4660-ac88-16920ac96298-kube-api-access-r5gk5\") pod \"payload-processing-e2e-worker-w3-de7e3a-7b788bbfd6-mlkk9\" (UID: \"fc4a9a79-c6c6-4660-ac88-16920ac96298\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-de7e3a-7b788bbfd6-mlkk9" Oct 30 18:10:07.266996 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.266901 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-45dc34-6cc4b7467c-8fgqt" podStartSLOduration=1.930068538 podStartE2EDuration="2.266882087s" totalImagesPullingTime="336.813549ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:07.233952115 +0000 UTC m=+704.324014455" watchObservedRunningTime="2026-10-30 18:10:07.266882087 +0000 UTC m=+704.356944428" Oct 30 18:10:07.275288 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.275257 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ba56e909-c84888fc5-mm82n"] Oct 30 18:10:07.283054 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.283026 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ba56e909-c84888fc5-mm82n"] Oct 30 18:10:07.319336 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.319300 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ba56e909-6f5476fdc4-vmz9q"] Oct 30 18:10:07.322771 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.322746 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ba56e909-6f5476fdc4-vmz9q"] Oct 30 18:10:07.344145 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.343507 4442 scope.go:122] "RemoveContainer" containerID="27bc1c83d1a22b5e49e3bfe66a08944accef9b477281fd5814f3f5ecbb22f0a2" Oct 30 18:10:07.344477 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:10:07.344377 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27bc1c83d1a22b5e49e3bfe66a08944accef9b477281fd5814f3f5ecbb22f0a2\": container with ID starting with 27bc1c83d1a22b5e49e3bfe66a08944accef9b477281fd5814f3f5ecbb22f0a2 not found: ID does not exist" containerID="27bc1c83d1a22b5e49e3bfe66a08944accef9b477281fd5814f3f5ecbb22f0a2" Oct 30 18:10:07.344477 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.344434 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27bc1c83d1a22b5e49e3bfe66a08944accef9b477281fd5814f3f5ecbb22f0a2"} err="failed to get container status \"27bc1c83d1a22b5e49e3bfe66a08944accef9b477281fd5814f3f5ecbb22f0a2\": rpc error: code = NotFound desc = could not find container \"27bc1c83d1a22b5e49e3bfe66a08944accef9b477281fd5814f3f5ecbb22f0a2\": container with ID starting with 27bc1c83d1a22b5e49e3bfe66a08944accef9b477281fd5814f3f5ecbb22f0a2 not found: ID does not exist" Oct 30 18:10:07.364818 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.347721 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-de7e3a-7b788bbfd6-mlkk9" Oct 30 18:10:07.519424 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.519293 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-de7e3a-6768cfb995-99nqx"] Oct 30 18:10:07.654542 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.649081 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54599e13-84bddc8b55-8hngq" Oct 30 18:10:07.722449 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.722416 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-54599e13-69cdd8dfb-fnppd" Oct 30 18:10:07.747915 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.744759 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a782e223-3075-4fb8-b38c-1c009b65de6e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a782e223-3075-4fb8-b38c-1c009b65de6e-maas-api-tls\") pod \"a782e223-3075-4fb8-b38c-1c009b65de6e\" (UID: \"a782e223-3075-4fb8-b38c-1c009b65de6e\") " Oct 30 18:10:07.747915 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.745372 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13cfe51c-22d4-49f0-827b-b10bac35f0c1-kube-api-access-4vxmx\" (UniqueName: \"kubernetes.io/projected/13cfe51c-22d4-49f0-827b-b10bac35f0c1-kube-api-access-4vxmx\") pod \"13cfe51c-22d4-49f0-827b-b10bac35f0c1\" (UID: \"13cfe51c-22d4-49f0-827b-b10bac35f0c1\") " Oct 30 18:10:07.747915 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.745913 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a782e223-3075-4fb8-b38c-1c009b65de6e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a782e223-3075-4fb8-b38c-1c009b65de6e-metrics-certs\") pod \"a782e223-3075-4fb8-b38c-1c009b65de6e\" (UID: \"a782e223-3075-4fb8-b38c-1c009b65de6e\") " Oct 30 18:10:07.747915 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.747467 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a782e223-3075-4fb8-b38c-1c009b65de6e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a782e223-3075-4fb8-b38c-1c009b65de6e-openshift-service-ca\") pod \"a782e223-3075-4fb8-b38c-1c009b65de6e\" (UID: \"a782e223-3075-4fb8-b38c-1c009b65de6e\") " Oct 30 18:10:07.747915 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.747903 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a782e223-3075-4fb8-b38c-1c009b65de6e-openshift-service-ca" pod "a782e223-3075-4fb8-b38c-1c009b65de6e" (UID: "a782e223-3075-4fb8-b38c-1c009b65de6e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:10:07.748333 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.747987 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a782e223-3075-4fb8-b38c-1c009b65de6e-kube-api-access-6q858\" (UniqueName: \"kubernetes.io/projected/a782e223-3075-4fb8-b38c-1c009b65de6e-kube-api-access-6q858\") pod \"a782e223-3075-4fb8-b38c-1c009b65de6e\" (UID: \"a782e223-3075-4fb8-b38c-1c009b65de6e\") " Oct 30 18:10:07.748333 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.748239 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13cfe51c-22d4-49f0-827b-b10bac35f0c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13cfe51c-22d4-49f0-827b-b10bac35f0c1-plugins-config-volume\") pod \"13cfe51c-22d4-49f0-827b-b10bac35f0c1\" (UID: \"13cfe51c-22d4-49f0-827b-b10bac35f0c1\") " Oct 30 18:10:07.759652 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.749343 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a782e223-3075-4fb8-b38c-1c009b65de6e-openshift-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:10:07.759652 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.751329 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13cfe51c-22d4-49f0-827b-b10bac35f0c1-plugins-config-volume" pod "13cfe51c-22d4-49f0-827b-b10bac35f0c1" (UID: "13cfe51c-22d4-49f0-827b-b10bac35f0c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:10:07.762199 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.760962 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a782e223-3075-4fb8-b38c-1c009b65de6e-metrics-certs" pod "a782e223-3075-4fb8-b38c-1c009b65de6e" (UID: "a782e223-3075-4fb8-b38c-1c009b65de6e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:10:07.762199 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.760991 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a782e223-3075-4fb8-b38c-1c009b65de6e-maas-api-tls" pod "a782e223-3075-4fb8-b38c-1c009b65de6e" (UID: "a782e223-3075-4fb8-b38c-1c009b65de6e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:10:07.762199 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.760999 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13cfe51c-22d4-49f0-827b-b10bac35f0c1-kube-api-access-4vxmx" pod "13cfe51c-22d4-49f0-827b-b10bac35f0c1" (UID: "13cfe51c-22d4-49f0-827b-b10bac35f0c1"). InnerVolumeSpecName "kube-api-access-4vxmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:10:07.762199 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.761231 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a782e223-3075-4fb8-b38c-1c009b65de6e-kube-api-access-6q858" pod "a782e223-3075-4fb8-b38c-1c009b65de6e" (UID: "a782e223-3075-4fb8-b38c-1c009b65de6e"). InnerVolumeSpecName "kube-api-access-6q858". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:10:07.789336 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.789307 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-54599e13-9fdd4c687-9dncf" Oct 30 18:10:07.803906 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.803869 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-de7e3a-7b788bbfd6-mlkk9"] Oct 30 18:10:07.813636 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:10:07.813607 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc4a9a79_c6c6_4660_ac88_16920ac96298.slice/crio-b49b514d8684101497137cfe1b75d2b140550de3237a556e3908f04e984c74ce WatchSource:0}: Error finding container b49b514d8684101497137cfe1b75d2b140550de3237a556e3908f04e984c74ce: Status 404 returned error can't find the container with id b49b514d8684101497137cfe1b75d2b140550de3237a556e3908f04e984c74ce Oct 30 18:10:07.850190 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.850157 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8bcefce-471d-44ab-93c7-e5917c7e6231-kube-api-access-zrkxw\" (UniqueName: \"kubernetes.io/projected/d8bcefce-471d-44ab-93c7-e5917c7e6231-kube-api-access-zrkxw\") pod \"d8bcefce-471d-44ab-93c7-e5917c7e6231\" (UID: \"d8bcefce-471d-44ab-93c7-e5917c7e6231\") " Oct 30 18:10:07.850355 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.850245 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8bcefce-471d-44ab-93c7-e5917c7e6231-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8bcefce-471d-44ab-93c7-e5917c7e6231-plugins-config-volume\") pod \"d8bcefce-471d-44ab-93c7-e5917c7e6231\" (UID: \"d8bcefce-471d-44ab-93c7-e5917c7e6231\") " Oct 30 18:10:07.850678 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.850655 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vxmx\" (UniqueName: \"kubernetes.io/projected/13cfe51c-22d4-49f0-827b-b10bac35f0c1-kube-api-access-4vxmx\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:10:07.850767 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.850680 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a782e223-3075-4fb8-b38c-1c009b65de6e-metrics-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:10:07.850767 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.850694 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q858\" (UniqueName: \"kubernetes.io/projected/a782e223-3075-4fb8-b38c-1c009b65de6e-kube-api-access-6q858\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:10:07.850767 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.850707 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13cfe51c-22d4-49f0-827b-b10bac35f0c1-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:10:07.850767 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.850720 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a782e223-3075-4fb8-b38c-1c009b65de6e-maas-api-tls\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:10:07.851029 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.851006 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8bcefce-471d-44ab-93c7-e5917c7e6231-plugins-config-volume" pod "d8bcefce-471d-44ab-93c7-e5917c7e6231" (UID: "d8bcefce-471d-44ab-93c7-e5917c7e6231"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:10:07.852945 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.852914 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8bcefce-471d-44ab-93c7-e5917c7e6231-kube-api-access-zrkxw" pod "d8bcefce-471d-44ab-93c7-e5917c7e6231" (UID: "d8bcefce-471d-44ab-93c7-e5917c7e6231"). InnerVolumeSpecName "kube-api-access-zrkxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:10:07.953155 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.952915 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8bcefce-471d-44ab-93c7-e5917c7e6231-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:10:07.953155 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:07.952954 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrkxw\" (UniqueName: \"kubernetes.io/projected/d8bcefce-471d-44ab-93c7-e5917c7e6231-kube-api-access-zrkxw\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:10:08.144528 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:08.144494 4442 generic.go:356] "Generic (PLEG): container finished" podID="d8bcefce-471d-44ab-93c7-e5917c7e6231" containerID="2db9bc43604c2819fb2a969160d82fa7d44ad5ef7234b1b89a838506d7f36a4c" exitCode=0 Oct 30 18:10:08.144703 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:08.144541 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-54599e13-9fdd4c687-9dncf" Oct 30 18:10:08.144703 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:08.144577 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-54599e13-9fdd4c687-9dncf" event={"ID":"d8bcefce-471d-44ab-93c7-e5917c7e6231","Type":"ContainerDied","Data":"2db9bc43604c2819fb2a969160d82fa7d44ad5ef7234b1b89a838506d7f36a4c"} Oct 30 18:10:08.144703 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:08.144642 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-54599e13-9fdd4c687-9dncf" event={"ID":"d8bcefce-471d-44ab-93c7-e5917c7e6231","Type":"ContainerDied","Data":"4572ea1f698d54ab263a90b285839245c8f678d5e4eb7d650ec6c5f78cfa3820"} Oct 30 18:10:08.144703 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:08.144677 4442 scope.go:122] "RemoveContainer" containerID="2db9bc43604c2819fb2a969160d82fa7d44ad5ef7234b1b89a838506d7f36a4c" Oct 30 18:10:08.147528 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:08.147501 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-de7e3a-6768cfb995-99nqx" event={"ID":"c4f54adc-5fd4-45b0-acb5-761613564059","Type":"ContainerStarted","Data":"c457c428be9fa917a05438e3f488d88ce84b33caf6025743d9416a7f8807c1c0"} Oct 30 18:10:08.147528 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:08.147548 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-de7e3a-6768cfb995-99nqx" event={"ID":"c4f54adc-5fd4-45b0-acb5-761613564059","Type":"ContainerStarted","Data":"dcbd0d48f620e6a5af90d91f275db52624fdc39e8b04c649867af3ad3d09e756"} Oct 30 18:10:08.147777 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:08.147579 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-de7e3a-6768cfb995-99nqx" Oct 30 18:10:08.149762 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:08.149683 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-de7e3a-7b788bbfd6-mlkk9" event={"ID":"fc4a9a79-c6c6-4660-ac88-16920ac96298","Type":"ContainerStarted","Data":"d98bf67132b75a1ebd692bdf37c97569b86164b9669518d76d313eee15a2f45f"} Oct 30 18:10:08.149762 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:08.149727 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-de7e3a-7b788bbfd6-mlkk9" event={"ID":"fc4a9a79-c6c6-4660-ac88-16920ac96298","Type":"ContainerStarted","Data":"b49b514d8684101497137cfe1b75d2b140550de3237a556e3908f04e984c74ce"} Oct 30 18:10:08.149905 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:08.149805 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-de7e3a-7b788bbfd6-mlkk9" Oct 30 18:10:08.152401 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:08.152373 4442 generic.go:356] "Generic (PLEG): container finished" podID="13cfe51c-22d4-49f0-827b-b10bac35f0c1" containerID="bd4ef26a937e95fb0d094202344d83c2234f013ce5a4207b9e7513421ecb5ffd" exitCode=0 Oct 30 18:10:08.152657 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:08.152538 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-54599e13-69cdd8dfb-fnppd" event={"ID":"13cfe51c-22d4-49f0-827b-b10bac35f0c1","Type":"ContainerDied","Data":"bd4ef26a937e95fb0d094202344d83c2234f013ce5a4207b9e7513421ecb5ffd"} Oct 30 18:10:08.152657 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:08.152575 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-54599e13-69cdd8dfb-fnppd" event={"ID":"13cfe51c-22d4-49f0-827b-b10bac35f0c1","Type":"ContainerDied","Data":"8361ec365888f46eb1030adad70e564751c5f723285af294a9432f22ed8238f1"} Oct 30 18:10:08.152657 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:08.152654 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-54599e13-69cdd8dfb-fnppd" Oct 30 18:10:08.154321 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:08.154299 4442 generic.go:356] "Generic (PLEG): container finished" podID="a782e223-3075-4fb8-b38c-1c009b65de6e" containerID="5c9c84fb3cdde6d399216a670d9d50d51d904f6f1183dd0df74435945ad22a04" exitCode=0 Oct 30 18:10:08.154994 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:08.154975 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54599e13-84bddc8b55-8hngq" Oct 30 18:10:08.157175 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:08.157160 4442 scope.go:122] "RemoveContainer" containerID="2db9bc43604c2819fb2a969160d82fa7d44ad5ef7234b1b89a838506d7f36a4c" Oct 30 18:10:08.157431 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:08.157362 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54599e13-84bddc8b55-8hngq" event={"ID":"a782e223-3075-4fb8-b38c-1c009b65de6e","Type":"ContainerDied","Data":"5c9c84fb3cdde6d399216a670d9d50d51d904f6f1183dd0df74435945ad22a04"} Oct 30 18:10:08.157431 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:08.157405 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54599e13-84bddc8b55-8hngq" event={"ID":"a782e223-3075-4fb8-b38c-1c009b65de6e","Type":"ContainerDied","Data":"dde375ab584d5760ea27b5c7e9620e470570130f6d33f00c020a1cefc1851819"} Oct 30 18:10:08.157710 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:10:08.157680 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2db9bc43604c2819fb2a969160d82fa7d44ad5ef7234b1b89a838506d7f36a4c\": container with ID starting with 2db9bc43604c2819fb2a969160d82fa7d44ad5ef7234b1b89a838506d7f36a4c not found: ID does not exist" containerID="2db9bc43604c2819fb2a969160d82fa7d44ad5ef7234b1b89a838506d7f36a4c" Oct 30 18:10:08.157782 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:08.157721 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2db9bc43604c2819fb2a969160d82fa7d44ad5ef7234b1b89a838506d7f36a4c"} err="failed to get container status \"2db9bc43604c2819fb2a969160d82fa7d44ad5ef7234b1b89a838506d7f36a4c\": rpc error: code = NotFound desc = could not find container \"2db9bc43604c2819fb2a969160d82fa7d44ad5ef7234b1b89a838506d7f36a4c\": container with ID starting with 2db9bc43604c2819fb2a969160d82fa7d44ad5ef7234b1b89a838506d7f36a4c not found: ID does not exist" Oct 30 18:10:08.157782 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:08.157744 4442 scope.go:122] "RemoveContainer" containerID="bd4ef26a937e95fb0d094202344d83c2234f013ce5a4207b9e7513421ecb5ffd" Oct 30 18:10:08.206226 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:08.206203 4442 scope.go:122] "RemoveContainer" containerID="bd4ef26a937e95fb0d094202344d83c2234f013ce5a4207b9e7513421ecb5ffd" Oct 30 18:10:08.207227 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:10:08.207046 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd4ef26a937e95fb0d094202344d83c2234f013ce5a4207b9e7513421ecb5ffd\": container with ID starting with bd4ef26a937e95fb0d094202344d83c2234f013ce5a4207b9e7513421ecb5ffd not found: ID does not exist" containerID="bd4ef26a937e95fb0d094202344d83c2234f013ce5a4207b9e7513421ecb5ffd" Oct 30 18:10:08.207227 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:08.207084 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd4ef26a937e95fb0d094202344d83c2234f013ce5a4207b9e7513421ecb5ffd"} err="failed to get container status \"bd4ef26a937e95fb0d094202344d83c2234f013ce5a4207b9e7513421ecb5ffd\": rpc error: code = NotFound desc = could not find container \"bd4ef26a937e95fb0d094202344d83c2234f013ce5a4207b9e7513421ecb5ffd\": container with ID starting with bd4ef26a937e95fb0d094202344d83c2234f013ce5a4207b9e7513421ecb5ffd not found: ID does not exist" Oct 30 18:10:08.207227 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:08.207112 4442 scope.go:122] "RemoveContainer" containerID="5c9c84fb3cdde6d399216a670d9d50d51d904f6f1183dd0df74435945ad22a04" Oct 30 18:10:08.214272 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:08.214096 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-54599e13-9fdd4c687-9dncf"] Oct 30 18:10:08.215711 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:08.215682 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-54599e13-9fdd4c687-9dncf"] Oct 30 18:10:08.246880 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:08.246859 4442 scope.go:122] "RemoveContainer" containerID="5c9c84fb3cdde6d399216a670d9d50d51d904f6f1183dd0df74435945ad22a04" Oct 30 18:10:08.247268 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:10:08.247244 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c9c84fb3cdde6d399216a670d9d50d51d904f6f1183dd0df74435945ad22a04\": container with ID starting with 5c9c84fb3cdde6d399216a670d9d50d51d904f6f1183dd0df74435945ad22a04 not found: ID does not exist" containerID="5c9c84fb3cdde6d399216a670d9d50d51d904f6f1183dd0df74435945ad22a04" Oct 30 18:10:08.247353 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:08.247281 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c9c84fb3cdde6d399216a670d9d50d51d904f6f1183dd0df74435945ad22a04"} err="failed to get container status \"5c9c84fb3cdde6d399216a670d9d50d51d904f6f1183dd0df74435945ad22a04\": rpc error: code = NotFound desc = could not find container \"5c9c84fb3cdde6d399216a670d9d50d51d904f6f1183dd0df74435945ad22a04\": container with ID starting with 5c9c84fb3cdde6d399216a670d9d50d51d904f6f1183dd0df74435945ad22a04 not found: ID does not exist" Oct 30 18:10:08.253080 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:08.253033 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-de7e3a-6768cfb995-99nqx" podStartSLOduration=2.253019051 podStartE2EDuration="2.253019051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:08.250122675 +0000 UTC m=+705.340185015" watchObservedRunningTime="2026-10-30 18:10:08.253019051 +0000 UTC m=+705.343081390" Oct 30 18:10:08.305849 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:08.305783 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-de7e3a-7b788bbfd6-mlkk9" podStartSLOduration=2.305763661 podStartE2EDuration="2.305763661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:08.304243412 +0000 UTC m=+705.394305763" watchObservedRunningTime="2026-10-30 18:10:08.305763661 +0000 UTC m=+705.395826002" Oct 30 18:10:08.355810 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:08.355778 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-54599e13-84bddc8b55-8hngq"] Oct 30 18:10:08.358492 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:08.358468 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-54599e13-84bddc8b55-8hngq"] Oct 30 18:10:08.390039 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:08.390001 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-54599e13-69cdd8dfb-fnppd"] Oct 30 18:10:08.394111 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:08.394086 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-54599e13-69cdd8dfb-fnppd"] Oct 30 18:10:09.066052 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:09.066020 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13cfe51c-22d4-49f0-827b-b10bac35f0c1" path="/var/lib/kubelet/pods/13cfe51c-22d4-49f0-827b-b10bac35f0c1/volumes" Oct 30 18:10:09.066459 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:09.066444 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="383feced-9ea0-4524-a76e-84112d54fa10" path="/var/lib/kubelet/pods/383feced-9ea0-4524-a76e-84112d54fa10/volumes" Oct 30 18:10:09.066833 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:09.066818 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a782e223-3075-4fb8-b38c-1c009b65de6e" path="/var/lib/kubelet/pods/a782e223-3075-4fb8-b38c-1c009b65de6e/volumes" Oct 30 18:10:09.067441 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:09.067423 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8edb1cd-d37b-4dd8-8600-729318034c2a" path="/var/lib/kubelet/pods/b8edb1cd-d37b-4dd8-8600-729318034c2a/volumes" Oct 30 18:10:09.067777 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:09.067762 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8bcefce-471d-44ab-93c7-e5917c7e6231" path="/var/lib/kubelet/pods/d8bcefce-471d-44ab-93c7-e5917c7e6231/volumes" Oct 30 18:10:09.841057 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:09.841006 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-de7e3a-78ddc7c46b-lv7hj"] Oct 30 18:10:09.842243 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:09.842215 4442 cpu_manager.go:414] "containerMap: removing container" podUID="13cfe51c-22d4-49f0-827b-b10bac35f0c1" containerName="payload-processing" Oct 30 18:10:09.842421 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:09.842406 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13cfe51c-22d4-49f0-827b-b10bac35f0c1" containerName="payload-processing" Oct 30 18:10:09.842536 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:09.842524 4442 cpu_manager.go:414] "containerMap: removing container" podUID="a782e223-3075-4fb8-b38c-1c009b65de6e" containerName="maas-api" Oct 30 18:10:09.842634 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:09.842623 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a782e223-3075-4fb8-b38c-1c009b65de6e" containerName="maas-api" Oct 30 18:10:09.842738 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:09.842727 4442 cpu_manager.go:414] "containerMap: removing container" podUID="d8bcefce-471d-44ab-93c7-e5917c7e6231" containerName="payload-pre-processing" Oct 30 18:10:09.842833 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:09.842822 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8bcefce-471d-44ab-93c7-e5917c7e6231" containerName="payload-pre-processing" Oct 30 18:10:09.843083 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:09.843069 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="13cfe51c-22d4-49f0-827b-b10bac35f0c1" containerName="payload-processing" Oct 30 18:10:09.843207 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:09.843196 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="a782e223-3075-4fb8-b38c-1c009b65de6e" containerName="maas-api" Oct 30 18:10:09.843315 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:09.843304 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8bcefce-471d-44ab-93c7-e5917c7e6231" containerName="payload-pre-processing" Oct 30 18:10:09.865519 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:09.865467 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-de7e3a-78ddc7c46b-lv7hj"] Oct 30 18:10:09.865707 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:09.865602 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-de7e3a-78ddc7c46b-lv7hj" Oct 30 18:10:09.899691 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:09.899667 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-de7e3a-serving-cert\"" Oct 30 18:10:09.990287 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:09.990252 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj9g7\" (UniqueName: \"kubernetes.io/projected/3884511b-b54a-4d44-94b5-69e405ffce06-kube-api-access-gj9g7\") pod \"maas-api-e2e-worker-w3-de7e3a-78ddc7c46b-lv7hj\" (UID: \"3884511b-b54a-4d44-94b5-69e405ffce06\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-de7e3a-78ddc7c46b-lv7hj" Oct 30 18:10:09.990469 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:09.990413 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3884511b-b54a-4d44-94b5-69e405ffce06-metrics-certs\") pod \"maas-api-e2e-worker-w3-de7e3a-78ddc7c46b-lv7hj\" (UID: \"3884511b-b54a-4d44-94b5-69e405ffce06\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-de7e3a-78ddc7c46b-lv7hj" Oct 30 18:10:09.990469 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:09.990455 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3884511b-b54a-4d44-94b5-69e405ffce06-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-de7e3a-78ddc7c46b-lv7hj\" (UID: \"3884511b-b54a-4d44-94b5-69e405ffce06\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-de7e3a-78ddc7c46b-lv7hj" Oct 30 18:10:09.990559 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:09.990536 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3884511b-b54a-4d44-94b5-69e405ffce06-maas-api-tls\") pod \"maas-api-e2e-worker-w3-de7e3a-78ddc7c46b-lv7hj\" (UID: \"3884511b-b54a-4d44-94b5-69e405ffce06\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-de7e3a-78ddc7c46b-lv7hj" Oct 30 18:10:10.097199 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:10.093452 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3884511b-b54a-4d44-94b5-69e405ffce06-metrics-certs\") pod \"maas-api-e2e-worker-w3-de7e3a-78ddc7c46b-lv7hj\" (UID: \"3884511b-b54a-4d44-94b5-69e405ffce06\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-de7e3a-78ddc7c46b-lv7hj" Oct 30 18:10:10.097199 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:10.093970 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3884511b-b54a-4d44-94b5-69e405ffce06-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-de7e3a-78ddc7c46b-lv7hj\" (UID: \"3884511b-b54a-4d44-94b5-69e405ffce06\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-de7e3a-78ddc7c46b-lv7hj" Oct 30 18:10:10.097199 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:10.095194 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3884511b-b54a-4d44-94b5-69e405ffce06-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-de7e3a-78ddc7c46b-lv7hj\" (UID: \"3884511b-b54a-4d44-94b5-69e405ffce06\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-de7e3a-78ddc7c46b-lv7hj" Oct 30 18:10:10.097199 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:10.095574 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3884511b-b54a-4d44-94b5-69e405ffce06-maas-api-tls\") pod \"maas-api-e2e-worker-w3-de7e3a-78ddc7c46b-lv7hj\" (UID: \"3884511b-b54a-4d44-94b5-69e405ffce06\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-de7e3a-78ddc7c46b-lv7hj" Oct 30 18:10:10.097199 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:10.095641 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj9g7\" (UniqueName: \"kubernetes.io/projected/3884511b-b54a-4d44-94b5-69e405ffce06-kube-api-access-gj9g7\") pod \"maas-api-e2e-worker-w3-de7e3a-78ddc7c46b-lv7hj\" (UID: \"3884511b-b54a-4d44-94b5-69e405ffce06\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-de7e3a-78ddc7c46b-lv7hj" Oct 30 18:10:10.098861 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:10.098769 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3884511b-b54a-4d44-94b5-69e405ffce06-metrics-certs\") pod \"maas-api-e2e-worker-w3-de7e3a-78ddc7c46b-lv7hj\" (UID: \"3884511b-b54a-4d44-94b5-69e405ffce06\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-de7e3a-78ddc7c46b-lv7hj" Oct 30 18:10:10.116811 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:10.116676 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj9g7\" (UniqueName: \"kubernetes.io/projected/3884511b-b54a-4d44-94b5-69e405ffce06-kube-api-access-gj9g7\") pod \"maas-api-e2e-worker-w3-de7e3a-78ddc7c46b-lv7hj\" (UID: \"3884511b-b54a-4d44-94b5-69e405ffce06\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-de7e3a-78ddc7c46b-lv7hj" Oct 30 18:10:10.118433 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:10.118334 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3884511b-b54a-4d44-94b5-69e405ffce06-maas-api-tls\") pod \"maas-api-e2e-worker-w3-de7e3a-78ddc7c46b-lv7hj\" (UID: \"3884511b-b54a-4d44-94b5-69e405ffce06\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-de7e3a-78ddc7c46b-lv7hj" Oct 30 18:10:10.204262 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:10.204223 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-de7e3a-78ddc7c46b-lv7hj" Oct 30 18:10:10.510966 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:10.510920 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-de7e3a-78ddc7c46b-lv7hj"] Oct 30 18:10:10.513381 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:10:10.513351 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3884511b_b54a_4d44_94b5_69e405ffce06.slice/crio-970b65a251a7c2c2d321e7564c3f2b29261fd0e3fa4173a1c8528a9bef21fc9f WatchSource:0}: Error finding container 970b65a251a7c2c2d321e7564c3f2b29261fd0e3fa4173a1c8528a9bef21fc9f: Status 404 returned error can't find the container with id 970b65a251a7c2c2d321e7564c3f2b29261fd0e3fa4173a1c8528a9bef21fc9f Oct 30 18:10:11.066199 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:11.066171 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-921450-7f64fb4c79-xs76b" Oct 30 18:10:11.066385 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:11.066256 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-921450-8d6546c9b-5pxxc" Oct 30 18:10:11.160160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:11.158513 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-921450-7cc76dbcd5-vwjkc" Oct 30 18:10:11.210239 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:11.205924 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-de7e3a-78ddc7c46b-lv7hj" event={"ID":"3884511b-b54a-4d44-94b5-69e405ffce06","Type":"ContainerStarted","Data":"8f2bfbaa1fd86cc0492716057d304eddf82d07f631deeef1ee4a2ccbe3a21375"} Oct 30 18:10:11.210239 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:11.205995 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-de7e3a-78ddc7c46b-lv7hj" event={"ID":"3884511b-b54a-4d44-94b5-69e405ffce06","Type":"ContainerStarted","Data":"970b65a251a7c2c2d321e7564c3f2b29261fd0e3fa4173a1c8528a9bef21fc9f"} Oct 30 18:10:11.210239 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:11.208782 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-de7e3a-78ddc7c46b-lv7hj" Oct 30 18:10:13.039007 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:13.038949 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-de7e3a-78ddc7c46b-lv7hj" podStartSLOduration=3.622778993 podStartE2EDuration="4.038929198s" totalImagesPullingTime="416.150205ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:11.263115211 +0000 UTC m=+708.353177528" watchObservedRunningTime="2026-10-30 18:10:13.038929198 +0000 UTC m=+710.128991538" Oct 30 18:10:13.042398 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:13.042366 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-380c08-67478544f9-4pdqp"] Oct 30 18:10:13.046673 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:13.046652 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-380c08-67478544f9-4pdqp" Oct 30 18:10:13.065384 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:13.065360 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-380c08-67478544f9-4pdqp"] Oct 30 18:10:13.072239 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:13.072205 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-380c08\"" Oct 30 18:10:13.072449 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:13.072429 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-380c08-dockercfg-cqbfc\"" Oct 30 18:10:13.095009 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:13.094982 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-45dc34-79f9556b56-st7qr" Oct 30 18:10:13.097076 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:13.096998 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-45dc34-d7f945cff-rqxz2" Oct 30 18:10:13.124929 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:13.124902 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-380c08-65f99966c6-8p2v9"] Oct 30 18:10:13.128901 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:13.128880 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-380c08-65f99966c6-8p2v9" Oct 30 18:10:13.129788 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:13.129760 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b4cff9f-9eb5-4fc5-a125-0d92216168e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-380c08-67478544f9-4pdqp\" (UID: \"9b4cff9f-9eb5-4fc5-a125-0d92216168e6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-380c08-67478544f9-4pdqp" Oct 30 18:10:13.130370 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:13.130341 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzsz8\" (UniqueName: \"kubernetes.io/projected/9b4cff9f-9eb5-4fc5-a125-0d92216168e6-kube-api-access-pzsz8\") pod \"payload-pre-processing-e2e-worker-w1-380c08-67478544f9-4pdqp\" (UID: \"9b4cff9f-9eb5-4fc5-a125-0d92216168e6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-380c08-67478544f9-4pdqp" Oct 30 18:10:13.141927 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:13.141897 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-380c08-65f99966c6-8p2v9"] Oct 30 18:10:13.162630 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:13.162606 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-45dc34-6cc4b7467c-8fgqt" Oct 30 18:10:13.231632 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:13.231593 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3b6ea07-7632-44a9-9343-20b7ac9526c5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-380c08-65f99966c6-8p2v9\" (UID: \"a3b6ea07-7632-44a9-9343-20b7ac9526c5\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-380c08-65f99966c6-8p2v9" Oct 30 18:10:13.231811 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:13.231674 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b4cff9f-9eb5-4fc5-a125-0d92216168e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-380c08-67478544f9-4pdqp\" (UID: \"9b4cff9f-9eb5-4fc5-a125-0d92216168e6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-380c08-67478544f9-4pdqp" Oct 30 18:10:13.231811 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:13.231758 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tflz8\" (UniqueName: \"kubernetes.io/projected/a3b6ea07-7632-44a9-9343-20b7ac9526c5-kube-api-access-tflz8\") pod \"payload-processing-e2e-worker-w1-380c08-65f99966c6-8p2v9\" (UID: \"a3b6ea07-7632-44a9-9343-20b7ac9526c5\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-380c08-65f99966c6-8p2v9" Oct 30 18:10:13.231956 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:13.231915 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzsz8\" (UniqueName: \"kubernetes.io/projected/9b4cff9f-9eb5-4fc5-a125-0d92216168e6-kube-api-access-pzsz8\") pod \"payload-pre-processing-e2e-worker-w1-380c08-67478544f9-4pdqp\" (UID: \"9b4cff9f-9eb5-4fc5-a125-0d92216168e6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-380c08-67478544f9-4pdqp" Oct 30 18:10:13.232423 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:13.232400 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b4cff9f-9eb5-4fc5-a125-0d92216168e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-380c08-67478544f9-4pdqp\" (UID: \"9b4cff9f-9eb5-4fc5-a125-0d92216168e6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-380c08-67478544f9-4pdqp" Oct 30 18:10:13.251032 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:13.251001 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzsz8\" (UniqueName: \"kubernetes.io/projected/9b4cff9f-9eb5-4fc5-a125-0d92216168e6-kube-api-access-pzsz8\") pod \"payload-pre-processing-e2e-worker-w1-380c08-67478544f9-4pdqp\" (UID: \"9b4cff9f-9eb5-4fc5-a125-0d92216168e6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-380c08-67478544f9-4pdqp" Oct 30 18:10:13.332927 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:13.332830 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3b6ea07-7632-44a9-9343-20b7ac9526c5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-380c08-65f99966c6-8p2v9\" (UID: \"a3b6ea07-7632-44a9-9343-20b7ac9526c5\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-380c08-65f99966c6-8p2v9" Oct 30 18:10:13.333085 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:13.332930 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tflz8\" (UniqueName: \"kubernetes.io/projected/a3b6ea07-7632-44a9-9343-20b7ac9526c5-kube-api-access-tflz8\") pod \"payload-processing-e2e-worker-w1-380c08-65f99966c6-8p2v9\" (UID: \"a3b6ea07-7632-44a9-9343-20b7ac9526c5\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-380c08-65f99966c6-8p2v9" Oct 30 18:10:13.333680 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:13.333660 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3b6ea07-7632-44a9-9343-20b7ac9526c5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-380c08-65f99966c6-8p2v9\" (UID: \"a3b6ea07-7632-44a9-9343-20b7ac9526c5\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-380c08-65f99966c6-8p2v9" Oct 30 18:10:13.352215 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:13.352182 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tflz8\" (UniqueName: \"kubernetes.io/projected/a3b6ea07-7632-44a9-9343-20b7ac9526c5-kube-api-access-tflz8\") pod \"payload-processing-e2e-worker-w1-380c08-65f99966c6-8p2v9\" (UID: \"a3b6ea07-7632-44a9-9343-20b7ac9526c5\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-380c08-65f99966c6-8p2v9" Oct 30 18:10:13.361014 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:13.360986 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-380c08-67478544f9-4pdqp" Oct 30 18:10:13.450617 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:13.449372 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-380c08-65f99966c6-8p2v9" Oct 30 18:10:13.706641 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:13.706605 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-380c08-67478544f9-4pdqp"] Oct 30 18:10:13.709705 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:10:13.709674 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b4cff9f_9eb5_4fc5_a125_0d92216168e6.slice/crio-628ddd89de0850e3e41ce17a9b349e713ca443ec58df99f28bcc96eb842a319b WatchSource:0}: Error finding container 628ddd89de0850e3e41ce17a9b349e713ca443ec58df99f28bcc96eb842a319b: Status 404 returned error can't find the container with id 628ddd89de0850e3e41ce17a9b349e713ca443ec58df99f28bcc96eb842a319b Oct 30 18:10:13.806576 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:13.806552 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-380c08-65f99966c6-8p2v9"] Oct 30 18:10:13.809347 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:10:13.809318 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3b6ea07_7632_44a9_9343_20b7ac9526c5.slice/crio-b4f37265acb6a533ea93a161b14e14e2a8072d19c273a3ba907a9e03dcb269d0 WatchSource:0}: Error finding container b4f37265acb6a533ea93a161b14e14e2a8072d19c273a3ba907a9e03dcb269d0: Status 404 returned error can't find the container with id b4f37265acb6a533ea93a161b14e14e2a8072d19c273a3ba907a9e03dcb269d0 Oct 30 18:10:14.229755 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:14.229717 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-380c08-65f99966c6-8p2v9" event={"ID":"a3b6ea07-7632-44a9-9343-20b7ac9526c5","Type":"ContainerStarted","Data":"f98a7cabbe20754eb4b650e704ede40269e9fe27cd2a5df9413d1a2465b9cddd"} Oct 30 18:10:14.230275 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:14.229787 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-380c08-65f99966c6-8p2v9" event={"ID":"a3b6ea07-7632-44a9-9343-20b7ac9526c5","Type":"ContainerStarted","Data":"b4f37265acb6a533ea93a161b14e14e2a8072d19c273a3ba907a9e03dcb269d0"} Oct 30 18:10:14.230275 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:14.229944 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-380c08-65f99966c6-8p2v9" Oct 30 18:10:14.231719 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:14.231687 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-380c08-67478544f9-4pdqp" event={"ID":"9b4cff9f-9eb5-4fc5-a125-0d92216168e6","Type":"ContainerStarted","Data":"7c0527787e5aa1965595b7875173efdb86ae30c58ab6612cbbc58e79582bdd6b"} Oct 30 18:10:14.231875 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:14.231737 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-380c08-67478544f9-4pdqp" event={"ID":"9b4cff9f-9eb5-4fc5-a125-0d92216168e6","Type":"ContainerStarted","Data":"628ddd89de0850e3e41ce17a9b349e713ca443ec58df99f28bcc96eb842a319b"} Oct 30 18:10:14.231875 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:14.231856 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-380c08-67478544f9-4pdqp" Oct 30 18:10:14.271918 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:14.271833 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-380c08-65f99966c6-8p2v9" podStartSLOduration=1.271817098 podStartE2EDuration="1.271817098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:14.267423107 +0000 UTC m=+711.357485448" watchObservedRunningTime="2026-10-30 18:10:14.271817098 +0000 UTC m=+711.361879430" Oct 30 18:10:14.313302 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:14.313248 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-380c08-67478544f9-4pdqp" podStartSLOduration=1.313235156 podStartE2EDuration="1.313235156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:14.309387942 +0000 UTC m=+711.399450282" watchObservedRunningTime="2026-10-30 18:10:14.313235156 +0000 UTC m=+711.403297495" Oct 30 18:10:17.382820 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:17.382790 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-fe1374-5c8465bf67-bl75c"] Oct 30 18:10:17.386869 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:17.386854 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fe1374-5c8465bf67-bl75c" Oct 30 18:10:17.404253 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:17.404221 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-fe1374-5c8465bf67-bl75c"] Oct 30 18:10:17.412916 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:17.412889 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-fe1374\"" Oct 30 18:10:17.412916 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:17.412889 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-fe1374-dockercfg-wvc5w\"" Oct 30 18:10:17.511159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:17.506458 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgl6k\" (UniqueName: \"kubernetes.io/projected/6170e608-fbfb-4cfe-acf6-2c09756134b5-kube-api-access-xgl6k\") pod \"payload-pre-processing-e2e-worker-w4-fe1374-5c8465bf67-bl75c\" (UID: \"6170e608-fbfb-4cfe-acf6-2c09756134b5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fe1374-5c8465bf67-bl75c" Oct 30 18:10:17.511159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:17.506522 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6170e608-fbfb-4cfe-acf6-2c09756134b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-fe1374-5c8465bf67-bl75c\" (UID: \"6170e608-fbfb-4cfe-acf6-2c09756134b5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fe1374-5c8465bf67-bl75c" Oct 30 18:10:17.561898 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:10:17.561864 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:10:17.607706 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:17.607650 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6170e608-fbfb-4cfe-acf6-2c09756134b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-fe1374-5c8465bf67-bl75c\" (UID: \"6170e608-fbfb-4cfe-acf6-2c09756134b5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fe1374-5c8465bf67-bl75c" Oct 30 18:10:17.607877 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:17.607792 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgl6k\" (UniqueName: \"kubernetes.io/projected/6170e608-fbfb-4cfe-acf6-2c09756134b5-kube-api-access-xgl6k\") pod \"payload-pre-processing-e2e-worker-w4-fe1374-5c8465bf67-bl75c\" (UID: \"6170e608-fbfb-4cfe-acf6-2c09756134b5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fe1374-5c8465bf67-bl75c" Oct 30 18:10:17.608430 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:17.608405 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6170e608-fbfb-4cfe-acf6-2c09756134b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-fe1374-5c8465bf67-bl75c\" (UID: \"6170e608-fbfb-4cfe-acf6-2c09756134b5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fe1374-5c8465bf67-bl75c" Oct 30 18:10:17.626658 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:17.626622 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgl6k\" (UniqueName: \"kubernetes.io/projected/6170e608-fbfb-4cfe-acf6-2c09756134b5-kube-api-access-xgl6k\") pod \"payload-pre-processing-e2e-worker-w4-fe1374-5c8465bf67-bl75c\" (UID: \"6170e608-fbfb-4cfe-acf6-2c09756134b5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fe1374-5c8465bf67-bl75c" Oct 30 18:10:17.700785 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:17.700695 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fe1374-5c8465bf67-bl75c" Oct 30 18:10:18.000962 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:18.000879 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-fe1374-5c8465bf67-bl75c"] Oct 30 18:10:18.004772 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:10:18.004747 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6170e608_fbfb_4cfe_acf6_2c09756134b5.slice/crio-e2f6928c84aead200706f29214237d94b7cabd5beafe8d2fc019084362a5df63 WatchSource:0}: Error finding container e2f6928c84aead200706f29214237d94b7cabd5beafe8d2fc019084362a5df63: Status 404 returned error can't find the container with id e2f6928c84aead200706f29214237d94b7cabd5beafe8d2fc019084362a5df63 Oct 30 18:10:18.227459 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:18.227434 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-de7e3a-78ddc7c46b-lv7hj" Oct 30 18:10:18.259784 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:18.259691 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fe1374-5c8465bf67-bl75c" event={"ID":"6170e608-fbfb-4cfe-acf6-2c09756134b5","Type":"ContainerStarted","Data":"04b2dee4587e21f965c8016a926185e606341aea019d4f1327971fcf1d3f7fbb"} Oct 30 18:10:18.259784 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:18.259755 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fe1374-5c8465bf67-bl75c" event={"ID":"6170e608-fbfb-4cfe-acf6-2c09756134b5","Type":"ContainerStarted","Data":"e2f6928c84aead200706f29214237d94b7cabd5beafe8d2fc019084362a5df63"} Oct 30 18:10:18.259968 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:18.259805 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fe1374-5c8465bf67-bl75c" Oct 30 18:10:18.308854 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:18.308801 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fe1374-5c8465bf67-bl75c" podStartSLOduration=1.308785565 podStartE2EDuration="1.308785565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:18.306983601 +0000 UTC m=+715.397045940" watchObservedRunningTime="2026-10-30 18:10:18.308785565 +0000 UTC m=+715.398847903" Oct 30 18:10:19.170636 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:19.170606 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-de7e3a-6768cfb995-99nqx" Oct 30 18:10:19.171156 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:19.170683 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-de7e3a-7b788bbfd6-mlkk9" Oct 30 18:10:24.662395 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:24.662363 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz"] Oct 30 18:10:24.668357 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:24.668326 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" Oct 30 18:10:24.684672 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:24.684644 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-45dc34-openshift-default-dockercfg-w86md\"" Oct 30 18:10:24.700391 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:24.700355 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz"] Oct 30 18:10:24.829255 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:24.829223 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-workload-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz\" (UID: \"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" Oct 30 18:10:24.829439 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:24.829263 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-credential-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz\" (UID: \"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" Oct 30 18:10:24.829439 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:24.829364 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-workload-certs\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz\" (UID: \"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" Oct 30 18:10:24.829439 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:24.829393 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b26ph\" (UniqueName: \"kubernetes.io/projected/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-kube-api-access-b26ph\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz\" (UID: \"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" Oct 30 18:10:24.829560 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:24.829504 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-istio-data\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz\" (UID: \"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" Oct 30 18:10:24.829560 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:24.829529 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-istiod-ca-cert\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz\" (UID: \"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" Oct 30 18:10:24.829665 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:24.829557 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-istio-token\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz\" (UID: \"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" Oct 30 18:10:24.829665 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:24.829583 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-istio-podinfo\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz\" (UID: \"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" Oct 30 18:10:24.829783 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:24.829759 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-istio-envoy\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz\" (UID: \"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" Oct 30 18:10:24.931318 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:24.931226 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-workload-certs\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz\" (UID: \"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" Oct 30 18:10:24.931318 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:24.931265 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b26ph\" (UniqueName: \"kubernetes.io/projected/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-kube-api-access-b26ph\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz\" (UID: \"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" Oct 30 18:10:24.931318 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:24.931309 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-istio-data\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz\" (UID: \"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" Oct 30 18:10:24.931590 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:24.931328 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-istiod-ca-cert\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz\" (UID: \"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" Oct 30 18:10:24.931590 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:24.931353 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-istio-token\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz\" (UID: \"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" Oct 30 18:10:24.931590 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:24.931376 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-istio-podinfo\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz\" (UID: \"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" Oct 30 18:10:24.931590 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:24.931442 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-istio-envoy\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz\" (UID: \"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" Oct 30 18:10:24.931590 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:24.931497 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-workload-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz\" (UID: \"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" Oct 30 18:10:24.931590 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:24.931519 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-credential-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz\" (UID: \"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" Oct 30 18:10:24.931822 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:24.931729 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-workload-certs\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz\" (UID: \"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" Oct 30 18:10:24.931932 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:24.931913 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-istio-data\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz\" (UID: \"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" Oct 30 18:10:24.932002 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:24.931979 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-credential-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz\" (UID: \"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" Oct 30 18:10:24.932113 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:24.932092 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-workload-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz\" (UID: \"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" Oct 30 18:10:24.932252 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:24.932229 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-istiod-ca-cert\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz\" (UID: \"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" Oct 30 18:10:24.933989 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:24.933969 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-istio-envoy\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz\" (UID: \"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" Oct 30 18:10:24.934223 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:24.934203 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-istio-podinfo\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz\" (UID: \"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" Oct 30 18:10:24.951041 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:24.951011 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-istio-token\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz\" (UID: \"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" Oct 30 18:10:24.951197 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:24.951112 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b26ph\" (UniqueName: \"kubernetes.io/projected/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-kube-api-access-b26ph\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz\" (UID: \"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" Oct 30 18:10:24.984205 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:24.984176 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" Oct 30 18:10:25.241810 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:25.241712 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-380c08-65f99966c6-8p2v9" Oct 30 18:10:25.241978 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:25.241836 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-380c08-67478544f9-4pdqp" Oct 30 18:10:25.332537 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:25.325123 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz"] Oct 30 18:10:25.339395 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:25.339356 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:10:25.339624 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:25.339602 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:10:25.339765 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:25.339748 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:10:26.389890 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:26.389858 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" event={"ID":"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b","Type":"ContainerStarted","Data":"c107c0144fd8662e10c3814030d867147b8aa0a211d6b1a9cc9ecf9081340b67"} Oct 30 18:10:26.390413 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:26.389949 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" event={"ID":"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b","Type":"ContainerStarted","Data":"64fe6c3c2875a2e9c645c06998623494a54dad168ecfdb73b63325abbbf497d4"} Oct 30 18:10:26.450163 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:26.449700 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" podStartSLOduration=2.449682116 podStartE2EDuration="2.449682116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:26.447570662 +0000 UTC m=+723.537633038" watchObservedRunningTime="2026-10-30 18:10:26.449682116 +0000 UTC m=+723.539744458" Oct 30 18:10:26.985065 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:26.985026 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" Oct 30 18:10:26.986662 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:26.986636 4442 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 18:10:26.986772 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:26.986690 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" podUID="5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 18:10:27.988499 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:27.988465 4442 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 18:10:27.989090 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:27.989062 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" podUID="5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 18:10:28.984901 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:28.984869 4442 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 18:10:28.985080 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:28.984926 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" podUID="5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 18:10:29.278651 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:29.278577 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fe1374-5c8465bf67-bl75c" Oct 30 18:10:29.985544 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:29.985510 4442 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 18:10:29.985711 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:29.985564 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" podUID="5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 18:10:30.985097 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:30.985065 4442 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 18:10:30.985677 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:30.985120 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" podUID="5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 18:10:32.095470 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:32.095442 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" Oct 30 18:10:32.096687 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:32.096665 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" Oct 30 18:10:32.097681 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:32.097659 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" Oct 30 18:10:47.559645 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:10:47.559600 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:10:54.722505 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:54.722467 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj"] Oct 30 18:10:54.726961 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:54.726932 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" Oct 30 18:10:54.759627 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:54.759586 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj"] Oct 30 18:10:54.764788 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:54.764753 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff"] Oct 30 18:10:54.776319 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:54.776287 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" Oct 30 18:10:54.793826 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:54.793797 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff"] Oct 30 18:10:54.912919 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:54.912882 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07931e20-5652-4e4c-a9d4-6cec3a6c7163-workload-certs\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj\" (UID: \"07931e20-5652-4e4c-a9d4-6cec3a6c7163\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" Oct 30 18:10:54.913107 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:54.912932 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07931e20-5652-4e4c-a9d4-6cec3a6c7163-istio-podinfo\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj\" (UID: \"07931e20-5652-4e4c-a9d4-6cec3a6c7163\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" Oct 30 18:10:54.913107 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:54.912962 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mgqt\" (UniqueName: \"kubernetes.io/projected/07931e20-5652-4e4c-a9d4-6cec3a6c7163-kube-api-access-9mgqt\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj\" (UID: \"07931e20-5652-4e4c-a9d4-6cec3a6c7163\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" Oct 30 18:10:54.913107 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:54.913027 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07931e20-5652-4e4c-a9d4-6cec3a6c7163-istiod-ca-cert\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj\" (UID: \"07931e20-5652-4e4c-a9d4-6cec3a6c7163\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" Oct 30 18:10:54.913258 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:54.913159 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec2a970c-4563-416f-9357-379fbec2f852-istio-podinfo\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff\" (UID: \"ec2a970c-4563-416f-9357-379fbec2f852\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" Oct 30 18:10:54.913258 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:54.913221 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2a970c-4563-416f-9357-379fbec2f852-credential-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff\" (UID: \"ec2a970c-4563-416f-9357-379fbec2f852\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" Oct 30 18:10:54.913323 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:54.913298 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec2a970c-4563-416f-9357-379fbec2f852-workload-certs\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff\" (UID: \"ec2a970c-4563-416f-9357-379fbec2f852\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" Oct 30 18:10:54.913323 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:54.913315 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkf2l\" (UniqueName: \"kubernetes.io/projected/ec2a970c-4563-416f-9357-379fbec2f852-kube-api-access-mkf2l\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff\" (UID: \"ec2a970c-4563-416f-9357-379fbec2f852\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" Oct 30 18:10:54.913384 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:54.913338 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07931e20-5652-4e4c-a9d4-6cec3a6c7163-istio-data\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj\" (UID: \"07931e20-5652-4e4c-a9d4-6cec3a6c7163\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" Oct 30 18:10:54.913417 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:54.913396 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec2a970c-4563-416f-9357-379fbec2f852-istiod-ca-cert\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff\" (UID: \"ec2a970c-4563-416f-9357-379fbec2f852\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" Oct 30 18:10:54.913450 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:54.913438 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07931e20-5652-4e4c-a9d4-6cec3a6c7163-credential-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj\" (UID: \"07931e20-5652-4e4c-a9d4-6cec3a6c7163\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" Oct 30 18:10:54.913527 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:54.913462 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec2a970c-4563-416f-9357-379fbec2f852-istio-envoy\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff\" (UID: \"ec2a970c-4563-416f-9357-379fbec2f852\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" Oct 30 18:10:54.913558 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:54.913531 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec2a970c-4563-416f-9357-379fbec2f852-istio-token\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff\" (UID: \"ec2a970c-4563-416f-9357-379fbec2f852\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" Oct 30 18:10:54.913590 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:54.913569 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec2a970c-4563-416f-9357-379fbec2f852-istio-data\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff\" (UID: \"ec2a970c-4563-416f-9357-379fbec2f852\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" Oct 30 18:10:54.913590 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:54.913585 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2a970c-4563-416f-9357-379fbec2f852-workload-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff\" (UID: \"ec2a970c-4563-416f-9357-379fbec2f852\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" Oct 30 18:10:54.913650 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:54.913600 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07931e20-5652-4e4c-a9d4-6cec3a6c7163-istio-envoy\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj\" (UID: \"07931e20-5652-4e4c-a9d4-6cec3a6c7163\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" Oct 30 18:10:54.913650 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:54.913615 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07931e20-5652-4e4c-a9d4-6cec3a6c7163-istio-token\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj\" (UID: \"07931e20-5652-4e4c-a9d4-6cec3a6c7163\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" Oct 30 18:10:54.913710 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:54.913656 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07931e20-5652-4e4c-a9d4-6cec3a6c7163-workload-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj\" (UID: \"07931e20-5652-4e4c-a9d4-6cec3a6c7163\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" Oct 30 18:10:55.014761 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.014683 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec2a970c-4563-416f-9357-379fbec2f852-istio-podinfo\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff\" (UID: \"ec2a970c-4563-416f-9357-379fbec2f852\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" Oct 30 18:10:55.014761 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.014737 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2a970c-4563-416f-9357-379fbec2f852-credential-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff\" (UID: \"ec2a970c-4563-416f-9357-379fbec2f852\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" Oct 30 18:10:55.014997 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.014772 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec2a970c-4563-416f-9357-379fbec2f852-workload-certs\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff\" (UID: \"ec2a970c-4563-416f-9357-379fbec2f852\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" Oct 30 18:10:55.014997 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.014790 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkf2l\" (UniqueName: \"kubernetes.io/projected/ec2a970c-4563-416f-9357-379fbec2f852-kube-api-access-mkf2l\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff\" (UID: \"ec2a970c-4563-416f-9357-379fbec2f852\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" Oct 30 18:10:55.014997 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.014814 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07931e20-5652-4e4c-a9d4-6cec3a6c7163-istio-data\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj\" (UID: \"07931e20-5652-4e4c-a9d4-6cec3a6c7163\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" Oct 30 18:10:55.014997 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.014847 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec2a970c-4563-416f-9357-379fbec2f852-istiod-ca-cert\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff\" (UID: \"ec2a970c-4563-416f-9357-379fbec2f852\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" Oct 30 18:10:55.014997 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.014888 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07931e20-5652-4e4c-a9d4-6cec3a6c7163-credential-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj\" (UID: \"07931e20-5652-4e4c-a9d4-6cec3a6c7163\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" Oct 30 18:10:55.014997 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.014916 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec2a970c-4563-416f-9357-379fbec2f852-istio-envoy\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff\" (UID: \"ec2a970c-4563-416f-9357-379fbec2f852\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" Oct 30 18:10:55.014997 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.014970 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec2a970c-4563-416f-9357-379fbec2f852-istio-token\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff\" (UID: \"ec2a970c-4563-416f-9357-379fbec2f852\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" Oct 30 18:10:55.015486 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.015169 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec2a970c-4563-416f-9357-379fbec2f852-istio-data\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff\" (UID: \"ec2a970c-4563-416f-9357-379fbec2f852\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" Oct 30 18:10:55.015486 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.015205 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2a970c-4563-416f-9357-379fbec2f852-workload-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff\" (UID: \"ec2a970c-4563-416f-9357-379fbec2f852\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" Oct 30 18:10:55.015486 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.015232 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07931e20-5652-4e4c-a9d4-6cec3a6c7163-istio-envoy\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj\" (UID: \"07931e20-5652-4e4c-a9d4-6cec3a6c7163\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" Oct 30 18:10:55.015486 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.015261 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07931e20-5652-4e4c-a9d4-6cec3a6c7163-istio-token\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj\" (UID: \"07931e20-5652-4e4c-a9d4-6cec3a6c7163\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" Oct 30 18:10:55.015486 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.015274 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07931e20-5652-4e4c-a9d4-6cec3a6c7163-istio-data\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj\" (UID: \"07931e20-5652-4e4c-a9d4-6cec3a6c7163\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" Oct 30 18:10:55.015486 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.015293 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07931e20-5652-4e4c-a9d4-6cec3a6c7163-workload-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj\" (UID: \"07931e20-5652-4e4c-a9d4-6cec3a6c7163\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" Oct 30 18:10:55.015486 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.015315 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec2a970c-4563-416f-9357-379fbec2f852-workload-certs\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff\" (UID: \"ec2a970c-4563-416f-9357-379fbec2f852\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" Oct 30 18:10:55.015486 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.015433 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07931e20-5652-4e4c-a9d4-6cec3a6c7163-workload-certs\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj\" (UID: \"07931e20-5652-4e4c-a9d4-6cec3a6c7163\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" Oct 30 18:10:55.016062 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.015481 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07931e20-5652-4e4c-a9d4-6cec3a6c7163-istio-podinfo\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj\" (UID: \"07931e20-5652-4e4c-a9d4-6cec3a6c7163\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" Oct 30 18:10:55.016062 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.015532 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mgqt\" (UniqueName: \"kubernetes.io/projected/07931e20-5652-4e4c-a9d4-6cec3a6c7163-kube-api-access-9mgqt\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj\" (UID: \"07931e20-5652-4e4c-a9d4-6cec3a6c7163\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" Oct 30 18:10:55.016062 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.015551 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07931e20-5652-4e4c-a9d4-6cec3a6c7163-istiod-ca-cert\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj\" (UID: \"07931e20-5652-4e4c-a9d4-6cec3a6c7163\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" Oct 30 18:10:55.016062 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.015989 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec2a970c-4563-416f-9357-379fbec2f852-istiod-ca-cert\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff\" (UID: \"ec2a970c-4563-416f-9357-379fbec2f852\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" Oct 30 18:10:55.016315 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.016062 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec2a970c-4563-416f-9357-379fbec2f852-istio-data\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff\" (UID: \"ec2a970c-4563-416f-9357-379fbec2f852\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" Oct 30 18:10:55.016315 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.016179 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07931e20-5652-4e4c-a9d4-6cec3a6c7163-istiod-ca-cert\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj\" (UID: \"07931e20-5652-4e4c-a9d4-6cec3a6c7163\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" Oct 30 18:10:55.016315 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.015482 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2a970c-4563-416f-9357-379fbec2f852-credential-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff\" (UID: \"ec2a970c-4563-416f-9357-379fbec2f852\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" Oct 30 18:10:55.016315 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.016284 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07931e20-5652-4e4c-a9d4-6cec3a6c7163-credential-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj\" (UID: \"07931e20-5652-4e4c-a9d4-6cec3a6c7163\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" Oct 30 18:10:55.016603 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.016552 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2a970c-4563-416f-9357-379fbec2f852-workload-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff\" (UID: \"ec2a970c-4563-416f-9357-379fbec2f852\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" Oct 30 18:10:55.016945 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.016921 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07931e20-5652-4e4c-a9d4-6cec3a6c7163-workload-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj\" (UID: \"07931e20-5652-4e4c-a9d4-6cec3a6c7163\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" Oct 30 18:10:55.017158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.017109 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07931e20-5652-4e4c-a9d4-6cec3a6c7163-workload-certs\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj\" (UID: \"07931e20-5652-4e4c-a9d4-6cec3a6c7163\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" Oct 30 18:10:55.018294 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.018267 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec2a970c-4563-416f-9357-379fbec2f852-istio-podinfo\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff\" (UID: \"ec2a970c-4563-416f-9357-379fbec2f852\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" Oct 30 18:10:55.018387 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.018349 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec2a970c-4563-416f-9357-379fbec2f852-istio-envoy\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff\" (UID: \"ec2a970c-4563-416f-9357-379fbec2f852\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" Oct 30 18:10:55.018542 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.018510 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07931e20-5652-4e4c-a9d4-6cec3a6c7163-istio-podinfo\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj\" (UID: \"07931e20-5652-4e4c-a9d4-6cec3a6c7163\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" Oct 30 18:10:55.018654 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.018638 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07931e20-5652-4e4c-a9d4-6cec3a6c7163-istio-envoy\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj\" (UID: \"07931e20-5652-4e4c-a9d4-6cec3a6c7163\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" Oct 30 18:10:55.034351 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.034324 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec2a970c-4563-416f-9357-379fbec2f852-istio-token\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff\" (UID: \"ec2a970c-4563-416f-9357-379fbec2f852\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" Oct 30 18:10:55.034886 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.034868 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkf2l\" (UniqueName: \"kubernetes.io/projected/ec2a970c-4563-416f-9357-379fbec2f852-kube-api-access-mkf2l\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff\" (UID: \"ec2a970c-4563-416f-9357-379fbec2f852\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" Oct 30 18:10:55.035549 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.035520 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07931e20-5652-4e4c-a9d4-6cec3a6c7163-istio-token\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj\" (UID: \"07931e20-5652-4e4c-a9d4-6cec3a6c7163\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" Oct 30 18:10:55.035815 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.035795 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mgqt\" (UniqueName: \"kubernetes.io/projected/07931e20-5652-4e4c-a9d4-6cec3a6c7163-kube-api-access-9mgqt\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj\" (UID: \"07931e20-5652-4e4c-a9d4-6cec3a6c7163\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" Oct 30 18:10:55.042622 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.042601 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" Oct 30 18:10:55.094863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.094826 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" Oct 30 18:10:55.370147 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.370106 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj"] Oct 30 18:10:55.372853 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:10:55.372812 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07931e20_5652_4e4c_a9d4_6cec3a6c7163.slice/crio-d22bec5580cfc782a71804f39395e14733f4b0b2847f9c7592827bafb0a4bbf9 WatchSource:0}: Error finding container d22bec5580cfc782a71804f39395e14733f4b0b2847f9c7592827bafb0a4bbf9: Status 404 returned error can't find the container with id d22bec5580cfc782a71804f39395e14733f4b0b2847f9c7592827bafb0a4bbf9 Oct 30 18:10:55.375569 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.375528 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:10:55.375703 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.375604 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:10:55.375703 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.375639 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:10:55.411586 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.411559 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff"] Oct 30 18:10:55.413663 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:10:55.413628 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec2a970c_4563_416f_9357_379fbec2f852.slice/crio-a334c72f59a941b5c4aef154f49bc1328b7ae345f21459d9fe734797d6fa6b99 WatchSource:0}: Error finding container a334c72f59a941b5c4aef154f49bc1328b7ae345f21459d9fe734797d6fa6b99: Status 404 returned error can't find the container with id a334c72f59a941b5c4aef154f49bc1328b7ae345f21459d9fe734797d6fa6b99 Oct 30 18:10:55.416809 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.416775 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:10:55.416897 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.416846 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:10:55.416897 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.416879 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:10:55.591086 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.589735 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" event={"ID":"ec2a970c-4563-416f-9357-379fbec2f852","Type":"ContainerStarted","Data":"81b63abe77385480a2f2d4e0a273e4d4d98eb61dd19f8ca7e7ef46b21e33dffc"} Oct 30 18:10:55.591086 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.589794 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" event={"ID":"ec2a970c-4563-416f-9357-379fbec2f852","Type":"ContainerStarted","Data":"a334c72f59a941b5c4aef154f49bc1328b7ae345f21459d9fe734797d6fa6b99"} Oct 30 18:10:55.591606 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.591575 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" event={"ID":"07931e20-5652-4e4c-a9d4-6cec3a6c7163","Type":"ContainerStarted","Data":"fa87941b15d713233cd7be7cb46cfc6416cc195c2dcf625ae02c31e329384b97"} Oct 30 18:10:55.591731 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.591627 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" event={"ID":"07931e20-5652-4e4c-a9d4-6cec3a6c7163","Type":"ContainerStarted","Data":"d22bec5580cfc782a71804f39395e14733f4b0b2847f9c7592827bafb0a4bbf9"} Oct 30 18:10:55.637987 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.636070 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" podStartSLOduration=1.6360555780000001 podStartE2EDuration="1.636055578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:55.634095871 +0000 UTC m=+752.724158239" watchObservedRunningTime="2026-10-30 18:10:55.636055578 +0000 UTC m=+752.726117984" Oct 30 18:10:55.677514 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:55.677450 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" podStartSLOduration=1.6774285230000001 podStartE2EDuration="1.677428523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:55.675381951 +0000 UTC m=+752.765444289" watchObservedRunningTime="2026-10-30 18:10:55.677428523 +0000 UTC m=+752.767490864" Oct 30 18:10:56.045578 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:56.045492 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" Oct 30 18:10:56.056292 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:56.056257 4442 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:10:56.057535 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:56.057506 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" podUID="07931e20-5652-4e4c-a9d4-6cec3a6c7163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 18:10:56.096242 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:56.096200 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" Oct 30 18:10:56.100451 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:56.097813 4442 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 18:10:56.100451 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:56.097870 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" podUID="ec2a970c-4563-416f-9357-379fbec2f852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 18:10:57.043440 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:57.043402 4442 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:10:57.043759 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:57.043725 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" podUID="07931e20-5652-4e4c-a9d4-6cec3a6c7163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 18:10:57.095941 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:57.095905 4442 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 18:10:57.096363 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:57.095974 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" podUID="ec2a970c-4563-416f-9357-379fbec2f852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 18:10:58.048167 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:58.047978 4442 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:10:58.048167 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:58.048041 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" podUID="07931e20-5652-4e4c-a9d4-6cec3a6c7163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 18:10:58.095675 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:58.095640 4442 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 18:10:58.095847 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:58.095698 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" podUID="ec2a970c-4563-416f-9357-379fbec2f852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 18:10:59.046733 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:59.046697 4442 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:10:59.047518 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:59.047315 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" podUID="07931e20-5652-4e4c-a9d4-6cec3a6c7163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 18:10:59.095852 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:59.095818 4442 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 18:10:59.096167 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:10:59.096107 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" podUID="ec2a970c-4563-416f-9357-379fbec2f852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 18:11:00.047156 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:00.046546 4442 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:11:00.047156 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:00.046613 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" podUID="07931e20-5652-4e4c-a9d4-6cec3a6c7163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 18:11:00.095903 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:00.095857 4442 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 18:11:00.096082 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:00.095936 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" podUID="ec2a970c-4563-416f-9357-379fbec2f852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 18:11:01.055159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:01.054433 4442 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:11:01.055159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:01.054494 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" podUID="07931e20-5652-4e4c-a9d4-6cec3a6c7163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 18:11:01.095619 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:01.095522 4442 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 18:11:01.095619 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:01.095575 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" podUID="ec2a970c-4563-416f-9357-379fbec2f852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 18:11:02.104728 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:02.103527 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" Oct 30 18:11:02.104728 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:02.104669 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" Oct 30 18:11:02.105554 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:02.105515 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" Oct 30 18:11:02.228193 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:02.228162 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" Oct 30 18:11:02.229336 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:02.229298 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" Oct 30 18:11:02.267155 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:02.246434 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" Oct 30 18:11:09.781228 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:09.781127 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp"] Oct 30 18:11:09.786779 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:09.786746 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" Oct 30 18:11:09.805874 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:09.805720 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp"] Oct 30 18:11:09.810403 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:09.810367 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fae7458-0960-4008-b3ae-79d4518ea89e-istiod-ca-cert\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp\" (UID: \"5fae7458-0960-4008-b3ae-79d4518ea89e\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" Oct 30 18:11:09.810584 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:09.810428 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fae7458-0960-4008-b3ae-79d4518ea89e-istio-envoy\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp\" (UID: \"5fae7458-0960-4008-b3ae-79d4518ea89e\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" Oct 30 18:11:09.810584 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:09.810452 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r59zc\" (UniqueName: \"kubernetes.io/projected/5fae7458-0960-4008-b3ae-79d4518ea89e-kube-api-access-r59zc\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp\" (UID: \"5fae7458-0960-4008-b3ae-79d4518ea89e\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" Oct 30 18:11:09.810584 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:09.810512 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fae7458-0960-4008-b3ae-79d4518ea89e-istio-token\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp\" (UID: \"5fae7458-0960-4008-b3ae-79d4518ea89e\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" Oct 30 18:11:09.810765 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:09.810606 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fae7458-0960-4008-b3ae-79d4518ea89e-workload-certs\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp\" (UID: \"5fae7458-0960-4008-b3ae-79d4518ea89e\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" Oct 30 18:11:09.810765 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:09.810632 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fae7458-0960-4008-b3ae-79d4518ea89e-istio-data\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp\" (UID: \"5fae7458-0960-4008-b3ae-79d4518ea89e\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" Oct 30 18:11:09.810765 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:09.810660 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fae7458-0960-4008-b3ae-79d4518ea89e-istio-podinfo\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp\" (UID: \"5fae7458-0960-4008-b3ae-79d4518ea89e\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" Oct 30 18:11:09.810765 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:09.810707 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fae7458-0960-4008-b3ae-79d4518ea89e-workload-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp\" (UID: \"5fae7458-0960-4008-b3ae-79d4518ea89e\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" Oct 30 18:11:09.810765 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:09.810747 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fae7458-0960-4008-b3ae-79d4518ea89e-credential-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp\" (UID: \"5fae7458-0960-4008-b3ae-79d4518ea89e\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" Oct 30 18:11:09.912320 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:09.912280 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fae7458-0960-4008-b3ae-79d4518ea89e-workload-certs\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp\" (UID: \"5fae7458-0960-4008-b3ae-79d4518ea89e\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" Oct 30 18:11:09.913003 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:09.912973 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fae7458-0960-4008-b3ae-79d4518ea89e-istio-data\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp\" (UID: \"5fae7458-0960-4008-b3ae-79d4518ea89e\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" Oct 30 18:11:09.913128 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:09.913017 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fae7458-0960-4008-b3ae-79d4518ea89e-istio-podinfo\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp\" (UID: \"5fae7458-0960-4008-b3ae-79d4518ea89e\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" Oct 30 18:11:09.913128 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:09.913061 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fae7458-0960-4008-b3ae-79d4518ea89e-workload-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp\" (UID: \"5fae7458-0960-4008-b3ae-79d4518ea89e\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" Oct 30 18:11:09.913128 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:09.913093 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fae7458-0960-4008-b3ae-79d4518ea89e-credential-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp\" (UID: \"5fae7458-0960-4008-b3ae-79d4518ea89e\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" Oct 30 18:11:09.913346 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:09.913155 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fae7458-0960-4008-b3ae-79d4518ea89e-workload-certs\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp\" (UID: \"5fae7458-0960-4008-b3ae-79d4518ea89e\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" Oct 30 18:11:09.913346 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:09.913179 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fae7458-0960-4008-b3ae-79d4518ea89e-istiod-ca-cert\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp\" (UID: \"5fae7458-0960-4008-b3ae-79d4518ea89e\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" Oct 30 18:11:09.913346 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:09.913215 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fae7458-0960-4008-b3ae-79d4518ea89e-istio-envoy\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp\" (UID: \"5fae7458-0960-4008-b3ae-79d4518ea89e\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" Oct 30 18:11:09.913346 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:09.913242 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r59zc\" (UniqueName: \"kubernetes.io/projected/5fae7458-0960-4008-b3ae-79d4518ea89e-kube-api-access-r59zc\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp\" (UID: \"5fae7458-0960-4008-b3ae-79d4518ea89e\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" Oct 30 18:11:09.913346 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:09.913309 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fae7458-0960-4008-b3ae-79d4518ea89e-istio-token\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp\" (UID: \"5fae7458-0960-4008-b3ae-79d4518ea89e\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" Oct 30 18:11:09.913346 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:09.913333 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fae7458-0960-4008-b3ae-79d4518ea89e-istio-data\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp\" (UID: \"5fae7458-0960-4008-b3ae-79d4518ea89e\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" Oct 30 18:11:09.914000 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:09.913408 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fae7458-0960-4008-b3ae-79d4518ea89e-credential-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp\" (UID: \"5fae7458-0960-4008-b3ae-79d4518ea89e\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" Oct 30 18:11:09.914000 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:09.913493 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fae7458-0960-4008-b3ae-79d4518ea89e-workload-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp\" (UID: \"5fae7458-0960-4008-b3ae-79d4518ea89e\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" Oct 30 18:11:09.914105 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:09.914085 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fae7458-0960-4008-b3ae-79d4518ea89e-istiod-ca-cert\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp\" (UID: \"5fae7458-0960-4008-b3ae-79d4518ea89e\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" Oct 30 18:11:09.915814 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:09.915787 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fae7458-0960-4008-b3ae-79d4518ea89e-istio-envoy\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp\" (UID: \"5fae7458-0960-4008-b3ae-79d4518ea89e\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" Oct 30 18:11:09.916426 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:09.916410 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fae7458-0960-4008-b3ae-79d4518ea89e-istio-podinfo\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp\" (UID: \"5fae7458-0960-4008-b3ae-79d4518ea89e\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" Oct 30 18:11:09.932243 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:09.932217 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fae7458-0960-4008-b3ae-79d4518ea89e-istio-token\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp\" (UID: \"5fae7458-0960-4008-b3ae-79d4518ea89e\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" Oct 30 18:11:09.932743 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:09.932718 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r59zc\" (UniqueName: \"kubernetes.io/projected/5fae7458-0960-4008-b3ae-79d4518ea89e-kube-api-access-r59zc\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp\" (UID: \"5fae7458-0960-4008-b3ae-79d4518ea89e\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" Oct 30 18:11:10.102455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:10.102377 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" Oct 30 18:11:10.426848 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:10.426806 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp"] Oct 30 18:11:10.429602 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:11:10.429577 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fae7458_0960_4008_b3ae_79d4518ea89e.slice/crio-5d60b5e6329dac95d3e408cbf38b062a648f96c0e6c15a2710cf1d3215218c1b WatchSource:0}: Error finding container 5d60b5e6329dac95d3e408cbf38b062a648f96c0e6c15a2710cf1d3215218c1b: Status 404 returned error can't find the container with id 5d60b5e6329dac95d3e408cbf38b062a648f96c0e6c15a2710cf1d3215218c1b Oct 30 18:11:10.432350 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:10.432312 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:11:10.432497 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:10.432410 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:11:10.432497 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:10.432460 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:11:10.723741 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:10.721888 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" event={"ID":"5fae7458-0960-4008-b3ae-79d4518ea89e","Type":"ContainerStarted","Data":"c861c8220d8ad843ba519bc95760067264d3f944a9e4b1493ba8e3eb54fc56fa"} Oct 30 18:11:10.723741 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:10.721992 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" event={"ID":"5fae7458-0960-4008-b3ae-79d4518ea89e","Type":"ContainerStarted","Data":"5d60b5e6329dac95d3e408cbf38b062a648f96c0e6c15a2710cf1d3215218c1b"} Oct 30 18:11:10.767099 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:10.767028 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" podStartSLOduration=1.767007472 podStartE2EDuration="1.767007472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:10.762984494 +0000 UTC m=+767.853046848" watchObservedRunningTime="2026-10-30 18:11:10.767007472 +0000 UTC m=+767.857069811" Oct 30 18:11:11.103612 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:11.103523 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" Oct 30 18:11:11.105266 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:11.105235 4442 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:11:11.105404 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:11.105293 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" podUID="5fae7458-0960-4008-b3ae-79d4518ea89e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 18:11:12.103224 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:12.103190 4442 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:11:12.103404 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:12.103244 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" podUID="5fae7458-0960-4008-b3ae-79d4518ea89e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 18:11:13.103437 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:13.103403 4442 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:11:13.104002 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:13.103466 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" podUID="5fae7458-0960-4008-b3ae-79d4518ea89e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 18:11:14.103474 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:14.103431 4442 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:11:14.103942 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:14.103502 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" podUID="5fae7458-0960-4008-b3ae-79d4518ea89e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 18:11:15.103426 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:15.103392 4442 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:11:15.103606 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:15.103443 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" podUID="5fae7458-0960-4008-b3ae-79d4518ea89e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 18:11:16.103767 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:16.103727 4442 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:11:16.104250 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:16.103802 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" podUID="5fae7458-0960-4008-b3ae-79d4518ea89e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 18:11:17.434713 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:17.434681 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" Oct 30 18:11:17.435691 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:17.435664 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" Oct 30 18:11:17.436683 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:17.436662 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" Oct 30 18:11:17.572559 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:11:17.572520 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:11:38.215962 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:38.215926 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n"] Oct 30 18:11:38.220148 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:38.220116 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" Oct 30 18:11:38.234588 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:38.234562 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-921450-openshift-default-dockercfg-rt5dw\"" Oct 30 18:11:38.250318 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:38.250286 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n"] Oct 30 18:11:38.351382 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:38.351351 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7084397f-85eb-43f9-8314-530af31c81c8-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n\" (UID: \"7084397f-85eb-43f9-8314-530af31c81c8\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" Oct 30 18:11:38.351585 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:38.351390 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7084397f-85eb-43f9-8314-530af31c81c8-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n\" (UID: \"7084397f-85eb-43f9-8314-530af31c81c8\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" Oct 30 18:11:38.351585 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:38.351429 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7084397f-85eb-43f9-8314-530af31c81c8-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n\" (UID: \"7084397f-85eb-43f9-8314-530af31c81c8\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" Oct 30 18:11:38.351725 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:38.351654 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7084397f-85eb-43f9-8314-530af31c81c8-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n\" (UID: \"7084397f-85eb-43f9-8314-530af31c81c8\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" Oct 30 18:11:38.351725 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:38.351716 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7084397f-85eb-43f9-8314-530af31c81c8-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n\" (UID: \"7084397f-85eb-43f9-8314-530af31c81c8\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" Oct 30 18:11:38.351837 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:38.351812 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w55w\" (UniqueName: \"kubernetes.io/projected/7084397f-85eb-43f9-8314-530af31c81c8-kube-api-access-9w55w\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n\" (UID: \"7084397f-85eb-43f9-8314-530af31c81c8\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" Oct 30 18:11:38.351881 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:38.351861 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7084397f-85eb-43f9-8314-530af31c81c8-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n\" (UID: \"7084397f-85eb-43f9-8314-530af31c81c8\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" Oct 30 18:11:38.351963 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:38.351949 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7084397f-85eb-43f9-8314-530af31c81c8-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n\" (UID: \"7084397f-85eb-43f9-8314-530af31c81c8\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" Oct 30 18:11:38.352020 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:38.351977 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7084397f-85eb-43f9-8314-530af31c81c8-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n\" (UID: \"7084397f-85eb-43f9-8314-530af31c81c8\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" Oct 30 18:11:38.453510 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:38.453475 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7084397f-85eb-43f9-8314-530af31c81c8-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n\" (UID: \"7084397f-85eb-43f9-8314-530af31c81c8\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" Oct 30 18:11:38.453711 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:38.453551 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7084397f-85eb-43f9-8314-530af31c81c8-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n\" (UID: \"7084397f-85eb-43f9-8314-530af31c81c8\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" Oct 30 18:11:38.453711 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:38.453577 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7084397f-85eb-43f9-8314-530af31c81c8-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n\" (UID: \"7084397f-85eb-43f9-8314-530af31c81c8\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" Oct 30 18:11:38.453711 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:38.453603 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w55w\" (UniqueName: \"kubernetes.io/projected/7084397f-85eb-43f9-8314-530af31c81c8-kube-api-access-9w55w\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n\" (UID: \"7084397f-85eb-43f9-8314-530af31c81c8\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" Oct 30 18:11:38.453711 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:38.453623 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7084397f-85eb-43f9-8314-530af31c81c8-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n\" (UID: \"7084397f-85eb-43f9-8314-530af31c81c8\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" Oct 30 18:11:38.453711 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:38.453643 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7084397f-85eb-43f9-8314-530af31c81c8-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n\" (UID: \"7084397f-85eb-43f9-8314-530af31c81c8\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" Oct 30 18:11:38.453711 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:38.453665 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7084397f-85eb-43f9-8314-530af31c81c8-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n\" (UID: \"7084397f-85eb-43f9-8314-530af31c81c8\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" Oct 30 18:11:38.454031 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:38.453734 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7084397f-85eb-43f9-8314-530af31c81c8-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n\" (UID: \"7084397f-85eb-43f9-8314-530af31c81c8\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" Oct 30 18:11:38.454031 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:38.453773 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7084397f-85eb-43f9-8314-530af31c81c8-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n\" (UID: \"7084397f-85eb-43f9-8314-530af31c81c8\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" Oct 30 18:11:38.454163 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:38.454111 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7084397f-85eb-43f9-8314-530af31c81c8-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n\" (UID: \"7084397f-85eb-43f9-8314-530af31c81c8\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" Oct 30 18:11:38.454231 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:38.454198 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7084397f-85eb-43f9-8314-530af31c81c8-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n\" (UID: \"7084397f-85eb-43f9-8314-530af31c81c8\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" Oct 30 18:11:38.454359 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:38.454326 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7084397f-85eb-43f9-8314-530af31c81c8-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n\" (UID: \"7084397f-85eb-43f9-8314-530af31c81c8\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" Oct 30 18:11:38.454545 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:38.454519 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7084397f-85eb-43f9-8314-530af31c81c8-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n\" (UID: \"7084397f-85eb-43f9-8314-530af31c81c8\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" Oct 30 18:11:38.454889 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:38.454865 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7084397f-85eb-43f9-8314-530af31c81c8-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n\" (UID: \"7084397f-85eb-43f9-8314-530af31c81c8\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" Oct 30 18:11:38.456478 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:38.456447 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7084397f-85eb-43f9-8314-530af31c81c8-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n\" (UID: \"7084397f-85eb-43f9-8314-530af31c81c8\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" Oct 30 18:11:38.456588 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:38.456451 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7084397f-85eb-43f9-8314-530af31c81c8-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n\" (UID: \"7084397f-85eb-43f9-8314-530af31c81c8\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" Oct 30 18:11:38.472847 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:38.472776 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7084397f-85eb-43f9-8314-530af31c81c8-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n\" (UID: \"7084397f-85eb-43f9-8314-530af31c81c8\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" Oct 30 18:11:38.473227 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:38.473207 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w55w\" (UniqueName: \"kubernetes.io/projected/7084397f-85eb-43f9-8314-530af31c81c8-kube-api-access-9w55w\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n\" (UID: \"7084397f-85eb-43f9-8314-530af31c81c8\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" Oct 30 18:11:38.537415 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:38.537374 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" Oct 30 18:11:38.854302 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:38.854270 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n"] Oct 30 18:11:38.855280 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:11:38.855255 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7084397f_85eb_43f9_8314_530af31c81c8.slice/crio-73ec8eb807c38641ef3be1df3523f8f84f0bca72f8b4fd36e918644f60a72440 WatchSource:0}: Error finding container 73ec8eb807c38641ef3be1df3523f8f84f0bca72f8b4fd36e918644f60a72440: Status 404 returned error can't find the container with id 73ec8eb807c38641ef3be1df3523f8f84f0bca72f8b4fd36e918644f60a72440 Oct 30 18:11:38.858482 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:38.858442 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:11:38.858603 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:38.858515 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:11:38.858603 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:38.858546 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:11:38.960930 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:38.960894 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" event={"ID":"7084397f-85eb-43f9-8314-530af31c81c8","Type":"ContainerStarted","Data":"cdbdf5c245588d68a0c29b38270a6b0f6c182f8b99256f6a7ed6a8512dedbc57"} Oct 30 18:11:38.961192 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:38.960950 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" event={"ID":"7084397f-85eb-43f9-8314-530af31c81c8","Type":"ContainerStarted","Data":"73ec8eb807c38641ef3be1df3523f8f84f0bca72f8b4fd36e918644f60a72440"} Oct 30 18:11:39.003243 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:39.003184 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" podStartSLOduration=1.003165411 podStartE2EDuration="1.003165411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:39.000736147 +0000 UTC m=+796.090798518" watchObservedRunningTime="2026-10-30 18:11:39.003165411 +0000 UTC m=+796.093227752" Oct 30 18:11:39.537981 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:39.537942 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" Oct 30 18:11:39.539487 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:39.539461 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:11:39.539615 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:39.539510 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" podUID="7084397f-85eb-43f9-8314-530af31c81c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 18:11:40.547160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:40.541550 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:11:40.547160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:40.541616 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" podUID="7084397f-85eb-43f9-8314-530af31c81c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 18:11:41.544158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:41.543562 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:11:41.544158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:41.543626 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" podUID="7084397f-85eb-43f9-8314-530af31c81c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 18:11:42.548158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:42.544517 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:11:42.548158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:42.544616 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" podUID="7084397f-85eb-43f9-8314-530af31c81c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 18:11:43.540564 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:43.540526 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:11:43.540871 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:43.540841 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" podUID="7084397f-85eb-43f9-8314-530af31c81c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 18:11:44.545552 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:44.545516 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:11:44.546191 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:44.546161 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" podUID="7084397f-85eb-43f9-8314-530af31c81c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 18:11:45.538660 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:45.538615 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:11:45.538846 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:45.538699 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" podUID="7084397f-85eb-43f9-8314-530af31c81c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 18:11:46.543321 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:46.543286 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" Oct 30 18:11:46.543800 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:46.543683 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" Oct 30 18:11:46.544877 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:46.544857 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" Oct 30 18:11:47.565118 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:11:47.565072 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:11:56.354193 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:56.354150 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9"] Oct 30 18:11:56.360033 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:56.360012 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" Oct 30 18:11:56.374033 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:56.374005 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-fe1374-openshift-default-dockercfg-dg2jw\"" Oct 30 18:11:56.389800 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:56.389774 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9"] Oct 30 18:11:56.465210 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:56.465180 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a1649e-7af6-4854-bb2f-0c44af4b0276-credential-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9\" (UID: \"18a1649e-7af6-4854-bb2f-0c44af4b0276\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" Oct 30 18:11:56.465210 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:56.465212 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18a1649e-7af6-4854-bb2f-0c44af4b0276-istiod-ca-cert\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9\" (UID: \"18a1649e-7af6-4854-bb2f-0c44af4b0276\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" Oct 30 18:11:56.465449 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:56.465244 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18a1649e-7af6-4854-bb2f-0c44af4b0276-istio-envoy\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9\" (UID: \"18a1649e-7af6-4854-bb2f-0c44af4b0276\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" Oct 30 18:11:56.465449 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:56.465369 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtj9z\" (UniqueName: \"kubernetes.io/projected/18a1649e-7af6-4854-bb2f-0c44af4b0276-kube-api-access-rtj9z\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9\" (UID: \"18a1649e-7af6-4854-bb2f-0c44af4b0276\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" Oct 30 18:11:56.465449 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:56.465437 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18a1649e-7af6-4854-bb2f-0c44af4b0276-workload-certs\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9\" (UID: \"18a1649e-7af6-4854-bb2f-0c44af4b0276\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" Oct 30 18:11:56.465569 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:56.465458 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18a1649e-7af6-4854-bb2f-0c44af4b0276-istio-token\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9\" (UID: \"18a1649e-7af6-4854-bb2f-0c44af4b0276\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" Oct 30 18:11:56.465569 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:56.465514 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18a1649e-7af6-4854-bb2f-0c44af4b0276-istio-data\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9\" (UID: \"18a1649e-7af6-4854-bb2f-0c44af4b0276\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" Oct 30 18:11:56.465635 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:56.465575 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18a1649e-7af6-4854-bb2f-0c44af4b0276-istio-podinfo\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9\" (UID: \"18a1649e-7af6-4854-bb2f-0c44af4b0276\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" Oct 30 18:11:56.465681 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:56.465633 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a1649e-7af6-4854-bb2f-0c44af4b0276-workload-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9\" (UID: \"18a1649e-7af6-4854-bb2f-0c44af4b0276\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" Oct 30 18:11:56.566658 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:56.566622 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18a1649e-7af6-4854-bb2f-0c44af4b0276-istio-data\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9\" (UID: \"18a1649e-7af6-4854-bb2f-0c44af4b0276\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" Oct 30 18:11:56.566839 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:56.566668 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18a1649e-7af6-4854-bb2f-0c44af4b0276-istio-podinfo\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9\" (UID: \"18a1649e-7af6-4854-bb2f-0c44af4b0276\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" Oct 30 18:11:56.566839 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:56.566744 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a1649e-7af6-4854-bb2f-0c44af4b0276-workload-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9\" (UID: \"18a1649e-7af6-4854-bb2f-0c44af4b0276\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" Oct 30 18:11:56.566839 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:56.566808 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a1649e-7af6-4854-bb2f-0c44af4b0276-credential-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9\" (UID: \"18a1649e-7af6-4854-bb2f-0c44af4b0276\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" Oct 30 18:11:56.566839 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:56.566832 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18a1649e-7af6-4854-bb2f-0c44af4b0276-istiod-ca-cert\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9\" (UID: \"18a1649e-7af6-4854-bb2f-0c44af4b0276\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" Oct 30 18:11:56.567072 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:56.566875 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18a1649e-7af6-4854-bb2f-0c44af4b0276-istio-envoy\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9\" (UID: \"18a1649e-7af6-4854-bb2f-0c44af4b0276\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" Oct 30 18:11:56.567072 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:56.566939 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtj9z\" (UniqueName: \"kubernetes.io/projected/18a1649e-7af6-4854-bb2f-0c44af4b0276-kube-api-access-rtj9z\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9\" (UID: \"18a1649e-7af6-4854-bb2f-0c44af4b0276\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" Oct 30 18:11:56.567072 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:56.566978 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18a1649e-7af6-4854-bb2f-0c44af4b0276-workload-certs\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9\" (UID: \"18a1649e-7af6-4854-bb2f-0c44af4b0276\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" Oct 30 18:11:56.567072 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:56.567000 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18a1649e-7af6-4854-bb2f-0c44af4b0276-istio-token\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9\" (UID: \"18a1649e-7af6-4854-bb2f-0c44af4b0276\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" Oct 30 18:11:56.567334 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:56.567097 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18a1649e-7af6-4854-bb2f-0c44af4b0276-istio-data\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9\" (UID: \"18a1649e-7af6-4854-bb2f-0c44af4b0276\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" Oct 30 18:11:56.567334 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:56.567206 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a1649e-7af6-4854-bb2f-0c44af4b0276-workload-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9\" (UID: \"18a1649e-7af6-4854-bb2f-0c44af4b0276\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" Oct 30 18:11:56.567565 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:56.567541 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a1649e-7af6-4854-bb2f-0c44af4b0276-credential-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9\" (UID: \"18a1649e-7af6-4854-bb2f-0c44af4b0276\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" Oct 30 18:11:56.567845 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:56.567824 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18a1649e-7af6-4854-bb2f-0c44af4b0276-workload-certs\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9\" (UID: \"18a1649e-7af6-4854-bb2f-0c44af4b0276\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" Oct 30 18:11:56.568296 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:56.568275 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18a1649e-7af6-4854-bb2f-0c44af4b0276-istiod-ca-cert\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9\" (UID: \"18a1649e-7af6-4854-bb2f-0c44af4b0276\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" Oct 30 18:11:56.569475 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:56.569454 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18a1649e-7af6-4854-bb2f-0c44af4b0276-istio-envoy\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9\" (UID: \"18a1649e-7af6-4854-bb2f-0c44af4b0276\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" Oct 30 18:11:56.569782 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:56.569766 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18a1649e-7af6-4854-bb2f-0c44af4b0276-istio-podinfo\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9\" (UID: \"18a1649e-7af6-4854-bb2f-0c44af4b0276\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" Oct 30 18:11:56.585702 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:56.585672 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18a1649e-7af6-4854-bb2f-0c44af4b0276-istio-token\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9\" (UID: \"18a1649e-7af6-4854-bb2f-0c44af4b0276\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" Oct 30 18:11:56.585898 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:56.585876 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtj9z\" (UniqueName: \"kubernetes.io/projected/18a1649e-7af6-4854-bb2f-0c44af4b0276-kube-api-access-rtj9z\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9\" (UID: \"18a1649e-7af6-4854-bb2f-0c44af4b0276\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" Oct 30 18:11:56.675373 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:56.675336 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" Oct 30 18:11:57.002494 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:57.002415 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9"] Oct 30 18:11:57.012007 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:57.011974 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:11:57.012094 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:57.012045 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:11:57.012094 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:57.012075 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:11:57.110056 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:57.110025 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" event={"ID":"18a1649e-7af6-4854-bb2f-0c44af4b0276","Type":"ContainerStarted","Data":"68eaa3648c083d54d9a33805cbc64cae442acb31f8c60e8591eb28c578b05d4b"} Oct 30 18:11:57.110265 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:57.110083 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" event={"ID":"18a1649e-7af6-4854-bb2f-0c44af4b0276","Type":"ContainerStarted","Data":"fe204c87278aafe243953793e710ddb687d3b5ff6341567ecca09bba394621d4"} Oct 30 18:11:57.150589 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:57.150528 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" podStartSLOduration=1.150511194 podStartE2EDuration="1.150511194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:57.149278146 +0000 UTC m=+814.239340487" watchObservedRunningTime="2026-10-30 18:11:57.150511194 +0000 UTC m=+814.240573532" Oct 30 18:11:57.676337 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:57.676301 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" Oct 30 18:11:58.677353 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:58.677316 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 18:11:58.677738 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:58.677375 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" podUID="18a1649e-7af6-4854-bb2f-0c44af4b0276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 18:11:59.675983 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:59.675949 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 18:11:59.676186 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:11:59.676002 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" podUID="18a1649e-7af6-4854-bb2f-0c44af4b0276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 18:12:00.676185 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:00.676124 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 18:12:00.676601 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:00.676221 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" podUID="18a1649e-7af6-4854-bb2f-0c44af4b0276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 18:12:01.676523 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:01.676481 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 18:12:01.677311 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:01.676557 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" podUID="18a1649e-7af6-4854-bb2f-0c44af4b0276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 18:12:02.675972 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:02.675923 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 18:12:02.676214 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:02.675997 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" podUID="18a1649e-7af6-4854-bb2f-0c44af4b0276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 18:12:03.871822 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:03.871789 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" Oct 30 18:12:03.872393 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:03.872370 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" Oct 30 18:12:03.873659 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:03.873627 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" Oct 30 18:12:07.286836 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:07.286797 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv"] Oct 30 18:12:07.291513 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:07.291488 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" Oct 30 18:12:07.317726 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:07.317689 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-380c08-openshift-default-dockercfg-74pzf\"" Oct 30 18:12:07.321382 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:07.321357 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv"] Oct 30 18:12:07.342716 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:07.342676 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d7355a2-0978-497c-8229-f5a0f6ae194d-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv\" (UID: \"8d7355a2-0978-497c-8229-f5a0f6ae194d\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" Oct 30 18:12:07.342942 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:07.342742 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d7355a2-0978-497c-8229-f5a0f6ae194d-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv\" (UID: \"8d7355a2-0978-497c-8229-f5a0f6ae194d\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" Oct 30 18:12:07.342942 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:07.342763 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d7355a2-0978-497c-8229-f5a0f6ae194d-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv\" (UID: \"8d7355a2-0978-497c-8229-f5a0f6ae194d\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" Oct 30 18:12:07.342942 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:07.342783 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwfqq\" (UniqueName: \"kubernetes.io/projected/8d7355a2-0978-497c-8229-f5a0f6ae194d-kube-api-access-xwfqq\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv\" (UID: \"8d7355a2-0978-497c-8229-f5a0f6ae194d\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" Oct 30 18:12:07.342942 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:07.342871 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7355a2-0978-497c-8229-f5a0f6ae194d-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv\" (UID: \"8d7355a2-0978-497c-8229-f5a0f6ae194d\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" Oct 30 18:12:07.342942 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:07.342889 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d7355a2-0978-497c-8229-f5a0f6ae194d-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv\" (UID: \"8d7355a2-0978-497c-8229-f5a0f6ae194d\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" Oct 30 18:12:07.343268 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:07.342961 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d7355a2-0978-497c-8229-f5a0f6ae194d-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv\" (UID: \"8d7355a2-0978-497c-8229-f5a0f6ae194d\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" Oct 30 18:12:07.343268 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:07.343017 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d7355a2-0978-497c-8229-f5a0f6ae194d-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv\" (UID: \"8d7355a2-0978-497c-8229-f5a0f6ae194d\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" Oct 30 18:12:07.343268 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:07.343052 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7355a2-0978-497c-8229-f5a0f6ae194d-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv\" (UID: \"8d7355a2-0978-497c-8229-f5a0f6ae194d\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" Oct 30 18:12:07.443688 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:07.443636 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d7355a2-0978-497c-8229-f5a0f6ae194d-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv\" (UID: \"8d7355a2-0978-497c-8229-f5a0f6ae194d\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" Oct 30 18:12:07.443884 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:07.443708 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7355a2-0978-497c-8229-f5a0f6ae194d-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv\" (UID: \"8d7355a2-0978-497c-8229-f5a0f6ae194d\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" Oct 30 18:12:07.443884 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:07.443739 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d7355a2-0978-497c-8229-f5a0f6ae194d-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv\" (UID: \"8d7355a2-0978-497c-8229-f5a0f6ae194d\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" Oct 30 18:12:07.443884 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:07.443789 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d7355a2-0978-497c-8229-f5a0f6ae194d-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv\" (UID: \"8d7355a2-0978-497c-8229-f5a0f6ae194d\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" Oct 30 18:12:07.443884 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:07.443820 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d7355a2-0978-497c-8229-f5a0f6ae194d-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv\" (UID: \"8d7355a2-0978-497c-8229-f5a0f6ae194d\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" Oct 30 18:12:07.443884 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:07.443852 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwfqq\" (UniqueName: \"kubernetes.io/projected/8d7355a2-0978-497c-8229-f5a0f6ae194d-kube-api-access-xwfqq\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv\" (UID: \"8d7355a2-0978-497c-8229-f5a0f6ae194d\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" Oct 30 18:12:07.444190 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:07.444064 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7355a2-0978-497c-8229-f5a0f6ae194d-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv\" (UID: \"8d7355a2-0978-497c-8229-f5a0f6ae194d\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" Oct 30 18:12:07.444190 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:07.444101 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d7355a2-0978-497c-8229-f5a0f6ae194d-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv\" (UID: \"8d7355a2-0978-497c-8229-f5a0f6ae194d\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" Oct 30 18:12:07.444289 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:07.444230 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d7355a2-0978-497c-8229-f5a0f6ae194d-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv\" (UID: \"8d7355a2-0978-497c-8229-f5a0f6ae194d\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" Oct 30 18:12:07.444520 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:07.444486 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d7355a2-0978-497c-8229-f5a0f6ae194d-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv\" (UID: \"8d7355a2-0978-497c-8229-f5a0f6ae194d\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" Oct 30 18:12:07.444593 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:07.444563 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d7355a2-0978-497c-8229-f5a0f6ae194d-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv\" (UID: \"8d7355a2-0978-497c-8229-f5a0f6ae194d\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" Oct 30 18:12:07.444744 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:07.444713 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7355a2-0978-497c-8229-f5a0f6ae194d-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv\" (UID: \"8d7355a2-0978-497c-8229-f5a0f6ae194d\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" Oct 30 18:12:07.444817 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:07.444778 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d7355a2-0978-497c-8229-f5a0f6ae194d-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv\" (UID: \"8d7355a2-0978-497c-8229-f5a0f6ae194d\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" Oct 30 18:12:07.444998 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:07.444979 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7355a2-0978-497c-8229-f5a0f6ae194d-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv\" (UID: \"8d7355a2-0978-497c-8229-f5a0f6ae194d\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" Oct 30 18:12:07.446966 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:07.446943 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d7355a2-0978-497c-8229-f5a0f6ae194d-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv\" (UID: \"8d7355a2-0978-497c-8229-f5a0f6ae194d\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" Oct 30 18:12:07.447072 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:07.446945 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d7355a2-0978-497c-8229-f5a0f6ae194d-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv\" (UID: \"8d7355a2-0978-497c-8229-f5a0f6ae194d\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" Oct 30 18:12:07.462908 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:07.462872 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d7355a2-0978-497c-8229-f5a0f6ae194d-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv\" (UID: \"8d7355a2-0978-497c-8229-f5a0f6ae194d\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" Oct 30 18:12:07.462908 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:07.462892 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwfqq\" (UniqueName: \"kubernetes.io/projected/8d7355a2-0978-497c-8229-f5a0f6ae194d-kube-api-access-xwfqq\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv\" (UID: \"8d7355a2-0978-497c-8229-f5a0f6ae194d\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" Oct 30 18:12:07.606648 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:07.606566 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" Oct 30 18:12:07.928056 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:07.928027 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv"] Oct 30 18:12:07.930249 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:12:07.930220 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d7355a2_0978_497c_8229_f5a0f6ae194d.slice/crio-3ab1f863cfb0e225fe85c36cb2ecdd12795718c8b791a37070903d254ac67513 WatchSource:0}: Error finding container 3ab1f863cfb0e225fe85c36cb2ecdd12795718c8b791a37070903d254ac67513: Status 404 returned error can't find the container with id 3ab1f863cfb0e225fe85c36cb2ecdd12795718c8b791a37070903d254ac67513 Oct 30 18:12:07.932686 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:07.932656 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:12:07.932795 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:07.932722 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:12:07.932795 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:07.932748 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:12:08.211015 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:08.210922 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" event={"ID":"8d7355a2-0978-497c-8229-f5a0f6ae194d","Type":"ContainerStarted","Data":"a8c9f2a9f1eb89996d239e8291466951c267e05bacb6ffa947340ae80ec9d58d"} Oct 30 18:12:08.211015 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:08.210978 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" event={"ID":"8d7355a2-0978-497c-8229-f5a0f6ae194d","Type":"ContainerStarted","Data":"3ab1f863cfb0e225fe85c36cb2ecdd12795718c8b791a37070903d254ac67513"} Oct 30 18:12:08.606998 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:08.606916 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" Oct 30 18:12:09.609328 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:09.609295 4442 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:12:09.609709 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:09.609359 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" podUID="8d7355a2-0978-497c-8229-f5a0f6ae194d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 18:12:10.607122 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:10.607088 4442 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:12:10.607315 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:10.607170 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" podUID="8d7355a2-0978-497c-8229-f5a0f6ae194d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 18:12:11.607696 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:11.607647 4442 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:12:11.608125 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:11.607711 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" podUID="8d7355a2-0978-497c-8229-f5a0f6ae194d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 18:12:12.607471 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:12.607439 4442 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:12:12.607777 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:12.607495 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" podUID="8d7355a2-0978-497c-8229-f5a0f6ae194d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 18:12:13.607581 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:13.607547 4442 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:12:13.607818 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:13.607599 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" podUID="8d7355a2-0978-497c-8229-f5a0f6ae194d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 18:12:14.608066 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:14.608024 4442 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:12:14.608592 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:14.608094 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" podUID="8d7355a2-0978-497c-8229-f5a0f6ae194d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 18:12:15.618172 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:15.618116 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" Oct 30 18:12:15.618797 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:15.618771 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" Oct 30 18:12:15.619640 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:15.619620 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" Oct 30 18:12:15.685183 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:15.685101 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" podStartSLOduration=8.685086951 podStartE2EDuration="8.685086951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:12:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:12:08.251409105 +0000 UTC m=+825.341471448" watchObservedRunningTime="2026-10-30 18:12:15.685086951 +0000 UTC m=+832.775149359" Oct 30 18:12:16.337161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:16.330120 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8"] Oct 30 18:12:16.342124 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:16.342091 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" Oct 30 18:12:16.360348 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:16.360316 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8"] Oct 30 18:12:16.374677 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:16.371723 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-de7e3a-openshift-default-dockercfg-p5ml9\"" Oct 30 18:12:16.512159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:16.505731 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8ac1217-0b8a-4d72-907f-c70814b60a55-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8\" (UID: \"e8ac1217-0b8a-4d72-907f-c70814b60a55\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" Oct 30 18:12:16.512159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:16.505856 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8ac1217-0b8a-4d72-907f-c70814b60a55-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8\" (UID: \"e8ac1217-0b8a-4d72-907f-c70814b60a55\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" Oct 30 18:12:16.512159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:16.505889 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8ac1217-0b8a-4d72-907f-c70814b60a55-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8\" (UID: \"e8ac1217-0b8a-4d72-907f-c70814b60a55\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" Oct 30 18:12:16.512159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:16.505915 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8ac1217-0b8a-4d72-907f-c70814b60a55-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8\" (UID: \"e8ac1217-0b8a-4d72-907f-c70814b60a55\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" Oct 30 18:12:16.512159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:16.505969 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8ac1217-0b8a-4d72-907f-c70814b60a55-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8\" (UID: \"e8ac1217-0b8a-4d72-907f-c70814b60a55\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" Oct 30 18:12:16.512159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:16.506067 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8ac1217-0b8a-4d72-907f-c70814b60a55-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8\" (UID: \"e8ac1217-0b8a-4d72-907f-c70814b60a55\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" Oct 30 18:12:16.512159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:16.506101 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4pj8\" (UniqueName: \"kubernetes.io/projected/e8ac1217-0b8a-4d72-907f-c70814b60a55-kube-api-access-d4pj8\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8\" (UID: \"e8ac1217-0b8a-4d72-907f-c70814b60a55\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" Oct 30 18:12:16.533160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:16.518397 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8ac1217-0b8a-4d72-907f-c70814b60a55-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8\" (UID: \"e8ac1217-0b8a-4d72-907f-c70814b60a55\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" Oct 30 18:12:16.537157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:16.533446 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8ac1217-0b8a-4d72-907f-c70814b60a55-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8\" (UID: \"e8ac1217-0b8a-4d72-907f-c70814b60a55\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" Oct 30 18:12:16.641741 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:16.641708 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8ac1217-0b8a-4d72-907f-c70814b60a55-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8\" (UID: \"e8ac1217-0b8a-4d72-907f-c70814b60a55\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" Oct 30 18:12:16.642353 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:16.642327 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8ac1217-0b8a-4d72-907f-c70814b60a55-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8\" (UID: \"e8ac1217-0b8a-4d72-907f-c70814b60a55\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" Oct 30 18:12:16.642488 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:16.642470 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8ac1217-0b8a-4d72-907f-c70814b60a55-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8\" (UID: \"e8ac1217-0b8a-4d72-907f-c70814b60a55\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" Oct 30 18:12:16.642633 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:16.642617 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8ac1217-0b8a-4d72-907f-c70814b60a55-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8\" (UID: \"e8ac1217-0b8a-4d72-907f-c70814b60a55\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" Oct 30 18:12:16.642785 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:16.642769 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8ac1217-0b8a-4d72-907f-c70814b60a55-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8\" (UID: \"e8ac1217-0b8a-4d72-907f-c70814b60a55\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" Oct 30 18:12:16.642909 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:16.642891 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4pj8\" (UniqueName: \"kubernetes.io/projected/e8ac1217-0b8a-4d72-907f-c70814b60a55-kube-api-access-d4pj8\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8\" (UID: \"e8ac1217-0b8a-4d72-907f-c70814b60a55\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" Oct 30 18:12:16.643100 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:16.643082 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8ac1217-0b8a-4d72-907f-c70814b60a55-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8\" (UID: \"e8ac1217-0b8a-4d72-907f-c70814b60a55\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" Oct 30 18:12:16.643285 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:16.643250 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8ac1217-0b8a-4d72-907f-c70814b60a55-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8\" (UID: \"e8ac1217-0b8a-4d72-907f-c70814b60a55\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" Oct 30 18:12:16.643439 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:16.643422 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8ac1217-0b8a-4d72-907f-c70814b60a55-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8\" (UID: \"e8ac1217-0b8a-4d72-907f-c70814b60a55\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" Oct 30 18:12:16.644061 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:16.644038 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8ac1217-0b8a-4d72-907f-c70814b60a55-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8\" (UID: \"e8ac1217-0b8a-4d72-907f-c70814b60a55\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" Oct 30 18:12:16.644964 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:16.644941 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8ac1217-0b8a-4d72-907f-c70814b60a55-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8\" (UID: \"e8ac1217-0b8a-4d72-907f-c70814b60a55\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" Oct 30 18:12:16.676823 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:16.660588 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8ac1217-0b8a-4d72-907f-c70814b60a55-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8\" (UID: \"e8ac1217-0b8a-4d72-907f-c70814b60a55\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" Oct 30 18:12:16.676823 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:16.661866 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8ac1217-0b8a-4d72-907f-c70814b60a55-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8\" (UID: \"e8ac1217-0b8a-4d72-907f-c70814b60a55\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" Oct 30 18:12:16.677902 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:16.677864 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8ac1217-0b8a-4d72-907f-c70814b60a55-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8\" (UID: \"e8ac1217-0b8a-4d72-907f-c70814b60a55\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" Oct 30 18:12:16.678654 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:16.678634 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8ac1217-0b8a-4d72-907f-c70814b60a55-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8\" (UID: \"e8ac1217-0b8a-4d72-907f-c70814b60a55\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" Oct 30 18:12:16.685173 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:16.681417 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8ac1217-0b8a-4d72-907f-c70814b60a55-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8\" (UID: \"e8ac1217-0b8a-4d72-907f-c70814b60a55\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" Oct 30 18:12:16.698882 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:16.698818 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4pj8\" (UniqueName: \"kubernetes.io/projected/e8ac1217-0b8a-4d72-907f-c70814b60a55-kube-api-access-d4pj8\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8\" (UID: \"e8ac1217-0b8a-4d72-907f-c70814b60a55\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" Oct 30 18:12:16.711835 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:16.711778 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8ac1217-0b8a-4d72-907f-c70814b60a55-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8\" (UID: \"e8ac1217-0b8a-4d72-907f-c70814b60a55\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" Oct 30 18:12:16.808268 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:16.808236 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" Oct 30 18:12:17.190717 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:17.190631 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8"] Oct 30 18:12:17.196538 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:12:17.196494 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8ac1217_0b8a_4d72_907f_c70814b60a55.slice/crio-45d2ef2aa9572bd2d1041e3d204b93c621b4b757d35a9049d0bfd8e84bf61fde WatchSource:0}: Error finding container 45d2ef2aa9572bd2d1041e3d204b93c621b4b757d35a9049d0bfd8e84bf61fde: Status 404 returned error can't find the container with id 45d2ef2aa9572bd2d1041e3d204b93c621b4b757d35a9049d0bfd8e84bf61fde Oct 30 18:12:17.199192 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:17.199157 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:12:17.199284 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:17.199230 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:12:17.199284 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:17.199273 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:12:17.394752 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:17.394714 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" event={"ID":"e8ac1217-0b8a-4d72-907f-c70814b60a55","Type":"ContainerStarted","Data":"41add33f7103e2c9fe91e53405c66e78c993031def0b6872ad593dcddd3a88c9"} Oct 30 18:12:17.394950 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:17.394777 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" event={"ID":"e8ac1217-0b8a-4d72-907f-c70814b60a55","Type":"ContainerStarted","Data":"45d2ef2aa9572bd2d1041e3d204b93c621b4b757d35a9049d0bfd8e84bf61fde"} Oct 30 18:12:17.451488 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:17.451361 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" podStartSLOduration=1.451338238 podStartE2EDuration="1.451338238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:12:17.450458592 +0000 UTC m=+834.540520950" watchObservedRunningTime="2026-10-30 18:12:17.451338238 +0000 UTC m=+834.541400579" Oct 30 18:12:17.565651 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:12:17.565619 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:12:17.809350 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:17.809268 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" Oct 30 18:12:18.810557 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:18.810528 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 18:12:18.810945 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:18.810580 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" podUID="e8ac1217-0b8a-4d72-907f-c70814b60a55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 18:12:19.810072 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:19.810033 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 18:12:19.810294 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:19.810090 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" podUID="e8ac1217-0b8a-4d72-907f-c70814b60a55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 18:12:20.809638 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:20.809606 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 18:12:20.810036 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:20.809672 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" podUID="e8ac1217-0b8a-4d72-907f-c70814b60a55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 18:12:21.809540 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:21.809500 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 18:12:21.809757 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:21.809564 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" podUID="e8ac1217-0b8a-4d72-907f-c70814b60a55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 18:12:22.810170 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:22.810103 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 18:12:22.810576 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:22.810180 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" podUID="e8ac1217-0b8a-4d72-907f-c70814b60a55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 18:12:23.947905 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:23.947867 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" Oct 30 18:12:23.948974 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:23.948944 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" Oct 30 18:12:23.950540 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:23.950516 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" Oct 30 18:12:47.231807 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:12:47.231771 4442 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-176.ec2.internal" podUID="9e81bd78300430e51b01d30caa1123e7" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded" Oct 30 18:12:47.561122 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:12:47.561037 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:13:17.560149 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:13:17.560086 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:13:23.181558 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:13:23.181527 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52rn6_516c6b6c-fc40-49cf-93b5-1c4471b02cf0/ovn-acl-logging/0.log" Oct 30 18:13:23.182911 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:13:23.182887 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52rn6_516c6b6c-fc40-49cf-93b5-1c4471b02cf0/ovn-acl-logging/0.log" Oct 30 18:13:47.560276 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:13:47.560226 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:14:11.908685 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:11.908609 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8"] Oct 30 18:14:11.912503 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:11.912485 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8" Oct 30 18:14:11.932204 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:11.932126 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8"] Oct 30 18:14:11.938440 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:11.938378 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-de7e3a\"/\"kube-root-ca.crt\"" Oct 30 18:14:11.938440 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:11.938408 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-de7e3a\"/\"openshift-service-ca.crt\"" Oct 30 18:14:11.944272 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:11.944251 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-de7e3a\"/\"distinct-2-w3-de7e3a-kserve-self-signed-certs\"" Oct 30 18:14:11.944272 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:11.944251 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-de7e3a\"/\"default-dockercfg-p66x8\"" Oct 30 18:14:12.026515 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:12.026482 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e2901e8-4919-4e5c-88c7-39126a7cca56-model-cache\") pod \"distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8\" (UID: \"1e2901e8-4919-4e5c-88c7-39126a7cca56\") " pod="e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8" Oct 30 18:14:12.026515 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:12.026524 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx52v\" (UniqueName: \"kubernetes.io/projected/1e2901e8-4919-4e5c-88c7-39126a7cca56-kube-api-access-zx52v\") pod \"distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8\" (UID: \"1e2901e8-4919-4e5c-88c7-39126a7cca56\") " pod="e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8" Oct 30 18:14:12.026777 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:12.026546 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e2901e8-4919-4e5c-88c7-39126a7cca56-tmp-dir\") pod \"distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8\" (UID: \"1e2901e8-4919-4e5c-88c7-39126a7cca56\") " pod="e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8" Oct 30 18:14:12.026777 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:12.026724 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e2901e8-4919-4e5c-88c7-39126a7cca56-dshm\") pod \"distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8\" (UID: \"1e2901e8-4919-4e5c-88c7-39126a7cca56\") " pod="e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8" Oct 30 18:14:12.026777 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:12.026753 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e2901e8-4919-4e5c-88c7-39126a7cca56-home\") pod \"distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8\" (UID: \"1e2901e8-4919-4e5c-88c7-39126a7cca56\") " pod="e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8" Oct 30 18:14:12.026935 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:12.026835 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e2901e8-4919-4e5c-88c7-39126a7cca56-tls-certs\") pod \"distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8\" (UID: \"1e2901e8-4919-4e5c-88c7-39126a7cca56\") " pod="e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8" Oct 30 18:14:12.130826 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:12.130780 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e2901e8-4919-4e5c-88c7-39126a7cca56-tls-certs\") pod \"distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8\" (UID: \"1e2901e8-4919-4e5c-88c7-39126a7cca56\") " pod="e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8" Oct 30 18:14:12.133436 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:12.132951 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e2901e8-4919-4e5c-88c7-39126a7cca56-model-cache\") pod \"distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8\" (UID: \"1e2901e8-4919-4e5c-88c7-39126a7cca56\") " pod="e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8" Oct 30 18:14:12.133689 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:12.133394 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e2901e8-4919-4e5c-88c7-39126a7cca56-model-cache\") pod \"distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8\" (UID: \"1e2901e8-4919-4e5c-88c7-39126a7cca56\") " pod="e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8" Oct 30 18:14:12.133903 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:12.133883 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx52v\" (UniqueName: \"kubernetes.io/projected/1e2901e8-4919-4e5c-88c7-39126a7cca56-kube-api-access-zx52v\") pod \"distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8\" (UID: \"1e2901e8-4919-4e5c-88c7-39126a7cca56\") " pod="e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8" Oct 30 18:14:12.134554 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:12.134534 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e2901e8-4919-4e5c-88c7-39126a7cca56-tmp-dir\") pod \"distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8\" (UID: \"1e2901e8-4919-4e5c-88c7-39126a7cca56\") " pod="e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8" Oct 30 18:14:12.138335 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:12.138312 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e2901e8-4919-4e5c-88c7-39126a7cca56-dshm\") pod \"distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8\" (UID: \"1e2901e8-4919-4e5c-88c7-39126a7cca56\") " pod="e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8" Oct 30 18:14:12.154191 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:12.138483 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e2901e8-4919-4e5c-88c7-39126a7cca56-home\") pod \"distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8\" (UID: \"1e2901e8-4919-4e5c-88c7-39126a7cca56\") " pod="e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8" Oct 30 18:14:12.154935 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:12.143844 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e2901e8-4919-4e5c-88c7-39126a7cca56-home\") pod \"distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8\" (UID: \"1e2901e8-4919-4e5c-88c7-39126a7cca56\") " pod="e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8" Oct 30 18:14:12.155113 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:12.134945 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e2901e8-4919-4e5c-88c7-39126a7cca56-tmp-dir\") pod \"distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8\" (UID: \"1e2901e8-4919-4e5c-88c7-39126a7cca56\") " pod="e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8" Oct 30 18:14:12.155265 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:12.150389 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e2901e8-4919-4e5c-88c7-39126a7cca56-dshm\") pod \"distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8\" (UID: \"1e2901e8-4919-4e5c-88c7-39126a7cca56\") " pod="e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8" Oct 30 18:14:12.179204 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:12.169352 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e2901e8-4919-4e5c-88c7-39126a7cca56-tls-certs\") pod \"distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8\" (UID: \"1e2901e8-4919-4e5c-88c7-39126a7cca56\") " pod="e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8" Oct 30 18:14:12.183183 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:12.180878 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx52v\" (UniqueName: \"kubernetes.io/projected/1e2901e8-4919-4e5c-88c7-39126a7cca56-kube-api-access-zx52v\") pod \"distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8\" (UID: \"1e2901e8-4919-4e5c-88c7-39126a7cca56\") " pod="e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8" Oct 30 18:14:12.239802 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:12.239766 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8" Oct 30 18:14:12.546749 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:12.546714 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8"] Oct 30 18:14:12.547950 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:14:12.547923 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e2901e8_4919_4e5c_88c7_39126a7cca56.slice/crio-6bd36c17b22b4339f27badd03092ccb20cd63afddb7f40db4c8f11c6ea0eadfa WatchSource:0}: Error finding container 6bd36c17b22b4339f27badd03092ccb20cd63afddb7f40db4c8f11c6ea0eadfa: Status 404 returned error can't find the container with id 6bd36c17b22b4339f27badd03092ccb20cd63afddb7f40db4c8f11c6ea0eadfa Oct 30 18:14:12.571806 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:12.571754 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8" event={"ID":"1e2901e8-4919-4e5c-88c7-39126a7cca56","Type":"ContainerStarted","Data":"6bd36c17b22b4339f27badd03092ccb20cd63afddb7f40db4c8f11c6ea0eadfa"} Oct 30 18:14:13.579431 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:13.579398 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8" event={"ID":"1e2901e8-4919-4e5c-88c7-39126a7cca56","Type":"ContainerStarted","Data":"a57bd7f7932f4112640be3d875fce6ae10248d7fa9cb1301e325a2e52a810dda"} Oct 30 18:14:13.618677 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:13.618628 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8" podStartSLOduration=2.6186148190000003 podStartE2EDuration="2.618614819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:13.617190409 +0000 UTC m=+950.707252750" watchObservedRunningTime="2026-10-30 18:14:13.618614819 +0000 UTC m=+950.708677179" Oct 30 18:14:14.240737 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:14.240699 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8" Oct 30 18:14:17.559692 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:14:17.559650 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:14:22.240974 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:22.240932 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8" Oct 30 18:14:22.254439 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:22.254411 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8" Oct 30 18:14:22.670419 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:22.670386 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8" Oct 30 18:14:23.714214 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:23.714183 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789"] Oct 30 18:14:23.743161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:23.731167 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" Oct 30 18:14:23.778953 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:23.778922 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789"] Oct 30 18:14:23.831468 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:23.830472 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wd789\" (UID: \"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" Oct 30 18:14:23.831468 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:23.830537 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wd789\" (UID: \"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" Oct 30 18:14:23.831468 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:23.830630 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wd789\" (UID: \"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" Oct 30 18:14:23.831468 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:23.830657 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4dgk\" (UniqueName: \"kubernetes.io/projected/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-kube-api-access-z4dgk\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wd789\" (UID: \"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" Oct 30 18:14:23.831468 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:23.830754 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wd789\" (UID: \"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" Oct 30 18:14:23.831468 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:23.830781 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wd789\" (UID: \"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" Oct 30 18:14:23.831468 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:23.830834 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wd789\" (UID: \"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" Oct 30 18:14:23.831468 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:23.830884 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wd789\" (UID: \"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" Oct 30 18:14:23.831468 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:23.830974 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wd789\" (UID: \"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" Oct 30 18:14:23.932437 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:23.932404 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wd789\" (UID: \"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" Oct 30 18:14:23.932615 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:23.932464 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wd789\" (UID: \"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" Oct 30 18:14:23.932615 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:23.932592 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4dgk\" (UniqueName: \"kubernetes.io/projected/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-kube-api-access-z4dgk\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wd789\" (UID: \"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" Oct 30 18:14:23.932736 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:23.932710 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wd789\" (UID: \"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" Oct 30 18:14:23.932829 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:23.932744 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wd789\" (UID: \"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" Oct 30 18:14:23.932829 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:23.932794 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wd789\" (UID: \"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" Oct 30 18:14:23.932947 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:23.932839 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wd789\" (UID: \"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" Oct 30 18:14:23.932947 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:23.932874 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wd789\" (UID: \"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" Oct 30 18:14:23.932947 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:23.932910 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wd789\" (UID: \"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" Oct 30 18:14:23.933097 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:23.932968 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wd789\" (UID: \"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" Oct 30 18:14:23.933177 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:23.933152 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wd789\" (UID: \"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" Oct 30 18:14:23.933247 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:23.933229 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wd789\" (UID: \"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" Oct 30 18:14:23.933420 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:23.933403 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wd789\" (UID: \"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" Oct 30 18:14:23.933691 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:23.933665 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wd789\" (UID: \"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" Oct 30 18:14:23.935742 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:23.935719 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wd789\" (UID: \"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" Oct 30 18:14:23.935914 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:23.935891 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wd789\" (UID: \"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" Oct 30 18:14:23.990208 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:23.973643 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wd789\" (UID: \"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" Oct 30 18:14:23.991924 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:23.988325 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4dgk\" (UniqueName: \"kubernetes.io/projected/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-kube-api-access-z4dgk\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wd789\" (UID: \"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" Oct 30 18:14:24.124886 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:24.124843 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" Oct 30 18:14:24.460880 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:24.460840 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789"] Oct 30 18:14:24.480378 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:24.480329 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:14:24.480483 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:24.480429 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:14:24.480529 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:24.480478 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:14:24.715398 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:24.714519 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" event={"ID":"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42","Type":"ContainerStarted","Data":"84431ef14fb17489f30111c3a32731796835c1e6d3fd6cdb900e025731d19026"} Oct 30 18:14:24.715398 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:24.714586 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" event={"ID":"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42","Type":"ContainerStarted","Data":"8140051ce436ca7bf3c3fa3836c182cc404b4617f6e29e58ec072189940bcf22"} Oct 30 18:14:24.814797 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:24.814732 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" podStartSLOduration=1.814711155 podStartE2EDuration="1.814711155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:24.813656771 +0000 UTC m=+961.903719113" watchObservedRunningTime="2026-10-30 18:14:24.814711155 +0000 UTC m=+961.904773495" Oct 30 18:14:25.146158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:25.145872 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" Oct 30 18:14:25.146158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:25.146082 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:14:25.146429 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:25.146164 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" podUID="b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 18:14:26.132159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:26.127519 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:14:26.132159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:26.127579 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" podUID="b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 18:14:27.144158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:27.132536 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:14:27.144158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:27.132611 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" podUID="b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 18:14:28.126669 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:28.126633 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:14:28.127025 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:28.126976 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" podUID="b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 18:14:29.131161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:29.127522 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:14:29.131161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:29.127585 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" podUID="b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 18:14:30.126172 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:30.125495 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:14:30.126172 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:30.125555 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" podUID="b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 18:14:31.125610 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:31.125573 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:14:31.126005 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:31.125632 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" podUID="b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 18:14:32.130062 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:32.130032 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" Oct 30 18:14:32.130578 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:32.130486 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" Oct 30 18:14:32.131264 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:32.131248 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" Oct 30 18:14:39.695719 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:39.695686 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4"] Oct 30 18:14:39.699859 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:39.699836 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4" Oct 30 18:14:39.721634 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:39.721604 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4"] Oct 30 18:14:39.728791 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:39.727874 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-921450\"/\"kube-root-ca.crt\"" Oct 30 18:14:39.728791 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:39.727998 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-921450\"/\"openshift-service-ca.crt\"" Oct 30 18:14:39.728791 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:39.728072 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-921450\"/\"default-dockercfg-nm4lf\"" Oct 30 18:14:39.728791 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:39.728167 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-921450\"/\"distinct-2-w0-921450-kserve-self-signed-certs\"" Oct 30 18:14:39.776748 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:39.776715 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf7d5f54-e091-4032-ab11-af2687e453c8-tls-certs\") pod \"distinct-2-w0-921450-kserve-784954bd5c-2ttw4\" (UID: \"cf7d5f54-e091-4032-ab11-af2687e453c8\") " pod="e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4" Oct 30 18:14:39.776937 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:39.776759 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf7d5f54-e091-4032-ab11-af2687e453c8-home\") pod \"distinct-2-w0-921450-kserve-784954bd5c-2ttw4\" (UID: \"cf7d5f54-e091-4032-ab11-af2687e453c8\") " pod="e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4" Oct 30 18:14:39.776937 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:39.776922 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d964n\" (UniqueName: \"kubernetes.io/projected/cf7d5f54-e091-4032-ab11-af2687e453c8-kube-api-access-d964n\") pod \"distinct-2-w0-921450-kserve-784954bd5c-2ttw4\" (UID: \"cf7d5f54-e091-4032-ab11-af2687e453c8\") " pod="e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4" Oct 30 18:14:39.777044 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:39.776970 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf7d5f54-e091-4032-ab11-af2687e453c8-dshm\") pod \"distinct-2-w0-921450-kserve-784954bd5c-2ttw4\" (UID: \"cf7d5f54-e091-4032-ab11-af2687e453c8\") " pod="e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4" Oct 30 18:14:39.777044 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:39.777019 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf7d5f54-e091-4032-ab11-af2687e453c8-tmp-dir\") pod \"distinct-2-w0-921450-kserve-784954bd5c-2ttw4\" (UID: \"cf7d5f54-e091-4032-ab11-af2687e453c8\") " pod="e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4" Oct 30 18:14:39.777182 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:39.777097 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf7d5f54-e091-4032-ab11-af2687e453c8-model-cache\") pod \"distinct-2-w0-921450-kserve-784954bd5c-2ttw4\" (UID: \"cf7d5f54-e091-4032-ab11-af2687e453c8\") " pod="e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4" Oct 30 18:14:39.878447 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:39.878396 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d964n\" (UniqueName: \"kubernetes.io/projected/cf7d5f54-e091-4032-ab11-af2687e453c8-kube-api-access-d964n\") pod \"distinct-2-w0-921450-kserve-784954bd5c-2ttw4\" (UID: \"cf7d5f54-e091-4032-ab11-af2687e453c8\") " pod="e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4" Oct 30 18:14:39.878447 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:39.878458 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf7d5f54-e091-4032-ab11-af2687e453c8-dshm\") pod \"distinct-2-w0-921450-kserve-784954bd5c-2ttw4\" (UID: \"cf7d5f54-e091-4032-ab11-af2687e453c8\") " pod="e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4" Oct 30 18:14:39.903278 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:39.878504 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf7d5f54-e091-4032-ab11-af2687e453c8-tmp-dir\") pod \"distinct-2-w0-921450-kserve-784954bd5c-2ttw4\" (UID: \"cf7d5f54-e091-4032-ab11-af2687e453c8\") " pod="e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4" Oct 30 18:14:39.903278 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:39.878536 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf7d5f54-e091-4032-ab11-af2687e453c8-model-cache\") pod \"distinct-2-w0-921450-kserve-784954bd5c-2ttw4\" (UID: \"cf7d5f54-e091-4032-ab11-af2687e453c8\") " pod="e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4" Oct 30 18:14:39.903278 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:39.878666 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf7d5f54-e091-4032-ab11-af2687e453c8-tls-certs\") pod \"distinct-2-w0-921450-kserve-784954bd5c-2ttw4\" (UID: \"cf7d5f54-e091-4032-ab11-af2687e453c8\") " pod="e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4" Oct 30 18:14:39.903278 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:39.878692 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf7d5f54-e091-4032-ab11-af2687e453c8-home\") pod \"distinct-2-w0-921450-kserve-784954bd5c-2ttw4\" (UID: \"cf7d5f54-e091-4032-ab11-af2687e453c8\") " pod="e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4" Oct 30 18:14:39.903278 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:39.879006 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf7d5f54-e091-4032-ab11-af2687e453c8-tmp-dir\") pod \"distinct-2-w0-921450-kserve-784954bd5c-2ttw4\" (UID: \"cf7d5f54-e091-4032-ab11-af2687e453c8\") " pod="e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4" Oct 30 18:14:39.903278 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:39.879312 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf7d5f54-e091-4032-ab11-af2687e453c8-model-cache\") pod \"distinct-2-w0-921450-kserve-784954bd5c-2ttw4\" (UID: \"cf7d5f54-e091-4032-ab11-af2687e453c8\") " pod="e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4" Oct 30 18:14:39.903278 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:39.879407 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf7d5f54-e091-4032-ab11-af2687e453c8-home\") pod \"distinct-2-w0-921450-kserve-784954bd5c-2ttw4\" (UID: \"cf7d5f54-e091-4032-ab11-af2687e453c8\") " pod="e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4" Oct 30 18:14:39.914159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:39.903710 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf7d5f54-e091-4032-ab11-af2687e453c8-dshm\") pod \"distinct-2-w0-921450-kserve-784954bd5c-2ttw4\" (UID: \"cf7d5f54-e091-4032-ab11-af2687e453c8\") " pod="e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4" Oct 30 18:14:39.947210 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:39.936820 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d964n\" (UniqueName: \"kubernetes.io/projected/cf7d5f54-e091-4032-ab11-af2687e453c8-kube-api-access-d964n\") pod \"distinct-2-w0-921450-kserve-784954bd5c-2ttw4\" (UID: \"cf7d5f54-e091-4032-ab11-af2687e453c8\") " pod="e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4" Oct 30 18:14:39.968164 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:39.948002 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf7d5f54-e091-4032-ab11-af2687e453c8-tls-certs\") pod \"distinct-2-w0-921450-kserve-784954bd5c-2ttw4\" (UID: \"cf7d5f54-e091-4032-ab11-af2687e453c8\") " pod="e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4" Oct 30 18:14:40.017713 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:40.017408 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4" Oct 30 18:14:40.325055 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:40.325028 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4"] Oct 30 18:14:40.327924 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:14:40.327894 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf7d5f54_e091_4032_ab11_af2687e453c8.slice/crio-63b8c4f7e77401768a7fb81b3a73f686b2dc3757b3e4d8aac2736897db7d8714 WatchSource:0}: Error finding container 63b8c4f7e77401768a7fb81b3a73f686b2dc3757b3e4d8aac2736897db7d8714: Status 404 returned error can't find the container with id 63b8c4f7e77401768a7fb81b3a73f686b2dc3757b3e4d8aac2736897db7d8714 Oct 30 18:14:41.012565 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:41.012524 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4" event={"ID":"cf7d5f54-e091-4032-ab11-af2687e453c8","Type":"ContainerStarted","Data":"2f553df37dc99256315d25087ceb4729bbe92e65039be46c311fcffbe3aeccd9"} Oct 30 18:14:41.012565 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:41.012567 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4" event={"ID":"cf7d5f54-e091-4032-ab11-af2687e453c8","Type":"ContainerStarted","Data":"63b8c4f7e77401768a7fb81b3a73f686b2dc3757b3e4d8aac2736897db7d8714"} Oct 30 18:14:41.019126 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:41.019102 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4" Oct 30 18:14:41.051819 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:41.051747 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4" podStartSLOduration=2.0517342 podStartE2EDuration="2.0517342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:41.050438468 +0000 UTC m=+978.140500800" watchObservedRunningTime="2026-10-30 18:14:41.0517342 +0000 UTC m=+978.141796539" Oct 30 18:14:41.832578 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:41.832533 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft"] Oct 30 18:14:41.838742 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:41.838719 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" Oct 30 18:14:41.878721 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:41.878687 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft"] Oct 30 18:14:41.920128 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:41.920091 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxp5n\" (UniqueName: \"kubernetes.io/projected/034e6427-0b95-4d7d-b6c9-05673f97a48a-kube-api-access-nxp5n\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft\" (UID: \"034e6427-0b95-4d7d-b6c9-05673f97a48a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" Oct 30 18:14:41.920330 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:41.920172 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/034e6427-0b95-4d7d-b6c9-05673f97a48a-istio-token\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft\" (UID: \"034e6427-0b95-4d7d-b6c9-05673f97a48a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" Oct 30 18:14:41.920330 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:41.920205 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/034e6427-0b95-4d7d-b6c9-05673f97a48a-workload-certs\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft\" (UID: \"034e6427-0b95-4d7d-b6c9-05673f97a48a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" Oct 30 18:14:41.920330 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:41.920240 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/034e6427-0b95-4d7d-b6c9-05673f97a48a-istio-envoy\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft\" (UID: \"034e6427-0b95-4d7d-b6c9-05673f97a48a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" Oct 30 18:14:41.920496 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:41.920323 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/034e6427-0b95-4d7d-b6c9-05673f97a48a-istiod-ca-cert\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft\" (UID: \"034e6427-0b95-4d7d-b6c9-05673f97a48a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" Oct 30 18:14:41.920496 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:41.920389 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/034e6427-0b95-4d7d-b6c9-05673f97a48a-istio-data\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft\" (UID: \"034e6427-0b95-4d7d-b6c9-05673f97a48a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" Oct 30 18:14:41.920496 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:41.920440 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/034e6427-0b95-4d7d-b6c9-05673f97a48a-credential-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft\" (UID: \"034e6427-0b95-4d7d-b6c9-05673f97a48a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" Oct 30 18:14:41.920640 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:41.920495 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/034e6427-0b95-4d7d-b6c9-05673f97a48a-istio-podinfo\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft\" (UID: \"034e6427-0b95-4d7d-b6c9-05673f97a48a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" Oct 30 18:14:41.920640 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:41.920581 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/034e6427-0b95-4d7d-b6c9-05673f97a48a-workload-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft\" (UID: \"034e6427-0b95-4d7d-b6c9-05673f97a48a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" Oct 30 18:14:42.022189 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:42.022157 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/034e6427-0b95-4d7d-b6c9-05673f97a48a-istio-data\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft\" (UID: \"034e6427-0b95-4d7d-b6c9-05673f97a48a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" Oct 30 18:14:42.022650 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:42.022199 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/034e6427-0b95-4d7d-b6c9-05673f97a48a-credential-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft\" (UID: \"034e6427-0b95-4d7d-b6c9-05673f97a48a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" Oct 30 18:14:42.022650 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:42.022216 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/034e6427-0b95-4d7d-b6c9-05673f97a48a-istio-podinfo\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft\" (UID: \"034e6427-0b95-4d7d-b6c9-05673f97a48a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" Oct 30 18:14:42.022650 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:42.022293 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/034e6427-0b95-4d7d-b6c9-05673f97a48a-workload-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft\" (UID: \"034e6427-0b95-4d7d-b6c9-05673f97a48a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" Oct 30 18:14:42.022650 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:42.022372 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxp5n\" (UniqueName: \"kubernetes.io/projected/034e6427-0b95-4d7d-b6c9-05673f97a48a-kube-api-access-nxp5n\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft\" (UID: \"034e6427-0b95-4d7d-b6c9-05673f97a48a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" Oct 30 18:14:42.022650 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:42.022407 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/034e6427-0b95-4d7d-b6c9-05673f97a48a-istio-token\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft\" (UID: \"034e6427-0b95-4d7d-b6c9-05673f97a48a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" Oct 30 18:14:42.022650 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:42.022446 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/034e6427-0b95-4d7d-b6c9-05673f97a48a-workload-certs\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft\" (UID: \"034e6427-0b95-4d7d-b6c9-05673f97a48a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" Oct 30 18:14:42.022650 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:42.022488 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/034e6427-0b95-4d7d-b6c9-05673f97a48a-istio-envoy\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft\" (UID: \"034e6427-0b95-4d7d-b6c9-05673f97a48a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" Oct 30 18:14:42.022650 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:42.022519 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/034e6427-0b95-4d7d-b6c9-05673f97a48a-istiod-ca-cert\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft\" (UID: \"034e6427-0b95-4d7d-b6c9-05673f97a48a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" Oct 30 18:14:42.022650 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:42.022629 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/034e6427-0b95-4d7d-b6c9-05673f97a48a-credential-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft\" (UID: \"034e6427-0b95-4d7d-b6c9-05673f97a48a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" Oct 30 18:14:42.023121 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:42.022664 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/034e6427-0b95-4d7d-b6c9-05673f97a48a-istio-data\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft\" (UID: \"034e6427-0b95-4d7d-b6c9-05673f97a48a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" Oct 30 18:14:42.023198 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:42.023123 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/034e6427-0b95-4d7d-b6c9-05673f97a48a-workload-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft\" (UID: \"034e6427-0b95-4d7d-b6c9-05673f97a48a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" Oct 30 18:14:42.023359 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:42.023264 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/034e6427-0b95-4d7d-b6c9-05673f97a48a-workload-certs\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft\" (UID: \"034e6427-0b95-4d7d-b6c9-05673f97a48a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" Oct 30 18:14:42.023612 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:42.023591 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/034e6427-0b95-4d7d-b6c9-05673f97a48a-istiod-ca-cert\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft\" (UID: \"034e6427-0b95-4d7d-b6c9-05673f97a48a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" Oct 30 18:14:42.025283 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:42.025254 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/034e6427-0b95-4d7d-b6c9-05673f97a48a-istio-envoy\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft\" (UID: \"034e6427-0b95-4d7d-b6c9-05673f97a48a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" Oct 30 18:14:42.025558 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:42.025537 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/034e6427-0b95-4d7d-b6c9-05673f97a48a-istio-podinfo\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft\" (UID: \"034e6427-0b95-4d7d-b6c9-05673f97a48a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" Oct 30 18:14:42.042149 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:42.042108 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/034e6427-0b95-4d7d-b6c9-05673f97a48a-istio-token\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft\" (UID: \"034e6427-0b95-4d7d-b6c9-05673f97a48a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" Oct 30 18:14:42.042792 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:42.042759 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxp5n\" (UniqueName: \"kubernetes.io/projected/034e6427-0b95-4d7d-b6c9-05673f97a48a-kube-api-access-nxp5n\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft\" (UID: \"034e6427-0b95-4d7d-b6c9-05673f97a48a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" Oct 30 18:14:42.154901 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:42.154855 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" Oct 30 18:14:42.475110 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:42.475081 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft"] Oct 30 18:14:42.516377 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:42.506249 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:14:42.516377 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:42.506342 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:14:42.516377 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:42.506385 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:14:43.033394 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:43.033359 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" event={"ID":"034e6427-0b95-4d7d-b6c9-05673f97a48a","Type":"ContainerStarted","Data":"dab9e5fecb7301368eaf8a03e5780bcf7583815446f8dd2a6945826d897b96f5"} Oct 30 18:14:43.034207 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:43.034187 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" event={"ID":"034e6427-0b95-4d7d-b6c9-05673f97a48a","Type":"ContainerStarted","Data":"610b21867b62745678032adfd2d9901772a68c0ae8c93c00e6bd98826ce13aab"} Oct 30 18:14:43.092970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:43.092911 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" podStartSLOduration=2.0928804 podStartE2EDuration="2.0928804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:43.088325274 +0000 UTC m=+980.178387613" watchObservedRunningTime="2026-10-30 18:14:43.0928804 +0000 UTC m=+980.182942740" Oct 30 18:14:43.155361 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:43.155323 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" Oct 30 18:14:43.157538 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:43.157507 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:14:43.157680 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:43.157580 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" podUID="034e6427-0b95-4d7d-b6c9-05673f97a48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 18:14:44.156380 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:44.156339 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:14:44.156850 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:44.156395 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" podUID="034e6427-0b95-4d7d-b6c9-05673f97a48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 18:14:45.156245 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:45.156210 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:14:45.156452 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:45.156266 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" podUID="034e6427-0b95-4d7d-b6c9-05673f97a48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 18:14:46.167160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:46.160180 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:14:46.176050 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:46.160246 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" podUID="034e6427-0b95-4d7d-b6c9-05673f97a48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 18:14:46.778039 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:46.777992 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw"] Oct 30 18:14:46.784627 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:46.784603 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw"] Oct 30 18:14:46.784922 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:46.784906 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" Oct 30 18:14:46.911930 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:46.911888 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8626d789-257d-46c2-8555-8af451ed6434-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw\" (UID: \"8626d789-257d-46c2-8555-8af451ed6434\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" Oct 30 18:14:46.911930 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:46.911932 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8626d789-257d-46c2-8555-8af451ed6434-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw\" (UID: \"8626d789-257d-46c2-8555-8af451ed6434\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" Oct 30 18:14:46.912216 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:46.912059 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8626d789-257d-46c2-8555-8af451ed6434-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw\" (UID: \"8626d789-257d-46c2-8555-8af451ed6434\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" Oct 30 18:14:46.912216 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:46.912172 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8626d789-257d-46c2-8555-8af451ed6434-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw\" (UID: \"8626d789-257d-46c2-8555-8af451ed6434\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" Oct 30 18:14:46.912328 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:46.912243 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8626d789-257d-46c2-8555-8af451ed6434-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw\" (UID: \"8626d789-257d-46c2-8555-8af451ed6434\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" Oct 30 18:14:46.912328 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:46.912297 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8626d789-257d-46c2-8555-8af451ed6434-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw\" (UID: \"8626d789-257d-46c2-8555-8af451ed6434\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" Oct 30 18:14:46.912426 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:46.912386 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8626d789-257d-46c2-8555-8af451ed6434-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw\" (UID: \"8626d789-257d-46c2-8555-8af451ed6434\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" Oct 30 18:14:46.912483 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:46.912440 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8626d789-257d-46c2-8555-8af451ed6434-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw\" (UID: \"8626d789-257d-46c2-8555-8af451ed6434\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" Oct 30 18:14:46.912536 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:46.912496 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmc8p\" (UniqueName: \"kubernetes.io/projected/8626d789-257d-46c2-8555-8af451ed6434-kube-api-access-dmc8p\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw\" (UID: \"8626d789-257d-46c2-8555-8af451ed6434\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" Oct 30 18:14:47.014237 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:47.014203 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8626d789-257d-46c2-8555-8af451ed6434-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw\" (UID: \"8626d789-257d-46c2-8555-8af451ed6434\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" Oct 30 18:14:47.014453 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:47.014246 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8626d789-257d-46c2-8555-8af451ed6434-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw\" (UID: \"8626d789-257d-46c2-8555-8af451ed6434\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" Oct 30 18:14:47.014453 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:47.014402 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8626d789-257d-46c2-8555-8af451ed6434-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw\" (UID: \"8626d789-257d-46c2-8555-8af451ed6434\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" Oct 30 18:14:47.014589 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:47.014500 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8626d789-257d-46c2-8555-8af451ed6434-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw\" (UID: \"8626d789-257d-46c2-8555-8af451ed6434\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" Oct 30 18:14:47.014675 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:47.014650 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8626d789-257d-46c2-8555-8af451ed6434-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw\" (UID: \"8626d789-257d-46c2-8555-8af451ed6434\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" Oct 30 18:14:47.014774 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:47.014752 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8626d789-257d-46c2-8555-8af451ed6434-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw\" (UID: \"8626d789-257d-46c2-8555-8af451ed6434\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" Oct 30 18:14:47.015229 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:47.015206 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8626d789-257d-46c2-8555-8af451ed6434-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw\" (UID: \"8626d789-257d-46c2-8555-8af451ed6434\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" Oct 30 18:14:47.015315 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:47.015237 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8626d789-257d-46c2-8555-8af451ed6434-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw\" (UID: \"8626d789-257d-46c2-8555-8af451ed6434\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" Oct 30 18:14:47.015315 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:47.015302 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8626d789-257d-46c2-8555-8af451ed6434-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw\" (UID: \"8626d789-257d-46c2-8555-8af451ed6434\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" Oct 30 18:14:47.015417 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:47.015400 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8626d789-257d-46c2-8555-8af451ed6434-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw\" (UID: \"8626d789-257d-46c2-8555-8af451ed6434\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" Oct 30 18:14:47.015471 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:47.015453 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8626d789-257d-46c2-8555-8af451ed6434-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw\" (UID: \"8626d789-257d-46c2-8555-8af451ed6434\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" Oct 30 18:14:47.015527 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:47.015507 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmc8p\" (UniqueName: \"kubernetes.io/projected/8626d789-257d-46c2-8555-8af451ed6434-kube-api-access-dmc8p\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw\" (UID: \"8626d789-257d-46c2-8555-8af451ed6434\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" Oct 30 18:14:47.015579 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:47.015541 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8626d789-257d-46c2-8555-8af451ed6434-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw\" (UID: \"8626d789-257d-46c2-8555-8af451ed6434\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" Oct 30 18:14:47.015652 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:47.015623 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8626d789-257d-46c2-8555-8af451ed6434-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw\" (UID: \"8626d789-257d-46c2-8555-8af451ed6434\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" Oct 30 18:14:47.017353 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:47.017329 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8626d789-257d-46c2-8555-8af451ed6434-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw\" (UID: \"8626d789-257d-46c2-8555-8af451ed6434\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" Oct 30 18:14:47.017850 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:47.017833 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8626d789-257d-46c2-8555-8af451ed6434-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw\" (UID: \"8626d789-257d-46c2-8555-8af451ed6434\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" Oct 30 18:14:47.060213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:47.047673 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8626d789-257d-46c2-8555-8af451ed6434-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw\" (UID: \"8626d789-257d-46c2-8555-8af451ed6434\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" Oct 30 18:14:47.060213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:47.048330 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmc8p\" (UniqueName: \"kubernetes.io/projected/8626d789-257d-46c2-8555-8af451ed6434-kube-api-access-dmc8p\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw\" (UID: \"8626d789-257d-46c2-8555-8af451ed6434\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" Oct 30 18:14:47.102727 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:47.102695 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" Oct 30 18:14:47.159566 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:47.159525 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:14:47.159790 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:47.159590 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" podUID="034e6427-0b95-4d7d-b6c9-05673f97a48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 18:14:47.459551 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:47.459519 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw"] Oct 30 18:14:47.461600 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:14:47.461574 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8626d789_257d_46c2_8555_8af451ed6434.slice/crio-5fca92dc668d0690698b2a5c28a07ce04b5c99d525dba893a1d63e563d1685df WatchSource:0}: Error finding container 5fca92dc668d0690698b2a5c28a07ce04b5c99d525dba893a1d63e563d1685df: Status 404 returned error can't find the container with id 5fca92dc668d0690698b2a5c28a07ce04b5c99d525dba893a1d63e563d1685df Oct 30 18:14:47.464273 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:47.464240 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:14:47.464387 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:47.464311 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:14:47.464387 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:47.464343 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:14:47.589616 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:14:47.589576 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:14:48.135168 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:48.133857 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" event={"ID":"8626d789-257d-46c2-8555-8af451ed6434","Type":"ContainerStarted","Data":"035b3424b7300cb83f722cfae01122c3113e4f67e6389a1cf0e44c2da1112f6d"} Oct 30 18:14:48.135168 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:48.133953 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" event={"ID":"8626d789-257d-46c2-8555-8af451ed6434","Type":"ContainerStarted","Data":"5fca92dc668d0690698b2a5c28a07ce04b5c99d525dba893a1d63e563d1685df"} Oct 30 18:14:48.165757 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:48.165718 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:14:48.166047 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:48.166018 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" podUID="034e6427-0b95-4d7d-b6c9-05673f97a48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 18:14:48.195224 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:48.195179 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" podStartSLOduration=2.19516566 podStartE2EDuration="2.19516566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:48.191334377 +0000 UTC m=+985.281396717" watchObservedRunningTime="2026-10-30 18:14:48.19516566 +0000 UTC m=+985.285227999" Oct 30 18:14:49.103449 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:49.103410 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" Oct 30 18:14:49.105279 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:49.105256 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:14:49.105389 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:49.105313 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" podUID="8626d789-257d-46c2-8555-8af451ed6434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 18:14:49.160529 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:49.160485 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:14:49.160744 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:49.160546 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" podUID="034e6427-0b95-4d7d-b6c9-05673f97a48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 18:14:50.018759 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:50.018723 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4" Oct 30 18:14:50.091625 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:50.091594 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4" Oct 30 18:14:50.103470 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:50.103441 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:14:50.103855 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:50.103502 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" podUID="8626d789-257d-46c2-8555-8af451ed6434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 18:14:50.182466 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:50.182444 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4" Oct 30 18:14:50.267713 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:50.267673 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" Oct 30 18:14:50.268923 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:50.268824 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" Oct 30 18:14:50.272666 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:50.272651 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" Oct 30 18:14:51.103975 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:51.103941 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:14:51.104435 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:51.104005 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" podUID="8626d789-257d-46c2-8555-8af451ed6434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 18:14:52.103315 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:52.103278 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:14:52.103516 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:52.103341 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" podUID="8626d789-257d-46c2-8555-8af451ed6434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 18:14:52.762082 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:52.762039 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq"] Oct 30 18:14:52.767420 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:52.767396 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" Oct 30 18:14:52.786639 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:52.786609 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq"] Oct 30 18:14:52.863960 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:52.863910 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01ac5b0b-929b-4a9c-9b91-e884edd40abb-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq\" (UID: \"01ac5b0b-929b-4a9c-9b91-e884edd40abb\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" Oct 30 18:14:52.864419 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:52.864397 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01ac5b0b-929b-4a9c-9b91-e884edd40abb-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq\" (UID: \"01ac5b0b-929b-4a9c-9b91-e884edd40abb\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" Oct 30 18:14:52.864597 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:52.864581 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ac5b0b-929b-4a9c-9b91-e884edd40abb-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq\" (UID: \"01ac5b0b-929b-4a9c-9b91-e884edd40abb\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" Oct 30 18:14:52.864745 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:52.864731 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zjsl\" (UniqueName: \"kubernetes.io/projected/01ac5b0b-929b-4a9c-9b91-e884edd40abb-kube-api-access-4zjsl\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq\" (UID: \"01ac5b0b-929b-4a9c-9b91-e884edd40abb\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" Oct 30 18:14:52.864875 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:52.864863 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01ac5b0b-929b-4a9c-9b91-e884edd40abb-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq\" (UID: \"01ac5b0b-929b-4a9c-9b91-e884edd40abb\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" Oct 30 18:14:52.865014 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:52.865001 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01ac5b0b-929b-4a9c-9b91-e884edd40abb-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq\" (UID: \"01ac5b0b-929b-4a9c-9b91-e884edd40abb\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" Oct 30 18:14:52.865202 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:52.865188 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01ac5b0b-929b-4a9c-9b91-e884edd40abb-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq\" (UID: \"01ac5b0b-929b-4a9c-9b91-e884edd40abb\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" Oct 30 18:14:52.865408 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:52.865362 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ac5b0b-929b-4a9c-9b91-e884edd40abb-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq\" (UID: \"01ac5b0b-929b-4a9c-9b91-e884edd40abb\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" Oct 30 18:14:52.865616 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:52.865601 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01ac5b0b-929b-4a9c-9b91-e884edd40abb-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq\" (UID: \"01ac5b0b-929b-4a9c-9b91-e884edd40abb\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" Oct 30 18:14:52.967389 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:52.967349 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01ac5b0b-929b-4a9c-9b91-e884edd40abb-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq\" (UID: \"01ac5b0b-929b-4a9c-9b91-e884edd40abb\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" Oct 30 18:14:52.967582 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:52.967409 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01ac5b0b-929b-4a9c-9b91-e884edd40abb-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq\" (UID: \"01ac5b0b-929b-4a9c-9b91-e884edd40abb\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" Oct 30 18:14:52.967582 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:52.967464 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01ac5b0b-929b-4a9c-9b91-e884edd40abb-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq\" (UID: \"01ac5b0b-929b-4a9c-9b91-e884edd40abb\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" Oct 30 18:14:52.967582 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:52.967486 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ac5b0b-929b-4a9c-9b91-e884edd40abb-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq\" (UID: \"01ac5b0b-929b-4a9c-9b91-e884edd40abb\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" Oct 30 18:14:52.967582 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:52.967531 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zjsl\" (UniqueName: \"kubernetes.io/projected/01ac5b0b-929b-4a9c-9b91-e884edd40abb-kube-api-access-4zjsl\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq\" (UID: \"01ac5b0b-929b-4a9c-9b91-e884edd40abb\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" Oct 30 18:14:52.967582 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:52.967561 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01ac5b0b-929b-4a9c-9b91-e884edd40abb-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq\" (UID: \"01ac5b0b-929b-4a9c-9b91-e884edd40abb\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" Oct 30 18:14:52.967761 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:52.967590 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01ac5b0b-929b-4a9c-9b91-e884edd40abb-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq\" (UID: \"01ac5b0b-929b-4a9c-9b91-e884edd40abb\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" Oct 30 18:14:52.967761 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:52.967635 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01ac5b0b-929b-4a9c-9b91-e884edd40abb-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq\" (UID: \"01ac5b0b-929b-4a9c-9b91-e884edd40abb\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" Oct 30 18:14:52.967761 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:52.967663 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ac5b0b-929b-4a9c-9b91-e884edd40abb-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq\" (UID: \"01ac5b0b-929b-4a9c-9b91-e884edd40abb\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" Oct 30 18:14:52.968095 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:52.968064 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ac5b0b-929b-4a9c-9b91-e884edd40abb-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq\" (UID: \"01ac5b0b-929b-4a9c-9b91-e884edd40abb\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" Oct 30 18:14:52.968256 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:52.968204 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01ac5b0b-929b-4a9c-9b91-e884edd40abb-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq\" (UID: \"01ac5b0b-929b-4a9c-9b91-e884edd40abb\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" Oct 30 18:14:52.968486 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:52.968360 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01ac5b0b-929b-4a9c-9b91-e884edd40abb-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq\" (UID: \"01ac5b0b-929b-4a9c-9b91-e884edd40abb\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" Oct 30 18:14:52.968652 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:52.968625 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01ac5b0b-929b-4a9c-9b91-e884edd40abb-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq\" (UID: \"01ac5b0b-929b-4a9c-9b91-e884edd40abb\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" Oct 30 18:14:52.968730 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:52.968658 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ac5b0b-929b-4a9c-9b91-e884edd40abb-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq\" (UID: \"01ac5b0b-929b-4a9c-9b91-e884edd40abb\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" Oct 30 18:14:52.970069 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:52.970049 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01ac5b0b-929b-4a9c-9b91-e884edd40abb-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq\" (UID: \"01ac5b0b-929b-4a9c-9b91-e884edd40abb\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" Oct 30 18:14:52.970809 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:52.970787 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01ac5b0b-929b-4a9c-9b91-e884edd40abb-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq\" (UID: \"01ac5b0b-929b-4a9c-9b91-e884edd40abb\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" Oct 30 18:14:52.987646 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:52.987623 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01ac5b0b-929b-4a9c-9b91-e884edd40abb-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq\" (UID: \"01ac5b0b-929b-4a9c-9b91-e884edd40abb\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" Oct 30 18:14:52.991360 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:52.991337 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zjsl\" (UniqueName: \"kubernetes.io/projected/01ac5b0b-929b-4a9c-9b91-e884edd40abb-kube-api-access-4zjsl\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq\" (UID: \"01ac5b0b-929b-4a9c-9b91-e884edd40abb\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" Oct 30 18:14:53.085088 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:53.085001 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" Oct 30 18:14:53.105818 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:53.105784 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:14:53.105967 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:53.105854 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" podUID="8626d789-257d-46c2-8555-8af451ed6434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 18:14:53.466849 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:53.466806 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq"] Oct 30 18:14:53.468211 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:14:53.468178 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01ac5b0b_929b_4a9c_9b91_e884edd40abb.slice/crio-3c8669f7e094f9bb98bb8018a80656bfc2139654a7aa3783ff19a94b6f30451c WatchSource:0}: Error finding container 3c8669f7e094f9bb98bb8018a80656bfc2139654a7aa3783ff19a94b6f30451c: Status 404 returned error can't find the container with id 3c8669f7e094f9bb98bb8018a80656bfc2139654a7aa3783ff19a94b6f30451c Oct 30 18:14:53.471326 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:53.471292 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:14:53.471433 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:53.471394 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:14:53.471494 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:53.471436 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:14:54.103148 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:54.103096 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:14:54.103558 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:54.103189 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" podUID="8626d789-257d-46c2-8555-8af451ed6434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 18:14:54.207045 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:54.206991 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" event={"ID":"01ac5b0b-929b-4a9c-9b91-e884edd40abb","Type":"ContainerStarted","Data":"e0c4510e20171ee0cc4126f77079dd46ed8e6dc312ab03809a574f514f527aeb"} Oct 30 18:14:54.207255 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:54.207094 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" event={"ID":"01ac5b0b-929b-4a9c-9b91-e884edd40abb","Type":"ContainerStarted","Data":"3c8669f7e094f9bb98bb8018a80656bfc2139654a7aa3783ff19a94b6f30451c"} Oct 30 18:14:54.274269 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:54.274217 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" podStartSLOduration=2.274200453 podStartE2EDuration="2.274200453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:54.259079828 +0000 UTC m=+991.349142167" watchObservedRunningTime="2026-10-30 18:14:54.274200453 +0000 UTC m=+991.364262792" Oct 30 18:14:55.086155 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:55.086098 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" Oct 30 18:14:55.088187 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:55.087886 4442 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 18:14:55.088187 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:55.087939 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" podUID="01ac5b0b-929b-4a9c-9b91-e884edd40abb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 18:14:55.389962 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:55.387879 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" Oct 30 18:14:55.389962 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:55.389870 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" Oct 30 18:14:55.391178 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:55.390950 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" Oct 30 18:14:56.085982 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:56.085946 4442 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 18:14:56.086162 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:56.086016 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" podUID="01ac5b0b-929b-4a9c-9b91-e884edd40abb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 18:14:57.087668 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:57.087632 4442 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 18:14:57.088322 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:57.088289 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" podUID="01ac5b0b-929b-4a9c-9b91-e884edd40abb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 18:14:58.091163 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:58.089571 4442 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 18:14:58.091163 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:58.089655 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" podUID="01ac5b0b-929b-4a9c-9b91-e884edd40abb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 18:14:59.111160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:59.108151 4442 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 18:14:59.111160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:14:59.108251 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" podUID="01ac5b0b-929b-4a9c-9b91-e884edd40abb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 18:15:00.161197 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:00.152679 4442 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 18:15:00.161197 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:00.152746 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" podUID="01ac5b0b-929b-4a9c-9b91-e884edd40abb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 18:15:00.397672 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:00.397629 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889735-nw7gs"] Oct 30 18:15:00.456792 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:00.456718 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889735-nw7gs"] Oct 30 18:15:00.457110 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:00.457039 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889735-nw7gs" Oct 30 18:15:00.506164 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:00.497913 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:15:00.578567 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:00.578535 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7080735b-c5eb-4db4-8983-d7da5df88242-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-921450-29889735-nw7gs\" (UID: \"7080735b-c5eb-4db4-8983-d7da5df88242\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889735-nw7gs" Oct 30 18:15:00.579093 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:00.579072 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmkx7\" (UniqueName: \"kubernetes.io/projected/7080735b-c5eb-4db4-8983-d7da5df88242-kube-api-access-dmkx7\") pod \"maas-api-key-cleanup-e2e-worker-w0-921450-29889735-nw7gs\" (UID: \"7080735b-c5eb-4db4-8983-d7da5df88242\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889735-nw7gs" Oct 30 18:15:00.695160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:00.683981 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7080735b-c5eb-4db4-8983-d7da5df88242-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-921450-29889735-nw7gs\" (UID: \"7080735b-c5eb-4db4-8983-d7da5df88242\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889735-nw7gs" Oct 30 18:15:00.721215 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:00.695609 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmkx7\" (UniqueName: \"kubernetes.io/projected/7080735b-c5eb-4db4-8983-d7da5df88242-kube-api-access-dmkx7\") pod \"maas-api-key-cleanup-e2e-worker-w0-921450-29889735-nw7gs\" (UID: \"7080735b-c5eb-4db4-8983-d7da5df88242\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889735-nw7gs" Oct 30 18:15:00.721640 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:00.704939 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7080735b-c5eb-4db4-8983-d7da5df88242-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-921450-29889735-nw7gs\" (UID: \"7080735b-c5eb-4db4-8983-d7da5df88242\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889735-nw7gs" Oct 30 18:15:00.815327 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:00.815284 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmkx7\" (UniqueName: \"kubernetes.io/projected/7080735b-c5eb-4db4-8983-d7da5df88242-kube-api-access-dmkx7\") pod \"maas-api-key-cleanup-e2e-worker-w0-921450-29889735-nw7gs\" (UID: \"7080735b-c5eb-4db4-8983-d7da5df88242\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889735-nw7gs" Oct 30 18:15:01.044222 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:01.043685 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889735-nw7gs" Oct 30 18:15:01.089281 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:01.089234 4442 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4" podUID="cf7d5f54-e091-4032-ab11-af2687e453c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": context deadline exceeded" Oct 30 18:15:01.206207 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:01.197473 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4" podUID="cf7d5f54-e091-4032-ab11-af2687e453c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:01.369382 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:01.356201 4442 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 18:15:01.369382 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:01.356257 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" podUID="01ac5b0b-929b-4a9c-9b91-e884edd40abb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 18:15:01.670285 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:01.670239 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8" podUID="1e2901e8-4919-4e5c-88c7-39126a7cca56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:02.069698 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:02.069668 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889735-nw7gs"] Oct 30 18:15:02.073481 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:15:02.073436 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7080735b_c5eb_4db4_8983_d7da5df88242.slice/crio-0264fe7d60ea3040d15e87a26674929c92c87da5cc5f84435496610bdf3a5ee6 WatchSource:0}: Error finding container 0264fe7d60ea3040d15e87a26674929c92c87da5cc5f84435496610bdf3a5ee6: Status 404 returned error can't find the container with id 0264fe7d60ea3040d15e87a26674929c92c87da5cc5f84435496610bdf3a5ee6 Oct 30 18:15:02.078402 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:02.078379 4442 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:15:02.085501 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:02.085469 4442 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 18:15:02.085600 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:02.085532 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" podUID="01ac5b0b-929b-4a9c-9b91-e884edd40abb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 18:15:02.747864 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:02.747827 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889735-nw7gs" event={"ID":"7080735b-c5eb-4db4-8983-d7da5df88242","Type":"ContainerStarted","Data":"0264fe7d60ea3040d15e87a26674929c92c87da5cc5f84435496610bdf3a5ee6"} Oct 30 18:15:03.537162 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:03.534066 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" Oct 30 18:15:03.537162 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:03.535587 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" Oct 30 18:15:03.545717 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:03.545693 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" Oct 30 18:15:05.166959 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:05.166871 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-669b5044-wsbnn"] Oct 30 18:15:05.172816 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:05.172792 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-669b5044-wsbnn" Oct 30 18:15:05.191239 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:05.191214 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-669b5044-wsbnn"] Oct 30 18:15:05.197411 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:05.197366 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:15:05.213104 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:05.213071 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fed16271-5356-4f46-b6ba-98f5393aeef7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-669b5044-wsbnn\" (UID: \"fed16271-5356-4f46-b6ba-98f5393aeef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-669b5044-wsbnn" Oct 30 18:15:05.213310 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:05.213128 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fed16271-5356-4f46-b6ba-98f5393aeef7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-669b5044-wsbnn\" (UID: \"fed16271-5356-4f46-b6ba-98f5393aeef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-669b5044-wsbnn" Oct 30 18:15:05.213388 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:05.213317 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9g8v\" (UniqueName: \"kubernetes.io/projected/fed16271-5356-4f46-b6ba-98f5393aeef7-kube-api-access-b9g8v\") pod \"maas-api-revoke-sub-669b5044-wsbnn\" (UID: \"fed16271-5356-4f46-b6ba-98f5393aeef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-669b5044-wsbnn" Oct 30 18:15:05.315189 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:05.314523 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fed16271-5356-4f46-b6ba-98f5393aeef7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-669b5044-wsbnn\" (UID: \"fed16271-5356-4f46-b6ba-98f5393aeef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-669b5044-wsbnn" Oct 30 18:15:05.315189 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:05.314785 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9g8v\" (UniqueName: \"kubernetes.io/projected/fed16271-5356-4f46-b6ba-98f5393aeef7-kube-api-access-b9g8v\") pod \"maas-api-revoke-sub-669b5044-wsbnn\" (UID: \"fed16271-5356-4f46-b6ba-98f5393aeef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-669b5044-wsbnn" Oct 30 18:15:05.315189 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:05.314871 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fed16271-5356-4f46-b6ba-98f5393aeef7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-669b5044-wsbnn\" (UID: \"fed16271-5356-4f46-b6ba-98f5393aeef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-669b5044-wsbnn" Oct 30 18:15:05.316993 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:05.316799 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fed16271-5356-4f46-b6ba-98f5393aeef7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-669b5044-wsbnn\" (UID: \"fed16271-5356-4f46-b6ba-98f5393aeef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-669b5044-wsbnn" Oct 30 18:15:05.338067 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:05.338022 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9g8v\" (UniqueName: \"kubernetes.io/projected/fed16271-5356-4f46-b6ba-98f5393aeef7-kube-api-access-b9g8v\") pod \"maas-api-revoke-sub-669b5044-wsbnn\" (UID: \"fed16271-5356-4f46-b6ba-98f5393aeef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-669b5044-wsbnn" Oct 30 18:15:05.340364 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:05.339203 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fed16271-5356-4f46-b6ba-98f5393aeef7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-669b5044-wsbnn\" (UID: \"fed16271-5356-4f46-b6ba-98f5393aeef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-669b5044-wsbnn" Oct 30 18:15:05.487188 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:05.487158 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-669b5044-wsbnn" Oct 30 18:15:05.779883 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:05.779797 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889735-nw7gs" event={"ID":"7080735b-c5eb-4db4-8983-d7da5df88242","Type":"ContainerStarted","Data":"7f3c7a790200a1f7bfad93bb38bbc82047b875626dc4f39055877a7df2f2887e"} Oct 30 18:15:05.802161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:05.802120 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-669b5044-wsbnn"] Oct 30 18:15:05.803991 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:15:05.803946 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfed16271_5356_4f46_b6ba_98f5393aeef7.slice/crio-19dba21a6f87d0914b40d83e51cc7cffb889af1f6a9ea92451d00200cb419cd0 WatchSource:0}: Error finding container 19dba21a6f87d0914b40d83e51cc7cffb889af1f6a9ea92451d00200cb419cd0: Status 404 returned error can't find the container with id 19dba21a6f87d0914b40d83e51cc7cffb889af1f6a9ea92451d00200cb419cd0 Oct 30 18:15:05.825724 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:05.825526 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889735-nw7gs" podStartSLOduration=3.008673542 podStartE2EDuration="5.825506729s" totalImagesPullingTime="2.816833187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:05.819239121 +0000 UTC m=+1002.909301461" watchObservedRunningTime="2026-10-30 18:15:05.825506729 +0000 UTC m=+1002.915569069" Oct 30 18:15:06.872249 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:06.872204 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-669b5044-wsbnn" event={"ID":"fed16271-5356-4f46-b6ba-98f5393aeef7","Type":"ContainerStarted","Data":"3aeeb83180209441d0fcb39b72a72223ee72701904312e66ec34cc0c84c24d6b"} Oct 30 18:15:06.872731 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:06.872312 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-669b5044-wsbnn" event={"ID":"fed16271-5356-4f46-b6ba-98f5393aeef7","Type":"ContainerStarted","Data":"19dba21a6f87d0914b40d83e51cc7cffb889af1f6a9ea92451d00200cb419cd0"} Oct 30 18:15:06.873960 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:06.873932 4442 generic.go:356] "Generic (PLEG): container finished" podID="7080735b-c5eb-4db4-8983-d7da5df88242" containerID="7f3c7a790200a1f7bfad93bb38bbc82047b875626dc4f39055877a7df2f2887e" exitCode=60 Oct 30 18:15:06.874088 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:06.873970 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889735-nw7gs" event={"ID":"7080735b-c5eb-4db4-8983-d7da5df88242","Type":"ContainerDied","Data":"7f3c7a790200a1f7bfad93bb38bbc82047b875626dc4f39055877a7df2f2887e"} Oct 30 18:15:06.874237 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:06.874225 4442 scope.go:122] "RemoveContainer" containerID="7f3c7a790200a1f7bfad93bb38bbc82047b875626dc4f39055877a7df2f2887e" Oct 30 18:15:06.910071 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:06.910021 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-669b5044-wsbnn" podStartSLOduration=1.910007039 podStartE2EDuration="1.910007039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:06.90899561 +0000 UTC m=+1003.999057952" watchObservedRunningTime="2026-10-30 18:15:06.910007039 +0000 UTC m=+1004.000069379" Oct 30 18:15:07.882701 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:07.882670 4442 generic.go:356] "Generic (PLEG): container finished" podID="7080735b-c5eb-4db4-8983-d7da5df88242" containerID="f613c36aa92788de0d9fc8293e45c896c3a04d12128d55c9abb7ce183897d130" exitCode=60 Oct 30 18:15:07.883195 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:07.882752 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889735-nw7gs" event={"ID":"7080735b-c5eb-4db4-8983-d7da5df88242","Type":"ContainerDied","Data":"f613c36aa92788de0d9fc8293e45c896c3a04d12128d55c9abb7ce183897d130"} Oct 30 18:15:07.883195 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:07.882810 4442 scope.go:122] "RemoveContainer" containerID="7f3c7a790200a1f7bfad93bb38bbc82047b875626dc4f39055877a7df2f2887e" Oct 30 18:15:07.883195 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:07.883103 4442 scope.go:122] "RemoveContainer" containerID="f613c36aa92788de0d9fc8293e45c896c3a04d12128d55c9abb7ce183897d130" Oct 30 18:15:07.883504 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:15:07.883478 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-921450-29889735-nw7gs_odh-ai-gateway-infra(7080735b-c5eb-4db4-8983-d7da5df88242)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889735-nw7gs" podUID="7080735b-c5eb-4db4-8983-d7da5df88242" Oct 30 18:15:07.884582 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:07.884561 4442 generic.go:356] "Generic (PLEG): container finished" podID="fed16271-5356-4f46-b6ba-98f5393aeef7" containerID="3aeeb83180209441d0fcb39b72a72223ee72701904312e66ec34cc0c84c24d6b" exitCode=0 Oct 30 18:15:07.884665 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:07.884608 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-669b5044-wsbnn" event={"ID":"fed16271-5356-4f46-b6ba-98f5393aeef7","Type":"ContainerDied","Data":"3aeeb83180209441d0fcb39b72a72223ee72701904312e66ec34cc0c84c24d6b"} Oct 30 18:15:08.894475 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:08.894445 4442 scope.go:122] "RemoveContainer" containerID="f613c36aa92788de0d9fc8293e45c896c3a04d12128d55c9abb7ce183897d130" Oct 30 18:15:08.894939 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:15:08.894781 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-921450-29889735-nw7gs_odh-ai-gateway-infra(7080735b-c5eb-4db4-8983-d7da5df88242)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889735-nw7gs" podUID="7080735b-c5eb-4db4-8983-d7da5df88242" Oct 30 18:15:09.080591 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:09.080563 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-669b5044-wsbnn" Oct 30 18:15:09.194673 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:09.194580 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fed16271-5356-4f46-b6ba-98f5393aeef7-kube-api-access-b9g8v\" (UniqueName: \"kubernetes.io/projected/fed16271-5356-4f46-b6ba-98f5393aeef7-kube-api-access-b9g8v\") pod \"fed16271-5356-4f46-b6ba-98f5393aeef7\" (UID: \"fed16271-5356-4f46-b6ba-98f5393aeef7\") " Oct 30 18:15:09.194953 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:09.194933 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fed16271-5356-4f46-b6ba-98f5393aeef7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fed16271-5356-4f46-b6ba-98f5393aeef7-maas-api-openshift-ca\") pod \"fed16271-5356-4f46-b6ba-98f5393aeef7\" (UID: \"fed16271-5356-4f46-b6ba-98f5393aeef7\") " Oct 30 18:15:09.195338 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:09.195309 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fed16271-5356-4f46-b6ba-98f5393aeef7-maas-api-openshift-ca" pod "fed16271-5356-4f46-b6ba-98f5393aeef7" (UID: "fed16271-5356-4f46-b6ba-98f5393aeef7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:09.195713 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:09.195216 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fed16271-5356-4f46-b6ba-98f5393aeef7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fed16271-5356-4f46-b6ba-98f5393aeef7-maas-api-xks-ca\") pod \"fed16271-5356-4f46-b6ba-98f5393aeef7\" (UID: \"fed16271-5356-4f46-b6ba-98f5393aeef7\") " Oct 30 18:15:09.196456 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:09.196431 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fed16271-5356-4f46-b6ba-98f5393aeef7-maas-api-openshift-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:15:09.197476 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:09.197451 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fed16271-5356-4f46-b6ba-98f5393aeef7-kube-api-access-b9g8v" pod "fed16271-5356-4f46-b6ba-98f5393aeef7" (UID: "fed16271-5356-4f46-b6ba-98f5393aeef7"). InnerVolumeSpecName "kube-api-access-b9g8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:09.197892 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:09.197868 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fed16271-5356-4f46-b6ba-98f5393aeef7-maas-api-xks-ca" pod "fed16271-5356-4f46-b6ba-98f5393aeef7" (UID: "fed16271-5356-4f46-b6ba-98f5393aeef7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:09.298124 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:09.298093 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9g8v\" (UniqueName: \"kubernetes.io/projected/fed16271-5356-4f46-b6ba-98f5393aeef7-kube-api-access-b9g8v\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:15:09.298124 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:09.298121 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fed16271-5356-4f46-b6ba-98f5393aeef7-maas-api-xks-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:15:09.902671 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:09.902639 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-669b5044-wsbnn" Oct 30 18:15:09.903213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:09.902677 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-669b5044-wsbnn" event={"ID":"fed16271-5356-4f46-b6ba-98f5393aeef7","Type":"ContainerDied","Data":"19dba21a6f87d0914b40d83e51cc7cffb889af1f6a9ea92451d00200cb419cd0"} Oct 30 18:15:09.903213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:09.902758 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19dba21a6f87d0914b40d83e51cc7cffb889af1f6a9ea92451d00200cb419cd0" Oct 30 18:15:17.800184 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:15:17.799299 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:15:18.199167 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:18.187904 4442 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:15:18.199167 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:18.187975 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" podUID="5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:18.500931 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:18.500827 4442 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:15:18.501251 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:18.501218 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" podUID="5fae7458-0960-4008-b3ae-79d4518ea89e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:19.895159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:19.893045 4442 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:15:19.895159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:19.893121 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" podUID="01ac5b0b-929b-4a9c-9b91-e884edd40abb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:20.028192 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:20.026755 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:15:20.052159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:20.051435 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" podUID="18a1649e-7af6-4854-bb2f-0c44af4b0276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:20.102166 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:20.101507 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dctgm" podUID="ac100f87-aa55-49c3-8760-85523d3927db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 30 18:15:21.057297 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:21.057258 4442 scope.go:122] "RemoveContainer" containerID="f613c36aa92788de0d9fc8293e45c896c3a04d12128d55c9abb7ce183897d130" Oct 30 18:15:22.028498 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:22.028472 4442 generic.go:356] "Generic (PLEG): container finished" podID="7080735b-c5eb-4db4-8983-d7da5df88242" containerID="f05fb0ab8d465269d4655bf572fe11af0489d8dcb232428b1d36a8631661c0d1" exitCode=60 Oct 30 18:15:22.028631 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:22.028539 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889735-nw7gs" event={"ID":"7080735b-c5eb-4db4-8983-d7da5df88242","Type":"ContainerDied","Data":"f05fb0ab8d465269d4655bf572fe11af0489d8dcb232428b1d36a8631661c0d1"} Oct 30 18:15:22.028631 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:22.028585 4442 scope.go:122] "RemoveContainer" containerID="f613c36aa92788de0d9fc8293e45c896c3a04d12128d55c9abb7ce183897d130" Oct 30 18:15:22.028980 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:22.028956 4442 scope.go:122] "RemoveContainer" containerID="f05fb0ab8d465269d4655bf572fe11af0489d8dcb232428b1d36a8631661c0d1" Oct 30 18:15:22.029328 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:15:22.029298 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-921450-29889735-nw7gs_odh-ai-gateway-infra(7080735b-c5eb-4db4-8983-d7da5df88242)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889735-nw7gs" podUID="7080735b-c5eb-4db4-8983-d7da5df88242" Oct 30 18:15:22.846110 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:22.846022 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp"] Oct 30 18:15:22.846723 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:22.846707 4442 cpu_manager.go:414] "containerMap: removing container" podUID="fed16271-5356-4f46-b6ba-98f5393aeef7" containerName="revoke-keys" Oct 30 18:15:22.846767 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:22.846725 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fed16271-5356-4f46-b6ba-98f5393aeef7" containerName="revoke-keys" Oct 30 18:15:22.846837 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:22.846828 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="fed16271-5356-4f46-b6ba-98f5393aeef7" containerName="revoke-keys" Oct 30 18:15:22.851510 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:22.851488 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" Oct 30 18:15:22.886151 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:22.886100 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp"] Oct 30 18:15:22.969996 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:22.969962 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2399863-51b1-4b7a-acdd-ea28c9074e95-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp\" (UID: \"a2399863-51b1-4b7a-acdd-ea28c9074e95\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" Oct 30 18:15:22.970225 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:22.970123 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2399863-51b1-4b7a-acdd-ea28c9074e95-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp\" (UID: \"a2399863-51b1-4b7a-acdd-ea28c9074e95\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" Oct 30 18:15:22.970296 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:22.970217 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2399863-51b1-4b7a-acdd-ea28c9074e95-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp\" (UID: \"a2399863-51b1-4b7a-acdd-ea28c9074e95\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" Oct 30 18:15:22.970296 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:22.970257 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2399863-51b1-4b7a-acdd-ea28c9074e95-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp\" (UID: \"a2399863-51b1-4b7a-acdd-ea28c9074e95\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" Oct 30 18:15:22.970402 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:22.970361 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2399863-51b1-4b7a-acdd-ea28c9074e95-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp\" (UID: \"a2399863-51b1-4b7a-acdd-ea28c9074e95\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" Oct 30 18:15:22.970455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:22.970408 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2399863-51b1-4b7a-acdd-ea28c9074e95-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp\" (UID: \"a2399863-51b1-4b7a-acdd-ea28c9074e95\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" Oct 30 18:15:22.970455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:22.970428 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7588\" (UniqueName: \"kubernetes.io/projected/a2399863-51b1-4b7a-acdd-ea28c9074e95-kube-api-access-d7588\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp\" (UID: \"a2399863-51b1-4b7a-acdd-ea28c9074e95\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" Oct 30 18:15:22.970559 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:22.970479 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2399863-51b1-4b7a-acdd-ea28c9074e95-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp\" (UID: \"a2399863-51b1-4b7a-acdd-ea28c9074e95\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" Oct 30 18:15:22.970559 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:22.970539 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2399863-51b1-4b7a-acdd-ea28c9074e95-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp\" (UID: \"a2399863-51b1-4b7a-acdd-ea28c9074e95\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" Oct 30 18:15:23.071274 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:23.071242 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2399863-51b1-4b7a-acdd-ea28c9074e95-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp\" (UID: \"a2399863-51b1-4b7a-acdd-ea28c9074e95\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" Oct 30 18:15:23.071466 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:23.071279 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2399863-51b1-4b7a-acdd-ea28c9074e95-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp\" (UID: \"a2399863-51b1-4b7a-acdd-ea28c9074e95\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" Oct 30 18:15:23.071466 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:23.071297 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7588\" (UniqueName: \"kubernetes.io/projected/a2399863-51b1-4b7a-acdd-ea28c9074e95-kube-api-access-d7588\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp\" (UID: \"a2399863-51b1-4b7a-acdd-ea28c9074e95\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" Oct 30 18:15:23.071466 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:23.071368 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2399863-51b1-4b7a-acdd-ea28c9074e95-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp\" (UID: \"a2399863-51b1-4b7a-acdd-ea28c9074e95\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" Oct 30 18:15:23.071466 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:23.071420 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2399863-51b1-4b7a-acdd-ea28c9074e95-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp\" (UID: \"a2399863-51b1-4b7a-acdd-ea28c9074e95\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" Oct 30 18:15:23.071466 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:23.071458 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2399863-51b1-4b7a-acdd-ea28c9074e95-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp\" (UID: \"a2399863-51b1-4b7a-acdd-ea28c9074e95\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" Oct 30 18:15:23.071821 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:23.071567 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2399863-51b1-4b7a-acdd-ea28c9074e95-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp\" (UID: \"a2399863-51b1-4b7a-acdd-ea28c9074e95\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" Oct 30 18:15:23.071821 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:23.071613 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2399863-51b1-4b7a-acdd-ea28c9074e95-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp\" (UID: \"a2399863-51b1-4b7a-acdd-ea28c9074e95\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" Oct 30 18:15:23.071821 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:23.071676 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2399863-51b1-4b7a-acdd-ea28c9074e95-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp\" (UID: \"a2399863-51b1-4b7a-acdd-ea28c9074e95\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" Oct 30 18:15:23.071821 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:23.071696 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2399863-51b1-4b7a-acdd-ea28c9074e95-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp\" (UID: \"a2399863-51b1-4b7a-acdd-ea28c9074e95\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" Oct 30 18:15:23.072016 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:23.071931 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2399863-51b1-4b7a-acdd-ea28c9074e95-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp\" (UID: \"a2399863-51b1-4b7a-acdd-ea28c9074e95\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" Oct 30 18:15:23.072016 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:23.071994 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2399863-51b1-4b7a-acdd-ea28c9074e95-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp\" (UID: \"a2399863-51b1-4b7a-acdd-ea28c9074e95\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" Oct 30 18:15:23.072159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:23.072095 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2399863-51b1-4b7a-acdd-ea28c9074e95-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp\" (UID: \"a2399863-51b1-4b7a-acdd-ea28c9074e95\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" Oct 30 18:15:23.074658 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:23.074626 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2399863-51b1-4b7a-acdd-ea28c9074e95-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp\" (UID: \"a2399863-51b1-4b7a-acdd-ea28c9074e95\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" Oct 30 18:15:23.075916 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:23.075893 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2399863-51b1-4b7a-acdd-ea28c9074e95-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp\" (UID: \"a2399863-51b1-4b7a-acdd-ea28c9074e95\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" Oct 30 18:15:23.085978 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:23.085947 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889735-nw7gs"] Oct 30 18:15:23.089567 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:23.089542 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2399863-51b1-4b7a-acdd-ea28c9074e95-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp\" (UID: \"a2399863-51b1-4b7a-acdd-ea28c9074e95\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" Oct 30 18:15:23.096600 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:23.096521 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2399863-51b1-4b7a-acdd-ea28c9074e95-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp\" (UID: \"a2399863-51b1-4b7a-acdd-ea28c9074e95\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" Oct 30 18:15:23.098525 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:23.098497 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7588\" (UniqueName: \"kubernetes.io/projected/a2399863-51b1-4b7a-acdd-ea28c9074e95-kube-api-access-d7588\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp\" (UID: \"a2399863-51b1-4b7a-acdd-ea28c9074e95\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" Oct 30 18:15:23.188166 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:23.176745 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" Oct 30 18:15:23.372776 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:23.372744 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889735-nw7gs" Oct 30 18:15:23.482789 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:23.482754 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7080735b-c5eb-4db4-8983-d7da5df88242-kube-api-access-dmkx7\" (UniqueName: \"kubernetes.io/projected/7080735b-c5eb-4db4-8983-d7da5df88242-kube-api-access-dmkx7\") pod \"7080735b-c5eb-4db4-8983-d7da5df88242\" (UID: \"7080735b-c5eb-4db4-8983-d7da5df88242\") " Oct 30 18:15:23.482951 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:23.482854 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7080735b-c5eb-4db4-8983-d7da5df88242-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7080735b-c5eb-4db4-8983-d7da5df88242-maas-api-service-ca\") pod \"7080735b-c5eb-4db4-8983-d7da5df88242\" (UID: \"7080735b-c5eb-4db4-8983-d7da5df88242\") " Oct 30 18:15:23.483314 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:23.483280 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7080735b-c5eb-4db4-8983-d7da5df88242-maas-api-service-ca" pod "7080735b-c5eb-4db4-8983-d7da5df88242" (UID: "7080735b-c5eb-4db4-8983-d7da5df88242"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:23.485020 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:23.484996 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7080735b-c5eb-4db4-8983-d7da5df88242-kube-api-access-dmkx7" pod "7080735b-c5eb-4db4-8983-d7da5df88242" (UID: "7080735b-c5eb-4db4-8983-d7da5df88242"). InnerVolumeSpecName "kube-api-access-dmkx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:23.555057 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:23.554202 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp"] Oct 30 18:15:23.557109 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:15:23.557083 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2399863_51b1_4b7a_acdd_ea28c9074e95.slice/crio-b30034b140beafada7f484d78db9c74ccf7ffa169ab577d20e94695913c5725d WatchSource:0}: Error finding container b30034b140beafada7f484d78db9c74ccf7ffa169ab577d20e94695913c5725d: Status 404 returned error can't find the container with id b30034b140beafada7f484d78db9c74ccf7ffa169ab577d20e94695913c5725d Oct 30 18:15:23.559881 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:23.559849 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:15:23.559999 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:23.559915 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:15:23.559999 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:23.559953 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:15:23.584349 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:23.584325 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmkx7\" (UniqueName: \"kubernetes.io/projected/7080735b-c5eb-4db4-8983-d7da5df88242-kube-api-access-dmkx7\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:15:23.584349 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:23.584348 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7080735b-c5eb-4db4-8983-d7da5df88242-maas-api-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:15:24.055844 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:24.052321 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889735-nw7gs" event={"ID":"7080735b-c5eb-4db4-8983-d7da5df88242","Type":"ContainerDied","Data":"0264fe7d60ea3040d15e87a26674929c92c87da5cc5f84435496610bdf3a5ee6"} Oct 30 18:15:24.055844 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:24.052393 4442 scope.go:122] "RemoveContainer" containerID="f05fb0ab8d465269d4655bf572fe11af0489d8dcb232428b1d36a8631661c0d1" Oct 30 18:15:24.055844 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:24.052524 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889735-nw7gs" Oct 30 18:15:24.068998 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:24.068959 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" event={"ID":"a2399863-51b1-4b7a-acdd-ea28c9074e95","Type":"ContainerStarted","Data":"c19fecf022f3b46b5ba9a5ce7d8cc916830517e625d4a976ed22624d6c8ab02d"} Oct 30 18:15:24.069195 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:24.069032 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" event={"ID":"a2399863-51b1-4b7a-acdd-ea28c9074e95","Type":"ContainerStarted","Data":"b30034b140beafada7f484d78db9c74ccf7ffa169ab577d20e94695913c5725d"} Oct 30 18:15:24.111589 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:24.111556 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889735-nw7gs"] Oct 30 18:15:24.115638 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:24.115602 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889735-nw7gs"] Oct 30 18:15:24.163434 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:24.163374 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" podStartSLOduration=2.163351055 podStartE2EDuration="2.163351055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:24.158252954 +0000 UTC m=+1021.248315295" watchObservedRunningTime="2026-10-30 18:15:24.163351055 +0000 UTC m=+1021.253413395" Oct 30 18:15:24.177541 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:24.177510 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" Oct 30 18:15:24.179693 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:24.179663 4442 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 18:15:24.179834 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:24.179736 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" podUID="a2399863-51b1-4b7a-acdd-ea28c9074e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 18:15:25.065790 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:25.065762 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7080735b-c5eb-4db4-8983-d7da5df88242" path="/var/lib/kubelet/pods/7080735b-c5eb-4db4-8983-d7da5df88242/volumes" Oct 30 18:15:25.177597 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:25.177559 4442 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 18:15:25.177795 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:25.177615 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" podUID="a2399863-51b1-4b7a-acdd-ea28c9074e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 18:15:26.177893 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:26.177849 4442 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 18:15:26.178328 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:26.177924 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" podUID="a2399863-51b1-4b7a-acdd-ea28c9074e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 18:15:26.963823 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:26.963774 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm"] Oct 30 18:15:26.964582 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:26.964557 4442 cpu_manager.go:414] "containerMap: removing container" podUID="7080735b-c5eb-4db4-8983-d7da5df88242" containerName="cleanup" Oct 30 18:15:26.964582 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:26.964583 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7080735b-c5eb-4db4-8983-d7da5df88242" containerName="cleanup" Oct 30 18:15:26.964767 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:26.964610 4442 cpu_manager.go:414] "containerMap: removing container" podUID="7080735b-c5eb-4db4-8983-d7da5df88242" containerName="cleanup" Oct 30 18:15:26.964767 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:26.964617 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7080735b-c5eb-4db4-8983-d7da5df88242" containerName="cleanup" Oct 30 18:15:26.964767 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:26.964638 4442 cpu_manager.go:414] "containerMap: removing container" podUID="7080735b-c5eb-4db4-8983-d7da5df88242" containerName="cleanup" Oct 30 18:15:26.964767 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:26.964645 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7080735b-c5eb-4db4-8983-d7da5df88242" containerName="cleanup" Oct 30 18:15:26.964914 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:26.964780 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="7080735b-c5eb-4db4-8983-d7da5df88242" containerName="cleanup" Oct 30 18:15:26.964914 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:26.964794 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="7080735b-c5eb-4db4-8983-d7da5df88242" containerName="cleanup" Oct 30 18:15:26.970163 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:26.970124 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" Oct 30 18:15:27.004517 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:27.004485 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm"] Oct 30 18:15:27.070398 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:27.065958 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-istio-data\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm\" (UID: \"bafcd4d0-8a69-460a-9e0c-21b69dc4327a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" Oct 30 18:15:27.074606 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:27.071335 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-istio-podinfo\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm\" (UID: \"bafcd4d0-8a69-460a-9e0c-21b69dc4327a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" Oct 30 18:15:27.074606 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:27.071409 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-workload-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm\" (UID: \"bafcd4d0-8a69-460a-9e0c-21b69dc4327a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" Oct 30 18:15:27.074606 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:27.071453 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4plcf\" (UniqueName: \"kubernetes.io/projected/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-kube-api-access-4plcf\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm\" (UID: \"bafcd4d0-8a69-460a-9e0c-21b69dc4327a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" Oct 30 18:15:27.074606 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:27.071587 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-istio-token\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm\" (UID: \"bafcd4d0-8a69-460a-9e0c-21b69dc4327a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" Oct 30 18:15:27.074606 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:27.071677 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-istiod-ca-cert\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm\" (UID: \"bafcd4d0-8a69-460a-9e0c-21b69dc4327a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" Oct 30 18:15:27.074606 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:27.071861 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-istio-envoy\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm\" (UID: \"bafcd4d0-8a69-460a-9e0c-21b69dc4327a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" Oct 30 18:15:27.074606 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:27.071901 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-credential-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm\" (UID: \"bafcd4d0-8a69-460a-9e0c-21b69dc4327a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" Oct 30 18:15:27.074606 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:27.071926 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-workload-certs\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm\" (UID: \"bafcd4d0-8a69-460a-9e0c-21b69dc4327a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" Oct 30 18:15:27.173568 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:27.173534 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-workload-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm\" (UID: \"bafcd4d0-8a69-460a-9e0c-21b69dc4327a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" Oct 30 18:15:27.173782 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:27.173591 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4plcf\" (UniqueName: \"kubernetes.io/projected/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-kube-api-access-4plcf\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm\" (UID: \"bafcd4d0-8a69-460a-9e0c-21b69dc4327a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" Oct 30 18:15:27.173952 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:27.173928 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-istio-token\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm\" (UID: \"bafcd4d0-8a69-460a-9e0c-21b69dc4327a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" Oct 30 18:15:27.174118 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:27.174031 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-workload-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm\" (UID: \"bafcd4d0-8a69-460a-9e0c-21b69dc4327a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" Oct 30 18:15:27.174306 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:27.174271 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-istiod-ca-cert\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm\" (UID: \"bafcd4d0-8a69-460a-9e0c-21b69dc4327a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" Oct 30 18:15:27.174457 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:27.174441 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-istio-envoy\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm\" (UID: \"bafcd4d0-8a69-460a-9e0c-21b69dc4327a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" Oct 30 18:15:27.174518 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:27.174475 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-credential-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm\" (UID: \"bafcd4d0-8a69-460a-9e0c-21b69dc4327a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" Oct 30 18:15:27.174518 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:27.174502 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-workload-certs\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm\" (UID: \"bafcd4d0-8a69-460a-9e0c-21b69dc4327a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" Oct 30 18:15:27.174625 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:27.174549 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-istio-data\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm\" (UID: \"bafcd4d0-8a69-460a-9e0c-21b69dc4327a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" Oct 30 18:15:27.174677 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:27.174621 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-istio-podinfo\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm\" (UID: \"bafcd4d0-8a69-460a-9e0c-21b69dc4327a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" Oct 30 18:15:27.174843 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:27.174816 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-istiod-ca-cert\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm\" (UID: \"bafcd4d0-8a69-460a-9e0c-21b69dc4327a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" Oct 30 18:15:27.174926 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:27.174873 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-workload-certs\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm\" (UID: \"bafcd4d0-8a69-460a-9e0c-21b69dc4327a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" Oct 30 18:15:27.175048 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:27.175027 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-credential-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm\" (UID: \"bafcd4d0-8a69-460a-9e0c-21b69dc4327a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" Oct 30 18:15:27.175127 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:27.175035 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-istio-data\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm\" (UID: \"bafcd4d0-8a69-460a-9e0c-21b69dc4327a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" Oct 30 18:15:27.177036 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:27.177015 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-istio-envoy\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm\" (UID: \"bafcd4d0-8a69-460a-9e0c-21b69dc4327a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" Oct 30 18:15:27.177357 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:27.177335 4442 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 18:15:27.177458 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:27.177384 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" podUID="a2399863-51b1-4b7a-acdd-ea28c9074e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 18:15:27.178805 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:27.178784 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-istio-podinfo\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm\" (UID: \"bafcd4d0-8a69-460a-9e0c-21b69dc4327a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" Oct 30 18:15:27.192871 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:27.192839 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-istio-token\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm\" (UID: \"bafcd4d0-8a69-460a-9e0c-21b69dc4327a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" Oct 30 18:15:27.193000 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:27.192981 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4plcf\" (UniqueName: \"kubernetes.io/projected/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-kube-api-access-4plcf\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm\" (UID: \"bafcd4d0-8a69-460a-9e0c-21b69dc4327a\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" Oct 30 18:15:27.298205 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:27.298096 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" Oct 30 18:15:27.659273 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:27.659239 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm"] Oct 30 18:15:27.659692 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:15:27.659625 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbafcd4d0_8a69_460a_9e0c_21b69dc4327a.slice/crio-7c528e24d3ef1397c42640450601cc9d79ac94764c01fe51dfbc8f2d1d066c92 WatchSource:0}: Error finding container 7c528e24d3ef1397c42640450601cc9d79ac94764c01fe51dfbc8f2d1d066c92: Status 404 returned error can't find the container with id 7c528e24d3ef1397c42640450601cc9d79ac94764c01fe51dfbc8f2d1d066c92 Oct 30 18:15:27.663118 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:27.663084 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:15:27.663239 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:27.663219 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:15:27.663283 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:27.663272 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:15:28.102214 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:28.102099 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" event={"ID":"bafcd4d0-8a69-460a-9e0c-21b69dc4327a","Type":"ContainerStarted","Data":"b12af34751cdc6a6785aaff9bc012ddb7a00e3b2932340dd8b1c5d63fdfd6361"} Oct 30 18:15:28.102461 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:28.102441 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" event={"ID":"bafcd4d0-8a69-460a-9e0c-21b69dc4327a","Type":"ContainerStarted","Data":"7c528e24d3ef1397c42640450601cc9d79ac94764c01fe51dfbc8f2d1d066c92"} Oct 30 18:15:28.179189 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:28.177839 4442 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 18:15:28.179189 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:28.177895 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" podUID="a2399863-51b1-4b7a-acdd-ea28c9074e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 18:15:28.299200 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:28.299171 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" Oct 30 18:15:28.300385 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:28.300356 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:15:28.300493 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:28.300411 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" podUID="bafcd4d0-8a69-460a-9e0c-21b69dc4327a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 18:15:29.191056 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:29.191014 4442 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 18:15:29.191593 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:29.191079 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" podUID="a2399863-51b1-4b7a-acdd-ea28c9074e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 18:15:29.299370 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:29.299332 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:15:29.299547 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:29.299392 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" podUID="bafcd4d0-8a69-460a-9e0c-21b69dc4327a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 18:15:30.178422 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:30.178381 4442 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 18:15:30.178637 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:30.178444 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" podUID="a2399863-51b1-4b7a-acdd-ea28c9074e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 18:15:30.299094 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:30.299063 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:15:30.299507 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:30.299126 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" podUID="bafcd4d0-8a69-460a-9e0c-21b69dc4327a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 18:15:31.302291 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:31.299567 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:15:31.302291 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:31.299626 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" podUID="bafcd4d0-8a69-460a-9e0c-21b69dc4327a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 18:15:31.522160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:31.521438 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" Oct 30 18:15:31.545179 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:31.545114 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" Oct 30 18:15:31.661165 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:31.658624 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" podStartSLOduration=5.658597711 podStartE2EDuration="5.658597711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:28.171449896 +0000 UTC m=+1025.261512248" watchObservedRunningTime="2026-10-30 18:15:31.658597711 +0000 UTC m=+1028.748660052" Oct 30 18:15:31.669162 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:31.664180 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" Oct 30 18:15:31.934275 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:31.934188 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd"] Oct 30 18:15:31.938551 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:31.938524 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="7080735b-c5eb-4db4-8983-d7da5df88242" containerName="cleanup" Oct 30 18:15:32.005976 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:32.001452 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" Oct 30 18:15:32.162787 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:32.127236 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/963a8b12-5921-4d72-9c90-c31daf3d2475-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd\" (UID: \"963a8b12-5921-4d72-9c90-c31daf3d2475\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" Oct 30 18:15:32.162787 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:32.127344 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/963a8b12-5921-4d72-9c90-c31daf3d2475-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd\" (UID: \"963a8b12-5921-4d72-9c90-c31daf3d2475\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" Oct 30 18:15:32.162787 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:32.127445 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/963a8b12-5921-4d72-9c90-c31daf3d2475-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd\" (UID: \"963a8b12-5921-4d72-9c90-c31daf3d2475\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" Oct 30 18:15:32.162787 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:32.127526 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/963a8b12-5921-4d72-9c90-c31daf3d2475-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd\" (UID: \"963a8b12-5921-4d72-9c90-c31daf3d2475\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" Oct 30 18:15:32.162787 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:32.127583 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/963a8b12-5921-4d72-9c90-c31daf3d2475-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd\" (UID: \"963a8b12-5921-4d72-9c90-c31daf3d2475\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" Oct 30 18:15:32.162787 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:32.127615 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/963a8b12-5921-4d72-9c90-c31daf3d2475-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd\" (UID: \"963a8b12-5921-4d72-9c90-c31daf3d2475\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" Oct 30 18:15:32.162787 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:32.127669 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/963a8b12-5921-4d72-9c90-c31daf3d2475-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd\" (UID: \"963a8b12-5921-4d72-9c90-c31daf3d2475\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" Oct 30 18:15:32.162787 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:32.127702 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/963a8b12-5921-4d72-9c90-c31daf3d2475-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd\" (UID: \"963a8b12-5921-4d72-9c90-c31daf3d2475\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" Oct 30 18:15:32.162787 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:32.127796 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7nf5\" (UniqueName: \"kubernetes.io/projected/963a8b12-5921-4d72-9c90-c31daf3d2475-kube-api-access-n7nf5\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd\" (UID: \"963a8b12-5921-4d72-9c90-c31daf3d2475\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" Oct 30 18:15:32.248159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:32.245661 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/963a8b12-5921-4d72-9c90-c31daf3d2475-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd\" (UID: \"963a8b12-5921-4d72-9c90-c31daf3d2475\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" Oct 30 18:15:32.248159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:32.245713 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/963a8b12-5921-4d72-9c90-c31daf3d2475-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd\" (UID: \"963a8b12-5921-4d72-9c90-c31daf3d2475\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" Oct 30 18:15:32.248159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:32.245781 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/963a8b12-5921-4d72-9c90-c31daf3d2475-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd\" (UID: \"963a8b12-5921-4d72-9c90-c31daf3d2475\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" Oct 30 18:15:32.248159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:32.245833 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/963a8b12-5921-4d72-9c90-c31daf3d2475-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd\" (UID: \"963a8b12-5921-4d72-9c90-c31daf3d2475\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" Oct 30 18:15:32.248159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:32.245863 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/963a8b12-5921-4d72-9c90-c31daf3d2475-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd\" (UID: \"963a8b12-5921-4d72-9c90-c31daf3d2475\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" Oct 30 18:15:32.248159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:32.245907 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/963a8b12-5921-4d72-9c90-c31daf3d2475-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd\" (UID: \"963a8b12-5921-4d72-9c90-c31daf3d2475\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" Oct 30 18:15:32.248159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:32.245936 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/963a8b12-5921-4d72-9c90-c31daf3d2475-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd\" (UID: \"963a8b12-5921-4d72-9c90-c31daf3d2475\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" Oct 30 18:15:32.248159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:32.246028 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7nf5\" (UniqueName: \"kubernetes.io/projected/963a8b12-5921-4d72-9c90-c31daf3d2475-kube-api-access-n7nf5\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd\" (UID: \"963a8b12-5921-4d72-9c90-c31daf3d2475\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" Oct 30 18:15:32.248159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:32.246112 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/963a8b12-5921-4d72-9c90-c31daf3d2475-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd\" (UID: \"963a8b12-5921-4d72-9c90-c31daf3d2475\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" Oct 30 18:15:32.248159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:32.247835 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/963a8b12-5921-4d72-9c90-c31daf3d2475-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd\" (UID: \"963a8b12-5921-4d72-9c90-c31daf3d2475\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" Oct 30 18:15:32.274154 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:32.248859 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/963a8b12-5921-4d72-9c90-c31daf3d2475-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd\" (UID: \"963a8b12-5921-4d72-9c90-c31daf3d2475\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" Oct 30 18:15:32.274154 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:32.249087 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/963a8b12-5921-4d72-9c90-c31daf3d2475-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd\" (UID: \"963a8b12-5921-4d72-9c90-c31daf3d2475\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" Oct 30 18:15:32.274154 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:32.255784 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/963a8b12-5921-4d72-9c90-c31daf3d2475-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd\" (UID: \"963a8b12-5921-4d72-9c90-c31daf3d2475\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" Oct 30 18:15:32.274154 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:32.256185 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/963a8b12-5921-4d72-9c90-c31daf3d2475-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd\" (UID: \"963a8b12-5921-4d72-9c90-c31daf3d2475\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" Oct 30 18:15:32.274154 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:32.263773 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd"] Oct 30 18:15:32.299445 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:32.278614 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/963a8b12-5921-4d72-9c90-c31daf3d2475-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd\" (UID: \"963a8b12-5921-4d72-9c90-c31daf3d2475\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" Oct 30 18:15:32.322796 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:32.320568 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:15:32.322796 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:32.320643 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" podUID="bafcd4d0-8a69-460a-9e0c-21b69dc4327a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 18:15:32.430828 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:32.378282 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7nf5\" (UniqueName: \"kubernetes.io/projected/963a8b12-5921-4d72-9c90-c31daf3d2475-kube-api-access-n7nf5\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd\" (UID: \"963a8b12-5921-4d72-9c90-c31daf3d2475\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" Oct 30 18:15:32.463122 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:32.463045 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/963a8b12-5921-4d72-9c90-c31daf3d2475-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd\" (UID: \"963a8b12-5921-4d72-9c90-c31daf3d2475\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" Oct 30 18:15:32.563269 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:32.563202 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/963a8b12-5921-4d72-9c90-c31daf3d2475-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd\" (UID: \"963a8b12-5921-4d72-9c90-c31daf3d2475\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" Oct 30 18:15:32.677164 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:32.670325 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" Oct 30 18:15:33.184319 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:33.184278 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:15:33.184601 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:33.184577 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" podUID="b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:33.208285 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:33.208239 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4" podUID="cf7d5f54-e091-4032-ab11-af2687e453c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:33.353178 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:33.338748 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:15:33.353178 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:33.338803 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" podUID="bafcd4d0-8a69-460a-9e0c-21b69dc4327a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 18:15:33.741043 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:33.740940 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8" podUID="1e2901e8-4919-4e5c-88c7-39126a7cca56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:33.991698 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:33.990227 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd"] Oct 30 18:15:34.045174 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:15:34.045127 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod963a8b12_5921_4d72_9c90_c31daf3d2475.slice/crio-5c3c6b24cc14f426422770564cc006eab3613f8b6c92339b98306582f296bfec WatchSource:0}: Error finding container 5c3c6b24cc14f426422770564cc006eab3613f8b6c92339b98306582f296bfec: Status 404 returned error can't find the container with id 5c3c6b24cc14f426422770564cc006eab3613f8b6c92339b98306582f296bfec Oct 30 18:15:34.087767 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:34.086410 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:15:34.087767 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:34.086507 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:15:34.087767 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:34.086554 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:15:34.398218 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:34.395101 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:15:34.398218 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:34.395479 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" podUID="bafcd4d0-8a69-460a-9e0c-21b69dc4327a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 18:15:34.971017 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:34.970523 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" event={"ID":"963a8b12-5921-4d72-9c90-c31daf3d2475","Type":"ContainerStarted","Data":"5c3c6b24cc14f426422770564cc006eab3613f8b6c92339b98306582f296bfec"} Oct 30 18:15:35.304427 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:35.304253 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:15:35.304427 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:35.304317 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" podUID="bafcd4d0-8a69-460a-9e0c-21b69dc4327a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 18:15:36.241285 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:36.241248 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" event={"ID":"963a8b12-5921-4d72-9c90-c31daf3d2475","Type":"ContainerStarted","Data":"bde487d60584e625f8f08fc26f59e5e8ef0ef1a647ce814f3c84e7ae4fc2bf16"} Oct 30 18:15:36.278193 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:36.259007 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q"] Oct 30 18:15:36.280638 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:36.280592 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" podUID="50067d7b-521e-403e-a128-20223ce09044" containerName="istio-proxy" containerID="cri-o://634bbeed132f31a37362f2854acca358c5265126d0e02af5f853efa21ed39a90" gracePeriod=30 Oct 30 18:15:36.592720 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:36.591258 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:15:36.592720 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:36.591323 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" podUID="bafcd4d0-8a69-460a-9e0c-21b69dc4327a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 18:15:36.592952 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:36.592852 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:15:36.592952 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:36.592893 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" podUID="034e6427-0b95-4d7d-b6c9-05673f97a48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:36.680623 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:36.680588 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" Oct 30 18:15:36.713202 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:36.688553 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" podStartSLOduration=5.688532782 podStartE2EDuration="5.688532782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:36.496117043 +0000 UTC m=+1033.586179382" watchObservedRunningTime="2026-10-30 18:15:36.688532782 +0000 UTC m=+1033.778595123" Oct 30 18:15:36.739293 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:36.714016 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 18:15:36.740874 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:36.740815 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" podUID="963a8b12-5921-4d72-9c90-c31daf3d2475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 18:15:37.449484 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:37.433753 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:15:37.449484 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:37.433829 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" podUID="bafcd4d0-8a69-460a-9e0c-21b69dc4327a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 18:15:38.145164 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:38.142677 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 18:15:38.150560 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:38.150515 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" podUID="963a8b12-5921-4d72-9c90-c31daf3d2475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 18:15:38.262428 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:15:38.262381 4442 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.16s" Oct 30 18:15:38.401667 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:38.401567 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:15:38.401974 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:38.401939 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" podUID="bafcd4d0-8a69-460a-9e0c-21b69dc4327a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 18:15:38.839214 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:38.831259 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 18:15:38.839214 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:38.831329 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" podUID="963a8b12-5921-4d72-9c90-c31daf3d2475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 18:15:39.303155 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:39.301866 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:15:39.303441 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:39.303412 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" podUID="bafcd4d0-8a69-460a-9e0c-21b69dc4327a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 18:15:39.677162 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:39.676711 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 18:15:39.677162 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:39.676775 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" podUID="963a8b12-5921-4d72-9c90-c31daf3d2475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 18:15:40.301342 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:40.300850 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:15:40.301342 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:40.300915 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" podUID="bafcd4d0-8a69-460a-9e0c-21b69dc4327a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 18:15:40.488521 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:40.488485 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-x2rr2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:15:40.488846 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:40.488801 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" podUID="e03523d4-1b6f-4a80-9006-26b0345d809d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:40.687223 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:40.687189 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 18:15:40.687479 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:40.687451 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" podUID="963a8b12-5921-4d72-9c90-c31daf3d2475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 18:15:41.299004 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:41.298972 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:15:41.299398 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:41.299346 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" podUID="bafcd4d0-8a69-460a-9e0c-21b69dc4327a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 18:15:41.674589 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:41.674554 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 18:15:41.674989 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:41.674609 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" podUID="963a8b12-5921-4d72-9c90-c31daf3d2475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 18:15:42.298770 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:42.298740 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" Oct 30 18:15:42.684555 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:42.684516 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 18:15:42.685150 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:42.685098 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" podUID="963a8b12-5921-4d72-9c90-c31daf3d2475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 18:15:43.356258 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:43.353312 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:15:43.356258 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:43.353397 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" podUID="bafcd4d0-8a69-460a-9e0c-21b69dc4327a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:43.684210 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:43.684167 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 18:15:43.684521 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:43.684474 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" podUID="963a8b12-5921-4d72-9c90-c31daf3d2475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 18:15:43.743161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:43.742991 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" Oct 30 18:15:43.756163 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:43.755645 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" Oct 30 18:15:43.914406 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:43.914374 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-f5f5q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:15:43.914566 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:43.914433 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" podUID="50067d7b-521e-403e-a128-20223ce09044" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:15:44.681545 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:44.676681 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 18:15:44.681545 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:44.676745 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" podUID="963a8b12-5921-4d72-9c90-c31daf3d2475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 18:15:45.738156 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:45.737208 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" Oct 30 18:15:45.738675 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:45.738646 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" Oct 30 18:15:45.741318 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:45.741296 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" Oct 30 18:15:46.566643 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:46.566618 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" Oct 30 18:15:46.591743 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:46.591713 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50067d7b-521e-403e-a128-20223ce09044-kube-api-access-gxs75\" (UniqueName: \"kubernetes.io/projected/50067d7b-521e-403e-a128-20223ce09044-kube-api-access-gxs75\") pod \"50067d7b-521e-403e-a128-20223ce09044\" (UID: \"50067d7b-521e-403e-a128-20223ce09044\") " Oct 30 18:15:46.591869 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:46.591821 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50067d7b-521e-403e-a128-20223ce09044-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50067d7b-521e-403e-a128-20223ce09044-workload-certs\") pod \"50067d7b-521e-403e-a128-20223ce09044\" (UID: \"50067d7b-521e-403e-a128-20223ce09044\") " Oct 30 18:15:46.591929 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:46.591908 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50067d7b-521e-403e-a128-20223ce09044-istio-token\" (UniqueName: \"kubernetes.io/projected/50067d7b-521e-403e-a128-20223ce09044-istio-token\") pod \"50067d7b-521e-403e-a128-20223ce09044\" (UID: \"50067d7b-521e-403e-a128-20223ce09044\") " Oct 30 18:15:46.591964 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:46.591938 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50067d7b-521e-403e-a128-20223ce09044-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50067d7b-521e-403e-a128-20223ce09044-workload-socket\") pod \"50067d7b-521e-403e-a128-20223ce09044\" (UID: \"50067d7b-521e-403e-a128-20223ce09044\") " Oct 30 18:15:46.592024 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:46.592009 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50067d7b-521e-403e-a128-20223ce09044-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50067d7b-521e-403e-a128-20223ce09044-credential-socket\") pod \"50067d7b-521e-403e-a128-20223ce09044\" (UID: \"50067d7b-521e-403e-a128-20223ce09044\") " Oct 30 18:15:46.592098 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:46.592077 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50067d7b-521e-403e-a128-20223ce09044-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50067d7b-521e-403e-a128-20223ce09044-istio-podinfo\") pod \"50067d7b-521e-403e-a128-20223ce09044\" (UID: \"50067d7b-521e-403e-a128-20223ce09044\") " Oct 30 18:15:46.592247 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:46.592154 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50067d7b-521e-403e-a128-20223ce09044-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50067d7b-521e-403e-a128-20223ce09044-istio-data\") pod \"50067d7b-521e-403e-a128-20223ce09044\" (UID: \"50067d7b-521e-403e-a128-20223ce09044\") " Oct 30 18:15:46.592247 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:46.592168 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50067d7b-521e-403e-a128-20223ce09044-workload-certs" pod "50067d7b-521e-403e-a128-20223ce09044" (UID: "50067d7b-521e-403e-a128-20223ce09044"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:15:46.592247 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:46.592189 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50067d7b-521e-403e-a128-20223ce09044-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50067d7b-521e-403e-a128-20223ce09044-istio-envoy\") pod \"50067d7b-521e-403e-a128-20223ce09044\" (UID: \"50067d7b-521e-403e-a128-20223ce09044\") " Oct 30 18:15:46.592485 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:46.592279 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50067d7b-521e-403e-a128-20223ce09044-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50067d7b-521e-403e-a128-20223ce09044-istiod-ca-cert\") pod \"50067d7b-521e-403e-a128-20223ce09044\" (UID: \"50067d7b-521e-403e-a128-20223ce09044\") " Oct 30 18:15:46.592485 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:46.592336 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50067d7b-521e-403e-a128-20223ce09044-workload-socket" pod "50067d7b-521e-403e-a128-20223ce09044" (UID: "50067d7b-521e-403e-a128-20223ce09044"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:15:46.592485 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:46.592349 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50067d7b-521e-403e-a128-20223ce09044-credential-socket" pod "50067d7b-521e-403e-a128-20223ce09044" (UID: "50067d7b-521e-403e-a128-20223ce09044"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:15:46.592655 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:46.592574 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50067d7b-521e-403e-a128-20223ce09044-istio-data" pod "50067d7b-521e-403e-a128-20223ce09044" (UID: "50067d7b-521e-403e-a128-20223ce09044"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:15:46.593028 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:46.593003 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50067d7b-521e-403e-a128-20223ce09044-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:15:46.593153 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:46.593032 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50067d7b-521e-403e-a128-20223ce09044-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:15:46.593153 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:46.593046 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50067d7b-521e-403e-a128-20223ce09044-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:15:46.593153 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:46.593061 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50067d7b-521e-403e-a128-20223ce09044-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:15:46.593332 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:46.593126 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50067d7b-521e-403e-a128-20223ce09044-istiod-ca-cert" pod "50067d7b-521e-403e-a128-20223ce09044" (UID: "50067d7b-521e-403e-a128-20223ce09044"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:46.594304 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:46.594283 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50067d7b-521e-403e-a128-20223ce09044-kube-api-access-gxs75" pod "50067d7b-521e-403e-a128-20223ce09044" (UID: "50067d7b-521e-403e-a128-20223ce09044"). InnerVolumeSpecName "kube-api-access-gxs75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:46.594474 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:46.594457 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50067d7b-521e-403e-a128-20223ce09044-istio-token" pod "50067d7b-521e-403e-a128-20223ce09044" (UID: "50067d7b-521e-403e-a128-20223ce09044"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:46.594537 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:46.594478 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50067d7b-521e-403e-a128-20223ce09044-istio-envoy" pod "50067d7b-521e-403e-a128-20223ce09044" (UID: "50067d7b-521e-403e-a128-20223ce09044"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:15:46.594537 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:46.594518 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50067d7b-521e-403e-a128-20223ce09044-istio-podinfo" pod "50067d7b-521e-403e-a128-20223ce09044" (UID: "50067d7b-521e-403e-a128-20223ce09044"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:15:46.693792 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:46.693756 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50067d7b-521e-403e-a128-20223ce09044-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:15:46.693792 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:46.693788 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50067d7b-521e-403e-a128-20223ce09044-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:15:46.693792 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:46.693797 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50067d7b-521e-403e-a128-20223ce09044-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:15:46.694019 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:46.693805 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50067d7b-521e-403e-a128-20223ce09044-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:15:46.694019 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:46.693814 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxs75\" (UniqueName: \"kubernetes.io/projected/50067d7b-521e-403e-a128-20223ce09044-kube-api-access-gxs75\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:15:47.140452 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:47.140419 4442 generic.go:356] "Generic (PLEG): container finished" podID="50067d7b-521e-403e-a128-20223ce09044" containerID="634bbeed132f31a37362f2854acca358c5265126d0e02af5f853efa21ed39a90" exitCode=0 Oct 30 18:15:47.140901 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:47.140485 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" Oct 30 18:15:47.140901 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:47.140510 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" event={"ID":"50067d7b-521e-403e-a128-20223ce09044","Type":"ContainerDied","Data":"634bbeed132f31a37362f2854acca358c5265126d0e02af5f853efa21ed39a90"} Oct 30 18:15:47.140901 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:47.140582 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q" event={"ID":"50067d7b-521e-403e-a128-20223ce09044","Type":"ContainerDied","Data":"738b393143f4f2758c9d9252bbe5ec07984d6daa33776bc1f6744af3f57b4b8a"} Oct 30 18:15:47.140901 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:47.140621 4442 scope.go:122] "RemoveContainer" containerID="634bbeed132f31a37362f2854acca358c5265126d0e02af5f853efa21ed39a90" Oct 30 18:15:47.150766 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:47.150750 4442 scope.go:122] "RemoveContainer" containerID="634bbeed132f31a37362f2854acca358c5265126d0e02af5f853efa21ed39a90" Oct 30 18:15:47.151064 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:15:47.151045 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"634bbeed132f31a37362f2854acca358c5265126d0e02af5f853efa21ed39a90\": container with ID starting with 634bbeed132f31a37362f2854acca358c5265126d0e02af5f853efa21ed39a90 not found: ID does not exist" containerID="634bbeed132f31a37362f2854acca358c5265126d0e02af5f853efa21ed39a90" Oct 30 18:15:47.151147 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:47.151076 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"634bbeed132f31a37362f2854acca358c5265126d0e02af5f853efa21ed39a90"} err="failed to get container status \"634bbeed132f31a37362f2854acca358c5265126d0e02af5f853efa21ed39a90\": rpc error: code = NotFound desc = could not find container \"634bbeed132f31a37362f2854acca358c5265126d0e02af5f853efa21ed39a90\": container with ID starting with 634bbeed132f31a37362f2854acca358c5265126d0e02af5f853efa21ed39a90 not found: ID does not exist" Oct 30 18:15:47.203450 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:47.203423 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q"] Oct 30 18:15:47.214957 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:47.214912 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-f5f5q"] Oct 30 18:15:47.560751 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:15:47.560700 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:15:49.065893 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:49.065848 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50067d7b-521e-403e-a128-20223ce09044" path="/var/lib/kubelet/pods/50067d7b-521e-403e-a128-20223ce09044/volumes" Oct 30 18:15:52.207328 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:52.207294 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2"] Oct 30 18:15:52.207911 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:52.207897 4442 cpu_manager.go:414] "containerMap: removing container" podUID="50067d7b-521e-403e-a128-20223ce09044" containerName="istio-proxy" Oct 30 18:15:52.207966 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:52.207912 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50067d7b-521e-403e-a128-20223ce09044" containerName="istio-proxy" Oct 30 18:15:52.208029 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:52.208020 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="50067d7b-521e-403e-a128-20223ce09044" containerName="istio-proxy" Oct 30 18:15:52.213092 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:52.213070 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" Oct 30 18:15:52.229076 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:52.229048 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-4b709d-openshift-default-dockercfg-6cjqm\"" Oct 30 18:15:52.267530 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:52.264333 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2"] Oct 30 18:15:52.283592 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:52.283553 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/059512a2-ce79-4426-bd6d-b11fede6864f-istio-data\") pod \"e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2\" (UID: \"059512a2-ce79-4426-bd6d-b11fede6864f\") " pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" Oct 30 18:15:52.283757 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:52.283638 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/059512a2-ce79-4426-bd6d-b11fede6864f-istio-token\") pod \"e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2\" (UID: \"059512a2-ce79-4426-bd6d-b11fede6864f\") " pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" Oct 30 18:15:52.283799 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:52.283751 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/059512a2-ce79-4426-bd6d-b11fede6864f-workload-socket\") pod \"e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2\" (UID: \"059512a2-ce79-4426-bd6d-b11fede6864f\") " pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" Oct 30 18:15:52.283799 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:52.283786 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/059512a2-ce79-4426-bd6d-b11fede6864f-credential-socket\") pod \"e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2\" (UID: \"059512a2-ce79-4426-bd6d-b11fede6864f\") " pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" Oct 30 18:15:52.283906 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:52.283883 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/059512a2-ce79-4426-bd6d-b11fede6864f-istiod-ca-cert\") pod \"e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2\" (UID: \"059512a2-ce79-4426-bd6d-b11fede6864f\") " pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" Oct 30 18:15:52.283976 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:52.283959 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/059512a2-ce79-4426-bd6d-b11fede6864f-istio-envoy\") pod \"e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2\" (UID: \"059512a2-ce79-4426-bd6d-b11fede6864f\") " pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" Oct 30 18:15:52.284107 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:52.284092 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/059512a2-ce79-4426-bd6d-b11fede6864f-workload-certs\") pod \"e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2\" (UID: \"059512a2-ce79-4426-bd6d-b11fede6864f\") " pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" Oct 30 18:15:52.284200 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:52.284182 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnrvz\" (UniqueName: \"kubernetes.io/projected/059512a2-ce79-4426-bd6d-b11fede6864f-kube-api-access-wnrvz\") pod \"e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2\" (UID: \"059512a2-ce79-4426-bd6d-b11fede6864f\") " pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" Oct 30 18:15:52.284258 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:52.284231 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/059512a2-ce79-4426-bd6d-b11fede6864f-istio-podinfo\") pod \"e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2\" (UID: \"059512a2-ce79-4426-bd6d-b11fede6864f\") " pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" Oct 30 18:15:52.385631 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:52.385596 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/059512a2-ce79-4426-bd6d-b11fede6864f-istio-token\") pod \"e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2\" (UID: \"059512a2-ce79-4426-bd6d-b11fede6864f\") " pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" Oct 30 18:15:52.385826 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:52.385648 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/059512a2-ce79-4426-bd6d-b11fede6864f-workload-socket\") pod \"e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2\" (UID: \"059512a2-ce79-4426-bd6d-b11fede6864f\") " pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" Oct 30 18:15:52.385826 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:52.385674 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/059512a2-ce79-4426-bd6d-b11fede6864f-credential-socket\") pod \"e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2\" (UID: \"059512a2-ce79-4426-bd6d-b11fede6864f\") " pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" Oct 30 18:15:52.385826 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:52.385711 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/059512a2-ce79-4426-bd6d-b11fede6864f-istiod-ca-cert\") pod \"e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2\" (UID: \"059512a2-ce79-4426-bd6d-b11fede6864f\") " pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" Oct 30 18:15:52.385826 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:52.385753 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/059512a2-ce79-4426-bd6d-b11fede6864f-istio-envoy\") pod \"e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2\" (UID: \"059512a2-ce79-4426-bd6d-b11fede6864f\") " pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" Oct 30 18:15:52.386055 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:52.385856 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/059512a2-ce79-4426-bd6d-b11fede6864f-workload-certs\") pod \"e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2\" (UID: \"059512a2-ce79-4426-bd6d-b11fede6864f\") " pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" Oct 30 18:15:52.386055 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:52.385910 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnrvz\" (UniqueName: \"kubernetes.io/projected/059512a2-ce79-4426-bd6d-b11fede6864f-kube-api-access-wnrvz\") pod \"e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2\" (UID: \"059512a2-ce79-4426-bd6d-b11fede6864f\") " pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" Oct 30 18:15:52.386055 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:52.385951 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/059512a2-ce79-4426-bd6d-b11fede6864f-istio-podinfo\") pod \"e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2\" (UID: \"059512a2-ce79-4426-bd6d-b11fede6864f\") " pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" Oct 30 18:15:52.386278 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:52.386058 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/059512a2-ce79-4426-bd6d-b11fede6864f-workload-socket\") pod \"e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2\" (UID: \"059512a2-ce79-4426-bd6d-b11fede6864f\") " pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" Oct 30 18:15:52.386278 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:52.386077 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/059512a2-ce79-4426-bd6d-b11fede6864f-istio-data\") pod \"e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2\" (UID: \"059512a2-ce79-4426-bd6d-b11fede6864f\") " pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" Oct 30 18:15:52.386278 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:52.386107 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/059512a2-ce79-4426-bd6d-b11fede6864f-credential-socket\") pod \"e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2\" (UID: \"059512a2-ce79-4426-bd6d-b11fede6864f\") " pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" Oct 30 18:15:52.386447 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:52.386327 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/059512a2-ce79-4426-bd6d-b11fede6864f-workload-certs\") pod \"e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2\" (UID: \"059512a2-ce79-4426-bd6d-b11fede6864f\") " pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" Oct 30 18:15:52.386447 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:52.386387 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/059512a2-ce79-4426-bd6d-b11fede6864f-istio-data\") pod \"e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2\" (UID: \"059512a2-ce79-4426-bd6d-b11fede6864f\") " pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" Oct 30 18:15:52.386562 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:52.386530 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/059512a2-ce79-4426-bd6d-b11fede6864f-istiod-ca-cert\") pod \"e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2\" (UID: \"059512a2-ce79-4426-bd6d-b11fede6864f\") " pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" Oct 30 18:15:52.388147 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:52.388114 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/059512a2-ce79-4426-bd6d-b11fede6864f-istio-envoy\") pod \"e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2\" (UID: \"059512a2-ce79-4426-bd6d-b11fede6864f\") " pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" Oct 30 18:15:52.388763 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:52.388743 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/059512a2-ce79-4426-bd6d-b11fede6864f-istio-podinfo\") pod \"e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2\" (UID: \"059512a2-ce79-4426-bd6d-b11fede6864f\") " pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" Oct 30 18:15:52.405190 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:52.405166 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/059512a2-ce79-4426-bd6d-b11fede6864f-istio-token\") pod \"e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2\" (UID: \"059512a2-ce79-4426-bd6d-b11fede6864f\") " pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" Oct 30 18:15:52.405490 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:52.405468 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnrvz\" (UniqueName: \"kubernetes.io/projected/059512a2-ce79-4426-bd6d-b11fede6864f-kube-api-access-wnrvz\") pod \"e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2\" (UID: \"059512a2-ce79-4426-bd6d-b11fede6864f\") " pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" Oct 30 18:15:52.532853 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:52.532404 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" Oct 30 18:15:53.276193 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:53.275256 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:15:53.276193 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:53.275433 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:15:53.276193 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:53.275529 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:15:53.281174 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:53.279188 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2"] Oct 30 18:15:53.288792 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:53.288750 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" event={"ID":"059512a2-ce79-4426-bd6d-b11fede6864f","Type":"ContainerStarted","Data":"4f677f5a6ae28cebcb12542779ac9d9c1be15797bfcaf34739d087e662b618f8"} Oct 30 18:15:54.298729 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:54.298694 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" event={"ID":"059512a2-ce79-4426-bd6d-b11fede6864f","Type":"ContainerStarted","Data":"7fd2dc80a09a09123b64a0e7047b73ebd91c5533354165648e6f964709503e5e"} Oct 30 18:15:54.356963 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:54.356897 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" podStartSLOduration=2.356879772 podStartE2EDuration="2.356879772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:54.353419851 +0000 UTC m=+1051.443482191" watchObservedRunningTime="2026-10-30 18:15:54.356879772 +0000 UTC m=+1051.446942110" Oct 30 18:15:54.543483 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:54.543439 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" Oct 30 18:15:54.557962 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:54.554094 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" Oct 30 18:15:55.337159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:55.336454 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" Oct 30 18:15:55.349490 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:55.349463 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" Oct 30 18:15:59.270159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:59.214300 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4" podUID="cf7d5f54-e091-4032-ab11-af2687e453c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:59.846409 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:59.846240 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:15:59.846409 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:59.846309 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" podUID="bafcd4d0-8a69-460a-9e0c-21b69dc4327a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:59.934236 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:59.933852 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-vlm9c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:15:59.934236 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:15:59.933921 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" podUID="dab58a66-301a-4e74-9375-95bd4964b644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:00.707149 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:00.707094 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-5jbdk"] Oct 30 18:16:00.781181 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:00.775329 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-5jbdk" Oct 30 18:16:00.814090 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:00.813854 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-0c6613ea\"" Oct 30 18:16:00.814090 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:00.813985 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-0c6613ea-dockercfg-vjgg2\"" Oct 30 18:16:00.814090 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:00.814019 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-5jbdk"] Oct 30 18:16:01.102155 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:01.098013 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz2fs\" (UniqueName: \"kubernetes.io/projected/5b5cc5ed-a8f3-434e-ae4b-57217ad6b320-kube-api-access-dz2fs\") pod \"payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-5jbdk\" (UID: \"5b5cc5ed-a8f3-434e-ae4b-57217ad6b320\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-5jbdk" Oct 30 18:16:01.102155 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:01.098098 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b5cc5ed-a8f3-434e-ae4b-57217ad6b320-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-5jbdk\" (UID: \"5b5cc5ed-a8f3-434e-ae4b-57217ad6b320\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-5jbdk" Oct 30 18:16:01.298270 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:01.279661 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b5cc5ed-a8f3-434e-ae4b-57217ad6b320-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-5jbdk\" (UID: \"5b5cc5ed-a8f3-434e-ae4b-57217ad6b320\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-5jbdk" Oct 30 18:16:01.302605 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:01.302567 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz2fs\" (UniqueName: \"kubernetes.io/projected/5b5cc5ed-a8f3-434e-ae4b-57217ad6b320-kube-api-access-dz2fs\") pod \"payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-5jbdk\" (UID: \"5b5cc5ed-a8f3-434e-ae4b-57217ad6b320\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-5jbdk" Oct 30 18:16:01.303276 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:01.299541 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b5cc5ed-a8f3-434e-ae4b-57217ad6b320-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-5jbdk\" (UID: \"5b5cc5ed-a8f3-434e-ae4b-57217ad6b320\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-5jbdk" Oct 30 18:16:01.309192 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:01.309163 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-2zwb7"] Oct 30 18:16:01.337792 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:01.337752 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-2zwb7"] Oct 30 18:16:01.338192 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:01.338121 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-2zwb7" Oct 30 18:16:01.389566 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:01.389429 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz2fs\" (UniqueName: \"kubernetes.io/projected/5b5cc5ed-a8f3-434e-ae4b-57217ad6b320-kube-api-access-dz2fs\") pod \"payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-5jbdk\" (UID: \"5b5cc5ed-a8f3-434e-ae4b-57217ad6b320\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-5jbdk" Oct 30 18:16:01.468160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:01.454706 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-5jbdk" Oct 30 18:16:01.504981 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:01.504951 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e938b627-f318-4c80-abd3-8b4634431ec1-plugins-config-volume\") pod \"payload-processing-e2e-mt-0c6613ea-86469f66f4-2zwb7\" (UID: \"e938b627-f318-4c80-abd3-8b4634431ec1\") " pod="openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-2zwb7" Oct 30 18:16:01.505409 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:01.505387 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdf9w\" (UniqueName: \"kubernetes.io/projected/e938b627-f318-4c80-abd3-8b4634431ec1-kube-api-access-pdf9w\") pod \"payload-processing-e2e-mt-0c6613ea-86469f66f4-2zwb7\" (UID: \"e938b627-f318-4c80-abd3-8b4634431ec1\") " pod="openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-2zwb7" Oct 30 18:16:01.611988 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:01.611655 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdf9w\" (UniqueName: \"kubernetes.io/projected/e938b627-f318-4c80-abd3-8b4634431ec1-kube-api-access-pdf9w\") pod \"payload-processing-e2e-mt-0c6613ea-86469f66f4-2zwb7\" (UID: \"e938b627-f318-4c80-abd3-8b4634431ec1\") " pod="openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-2zwb7" Oct 30 18:16:01.611988 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:01.611717 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e938b627-f318-4c80-abd3-8b4634431ec1-plugins-config-volume\") pod \"payload-processing-e2e-mt-0c6613ea-86469f66f4-2zwb7\" (UID: \"e938b627-f318-4c80-abd3-8b4634431ec1\") " pod="openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-2zwb7" Oct 30 18:16:01.613079 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:01.612771 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e938b627-f318-4c80-abd3-8b4634431ec1-plugins-config-volume\") pod \"payload-processing-e2e-mt-0c6613ea-86469f66f4-2zwb7\" (UID: \"e938b627-f318-4c80-abd3-8b4634431ec1\") " pod="openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-2zwb7" Oct 30 18:16:01.659681 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:01.657563 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdf9w\" (UniqueName: \"kubernetes.io/projected/e938b627-f318-4c80-abd3-8b4634431ec1-kube-api-access-pdf9w\") pod \"payload-processing-e2e-mt-0c6613ea-86469f66f4-2zwb7\" (UID: \"e938b627-f318-4c80-abd3-8b4634431ec1\") " pod="openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-2zwb7" Oct 30 18:16:01.717315 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:01.716012 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-2zwb7" Oct 30 18:16:01.899638 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:01.899581 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-5jbdk"] Oct 30 18:16:02.320543 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:02.309906 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-5jbdk" event={"ID":"5b5cc5ed-a8f3-434e-ae4b-57217ad6b320","Type":"ContainerStarted","Data":"db50ddefcd21b0b5b6e1361e465de45e58e33f70ab69458c24162ea37995d959"} Oct 30 18:16:02.358246 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:02.358210 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-2zwb7"] Oct 30 18:16:02.376694 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:16:02.367901 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode938b627_f318_4c80_abd3_8b4634431ec1.slice/crio-33cdacf131f0308600aafce376dd8b7042fbb74e6706b486f8b2c986dea89393 WatchSource:0}: Error finding container 33cdacf131f0308600aafce376dd8b7042fbb74e6706b486f8b2c986dea89393: Status 404 returned error can't find the container with id 33cdacf131f0308600aafce376dd8b7042fbb74e6706b486f8b2c986dea89393 Oct 30 18:16:02.726229 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:02.726103 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4b709d-5df8f677bc-b4srr"] Oct 30 18:16:02.730811 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:02.730769 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4b709d-5df8f677bc-b4srr" Oct 30 18:16:02.748747 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:02.748712 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4b709d-5df8f677bc-b4srr"] Oct 30 18:16:02.755186 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:02.754478 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-4b709d\"" Oct 30 18:16:02.758428 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:02.755683 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-4b709d-dockercfg-5zrbc\"" Oct 30 18:16:02.855158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:02.851652 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbq96\" (UniqueName: \"kubernetes.io/projected/361a8ea8-96e6-404d-b694-052dbb6fb070-kube-api-access-tbq96\") pod \"payload-pre-processing-e2e-api-a-4b709d-5df8f677bc-b4srr\" (UID: \"361a8ea8-96e6-404d-b694-052dbb6fb070\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4b709d-5df8f677bc-b4srr" Oct 30 18:16:02.855158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:02.851862 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/361a8ea8-96e6-404d-b694-052dbb6fb070-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4b709d-5df8f677bc-b4srr\" (UID: \"361a8ea8-96e6-404d-b694-052dbb6fb070\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4b709d-5df8f677bc-b4srr" Oct 30 18:16:02.958170 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:02.957500 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbq96\" (UniqueName: \"kubernetes.io/projected/361a8ea8-96e6-404d-b694-052dbb6fb070-kube-api-access-tbq96\") pod \"payload-pre-processing-e2e-api-a-4b709d-5df8f677bc-b4srr\" (UID: \"361a8ea8-96e6-404d-b694-052dbb6fb070\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4b709d-5df8f677bc-b4srr" Oct 30 18:16:02.958170 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:02.957713 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/361a8ea8-96e6-404d-b694-052dbb6fb070-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4b709d-5df8f677bc-b4srr\" (UID: \"361a8ea8-96e6-404d-b694-052dbb6fb070\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4b709d-5df8f677bc-b4srr" Oct 30 18:16:02.984046 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:02.970975 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/361a8ea8-96e6-404d-b694-052dbb6fb070-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4b709d-5df8f677bc-b4srr\" (UID: \"361a8ea8-96e6-404d-b694-052dbb6fb070\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4b709d-5df8f677bc-b4srr" Oct 30 18:16:02.984046 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:02.976278 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4b709d-7d96656586-hxnbn"] Oct 30 18:16:03.079257 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:03.072221 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbq96\" (UniqueName: \"kubernetes.io/projected/361a8ea8-96e6-404d-b694-052dbb6fb070-kube-api-access-tbq96\") pod \"payload-pre-processing-e2e-api-a-4b709d-5df8f677bc-b4srr\" (UID: \"361a8ea8-96e6-404d-b694-052dbb6fb070\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4b709d-5df8f677bc-b4srr" Oct 30 18:16:03.079257 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:03.079244 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4b709d-7d96656586-hxnbn" Oct 30 18:16:03.096073 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:03.093830 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4b709d-7d96656586-hxnbn"] Oct 30 18:16:03.269661 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:03.269559 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cc9578c-0f6a-48a4-a71c-c0a30dc75171-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4b709d-7d96656586-hxnbn\" (UID: \"0cc9578c-0f6a-48a4-a71c-c0a30dc75171\") " pod="openshift-ingress/payload-processing-e2e-api-a-4b709d-7d96656586-hxnbn" Oct 30 18:16:03.269816 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:03.269726 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsg5q\" (UniqueName: \"kubernetes.io/projected/0cc9578c-0f6a-48a4-a71c-c0a30dc75171-kube-api-access-nsg5q\") pod \"payload-processing-e2e-api-a-4b709d-7d96656586-hxnbn\" (UID: \"0cc9578c-0f6a-48a4-a71c-c0a30dc75171\") " pod="openshift-ingress/payload-processing-e2e-api-a-4b709d-7d96656586-hxnbn" Oct 30 18:16:03.320875 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:03.320840 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-5jbdk" event={"ID":"5b5cc5ed-a8f3-434e-ae4b-57217ad6b320","Type":"ContainerStarted","Data":"8bd8ae9df8b5006b2e67d5c9d6eb109835d0b60aa777fec9ac71ff8a826c7093"} Oct 30 18:16:03.321098 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:03.320909 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-5jbdk" Oct 30 18:16:03.322736 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:03.322707 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-2zwb7" event={"ID":"e938b627-f318-4c80-abd3-8b4634431ec1","Type":"ContainerStarted","Data":"37a9a36934e6c724131ecc40decf8dd03833ed45270d1d99cf84dff06210f3d6"} Oct 30 18:16:03.322882 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:03.322756 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-2zwb7" event={"ID":"e938b627-f318-4c80-abd3-8b4634431ec1","Type":"ContainerStarted","Data":"33cdacf131f0308600aafce376dd8b7042fbb74e6706b486f8b2c986dea89393"} Oct 30 18:16:03.322882 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:03.322803 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-2zwb7" Oct 30 18:16:03.346870 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:03.346827 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4b709d-5df8f677bc-b4srr" Oct 30 18:16:03.359317 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:03.359242 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-5jbdk" podStartSLOduration=3.3592237369999998 podStartE2EDuration="3.359223737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:03.357076756 +0000 UTC m=+1060.447139095" watchObservedRunningTime="2026-10-30 18:16:03.359223737 +0000 UTC m=+1060.449286077" Oct 30 18:16:03.371281 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:03.371248 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cc9578c-0f6a-48a4-a71c-c0a30dc75171-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4b709d-7d96656586-hxnbn\" (UID: \"0cc9578c-0f6a-48a4-a71c-c0a30dc75171\") " pod="openshift-ingress/payload-processing-e2e-api-a-4b709d-7d96656586-hxnbn" Oct 30 18:16:03.371556 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:03.371533 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsg5q\" (UniqueName: \"kubernetes.io/projected/0cc9578c-0f6a-48a4-a71c-c0a30dc75171-kube-api-access-nsg5q\") pod \"payload-processing-e2e-api-a-4b709d-7d96656586-hxnbn\" (UID: \"0cc9578c-0f6a-48a4-a71c-c0a30dc75171\") " pod="openshift-ingress/payload-processing-e2e-api-a-4b709d-7d96656586-hxnbn" Oct 30 18:16:03.372567 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:03.372531 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cc9578c-0f6a-48a4-a71c-c0a30dc75171-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4b709d-7d96656586-hxnbn\" (UID: \"0cc9578c-0f6a-48a4-a71c-c0a30dc75171\") " pod="openshift-ingress/payload-processing-e2e-api-a-4b709d-7d96656586-hxnbn" Oct 30 18:16:03.390281 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:03.390245 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsg5q\" (UniqueName: \"kubernetes.io/projected/0cc9578c-0f6a-48a4-a71c-c0a30dc75171-kube-api-access-nsg5q\") pod \"payload-processing-e2e-api-a-4b709d-7d96656586-hxnbn\" (UID: \"0cc9578c-0f6a-48a4-a71c-c0a30dc75171\") " pod="openshift-ingress/payload-processing-e2e-api-a-4b709d-7d96656586-hxnbn" Oct 30 18:16:03.397650 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:03.397594 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-2zwb7" podStartSLOduration=3.397579048 podStartE2EDuration="3.397579048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:03.393506029 +0000 UTC m=+1060.483568365" watchObservedRunningTime="2026-10-30 18:16:03.397579048 +0000 UTC m=+1060.487641386" Oct 30 18:16:03.415019 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:03.414990 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4b709d-7d96656586-hxnbn" Oct 30 18:16:03.680828 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:03.680804 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4b709d-5df8f677bc-b4srr"] Oct 30 18:16:03.734953 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:03.734931 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4b709d-7d96656586-hxnbn"] Oct 30 18:16:03.740522 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:16:03.740498 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cc9578c_0f6a_48a4_a71c_c0a30dc75171.slice/crio-ab3061310ae5f9e8f7108496979e54b35da899a091e0f2fc91d9c47bcc005c62 WatchSource:0}: Error finding container ab3061310ae5f9e8f7108496979e54b35da899a091e0f2fc91d9c47bcc005c62: Status 404 returned error can't find the container with id ab3061310ae5f9e8f7108496979e54b35da899a091e0f2fc91d9c47bcc005c62 Oct 30 18:16:04.333371 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:04.333325 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4b709d-7d96656586-hxnbn" event={"ID":"0cc9578c-0f6a-48a4-a71c-c0a30dc75171","Type":"ContainerStarted","Data":"596a8c7fed6de35c869328364dd06b27ea5d79038fc3fd05af12ece017583ebb"} Oct 30 18:16:04.333547 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:04.333394 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4b709d-7d96656586-hxnbn" event={"ID":"0cc9578c-0f6a-48a4-a71c-c0a30dc75171","Type":"ContainerStarted","Data":"ab3061310ae5f9e8f7108496979e54b35da899a091e0f2fc91d9c47bcc005c62"} Oct 30 18:16:04.333547 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:04.333469 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-4b709d-7d96656586-hxnbn" Oct 30 18:16:04.335453 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:04.335424 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4b709d-5df8f677bc-b4srr" event={"ID":"361a8ea8-96e6-404d-b694-052dbb6fb070","Type":"ContainerStarted","Data":"731ee324d9f8808cabfe7b5e56c20227c13224411bc2470b75001129c093d403"} Oct 30 18:16:04.335613 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:04.335499 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4b709d-5df8f677bc-b4srr" Oct 30 18:16:04.335613 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:04.335523 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4b709d-5df8f677bc-b4srr" event={"ID":"361a8ea8-96e6-404d-b694-052dbb6fb070","Type":"ContainerStarted","Data":"1fc4417deef02e4c5f6a2c83fea22010a2986be951ce04520f6e379f8ec727cf"} Oct 30 18:16:04.374324 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:04.374261 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-4b709d-7d96656586-hxnbn" podStartSLOduration=2.374246191 podStartE2EDuration="2.374246191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:04.370369511 +0000 UTC m=+1061.460431841" watchObservedRunningTime="2026-10-30 18:16:04.374246191 +0000 UTC m=+1061.464308569" Oct 30 18:16:04.414776 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:04.414724 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4b709d-5df8f677bc-b4srr" podStartSLOduration=2.414710946 podStartE2EDuration="2.414710946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:04.410350556 +0000 UTC m=+1061.500412895" watchObservedRunningTime="2026-10-30 18:16:04.414710946 +0000 UTC m=+1061.504773284" Oct 30 18:16:06.325885 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:06.325852 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr"] Oct 30 18:16:06.330717 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:06.330695 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" Oct 30 18:16:06.360459 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:06.360424 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlfh9\" (UniqueName: \"kubernetes.io/projected/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-kube-api-access-mlfh9\") pod \"maas-default-gateway-openshift-default-76597d6855-fkxzr\" (UID: \"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" Oct 30 18:16:06.360629 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:06.360533 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-fkxzr\" (UID: \"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" Oct 30 18:16:06.360712 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:06.360621 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-fkxzr\" (UID: \"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" Oct 30 18:16:06.360773 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:06.360720 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-fkxzr\" (UID: \"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" Oct 30 18:16:06.360827 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:06.360814 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-fkxzr\" (UID: \"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" Oct 30 18:16:06.360879 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:06.360853 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-fkxzr\" (UID: \"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" Oct 30 18:16:06.360933 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:06.360896 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-fkxzr\" (UID: \"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" Oct 30 18:16:06.361001 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:06.360973 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-fkxzr\" (UID: \"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" Oct 30 18:16:06.361056 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:06.361018 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-fkxzr\" (UID: \"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" Oct 30 18:16:06.363803 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:06.363775 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr"] Oct 30 18:16:06.462098 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:06.462055 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlfh9\" (UniqueName: \"kubernetes.io/projected/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-kube-api-access-mlfh9\") pod \"maas-default-gateway-openshift-default-76597d6855-fkxzr\" (UID: \"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" Oct 30 18:16:06.462326 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:06.462150 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-fkxzr\" (UID: \"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" Oct 30 18:16:06.462326 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:06.462182 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-fkxzr\" (UID: \"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" Oct 30 18:16:06.462326 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:06.462309 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-fkxzr\" (UID: \"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" Oct 30 18:16:06.462490 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:06.462411 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-fkxzr\" (UID: \"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" Oct 30 18:16:06.462490 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:06.462453 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-fkxzr\" (UID: \"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" Oct 30 18:16:06.462490 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:06.462485 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-fkxzr\" (UID: \"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" Oct 30 18:16:06.462640 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:06.462509 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-fkxzr\" (UID: \"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" Oct 30 18:16:06.462640 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:06.462569 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-fkxzr\" (UID: \"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" Oct 30 18:16:06.462640 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:06.462605 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-fkxzr\" (UID: \"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" Oct 30 18:16:06.462795 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:06.462671 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-fkxzr\" (UID: \"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" Oct 30 18:16:06.462795 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:06.462748 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-fkxzr\" (UID: \"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" Oct 30 18:16:06.462897 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:06.462813 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-fkxzr\" (UID: \"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" Oct 30 18:16:06.463290 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:06.463271 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-fkxzr\" (UID: \"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" Oct 30 18:16:06.464911 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:06.464887 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-fkxzr\" (UID: \"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" Oct 30 18:16:06.465147 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:06.465115 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-fkxzr\" (UID: \"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" Oct 30 18:16:06.481238 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:06.481207 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlfh9\" (UniqueName: \"kubernetes.io/projected/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-kube-api-access-mlfh9\") pod \"maas-default-gateway-openshift-default-76597d6855-fkxzr\" (UID: \"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" Oct 30 18:16:06.481360 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:06.481208 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-fkxzr\" (UID: \"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" Oct 30 18:16:06.647559 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:06.647515 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" Oct 30 18:16:07.011710 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:07.011689 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr"] Oct 30 18:16:07.012726 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:16:07.012699 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14c086ee_c4ba_48d2_a1b9_62cc8b09cc77.slice/crio-415a184bc7530bbb58630723ac2c21a8761fd6464061f1b34e91f911aa186f98 WatchSource:0}: Error finding container 415a184bc7530bbb58630723ac2c21a8761fd6464061f1b34e91f911aa186f98: Status 404 returned error can't find the container with id 415a184bc7530bbb58630723ac2c21a8761fd6464061f1b34e91f911aa186f98 Oct 30 18:16:07.015332 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:07.015300 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:16:07.015413 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:07.015362 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:16:07.015413 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:07.015394 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:16:07.366470 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:07.366431 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" event={"ID":"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77","Type":"ContainerStarted","Data":"5c2faa22a262a69ea313b229e32c09c3980ca9bc697bd9fbfe0e8944eb29a3a5"} Oct 30 18:16:07.366886 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:07.366499 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" event={"ID":"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77","Type":"ContainerStarted","Data":"415a184bc7530bbb58630723ac2c21a8761fd6464061f1b34e91f911aa186f98"} Oct 30 18:16:07.410774 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:07.410717 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" podStartSLOduration=1.410698694 podStartE2EDuration="1.410698694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:07.407591806 +0000 UTC m=+1064.497654171" watchObservedRunningTime="2026-10-30 18:16:07.410698694 +0000 UTC m=+1064.500761033" Oct 30 18:16:07.647771 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:07.647625 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" Oct 30 18:16:08.665158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:08.661272 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-fkxzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:16:08.665158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:08.661337 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" podUID="14c086ee-c4ba-48d2-a1b9-62cc8b09cc77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 18:16:09.080450 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:09.080376 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m"] Oct 30 18:16:09.087863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:09.085031 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" Oct 30 18:16:09.110479 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:09.110449 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m"] Oct 30 18:16:09.210979 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:09.210950 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q92v4\" (UniqueName: \"kubernetes.io/projected/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-kube-api-access-q92v4\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m\" (UID: \"d9b6ff41-9b4a-43e1-8a09-b99b2014530c\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" Oct 30 18:16:09.211189 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:09.210992 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m\" (UID: \"d9b6ff41-9b4a-43e1-8a09-b99b2014530c\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" Oct 30 18:16:09.211189 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:09.211060 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m\" (UID: \"d9b6ff41-9b4a-43e1-8a09-b99b2014530c\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" Oct 30 18:16:09.211189 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:09.211090 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m\" (UID: \"d9b6ff41-9b4a-43e1-8a09-b99b2014530c\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" Oct 30 18:16:09.211189 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:09.211170 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m\" (UID: \"d9b6ff41-9b4a-43e1-8a09-b99b2014530c\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" Oct 30 18:16:09.211337 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:09.211206 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m\" (UID: \"d9b6ff41-9b4a-43e1-8a09-b99b2014530c\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" Oct 30 18:16:09.211337 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:09.211246 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m\" (UID: \"d9b6ff41-9b4a-43e1-8a09-b99b2014530c\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" Oct 30 18:16:09.211434 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:09.211414 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m\" (UID: \"d9b6ff41-9b4a-43e1-8a09-b99b2014530c\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" Oct 30 18:16:09.211528 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:09.211516 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m\" (UID: \"d9b6ff41-9b4a-43e1-8a09-b99b2014530c\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" Oct 30 18:16:09.312495 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:09.312461 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m\" (UID: \"d9b6ff41-9b4a-43e1-8a09-b99b2014530c\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" Oct 30 18:16:09.312657 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:09.312555 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m\" (UID: \"d9b6ff41-9b4a-43e1-8a09-b99b2014530c\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" Oct 30 18:16:09.312657 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:09.312599 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q92v4\" (UniqueName: \"kubernetes.io/projected/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-kube-api-access-q92v4\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m\" (UID: \"d9b6ff41-9b4a-43e1-8a09-b99b2014530c\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" Oct 30 18:16:09.312657 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:09.312630 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m\" (UID: \"d9b6ff41-9b4a-43e1-8a09-b99b2014530c\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" Oct 30 18:16:09.312657 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:09.312653 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m\" (UID: \"d9b6ff41-9b4a-43e1-8a09-b99b2014530c\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" Oct 30 18:16:09.312878 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:09.312678 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m\" (UID: \"d9b6ff41-9b4a-43e1-8a09-b99b2014530c\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" Oct 30 18:16:09.312878 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:09.312710 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m\" (UID: \"d9b6ff41-9b4a-43e1-8a09-b99b2014530c\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" Oct 30 18:16:09.312878 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:09.312740 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m\" (UID: \"d9b6ff41-9b4a-43e1-8a09-b99b2014530c\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" Oct 30 18:16:09.312878 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:09.312808 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m\" (UID: \"d9b6ff41-9b4a-43e1-8a09-b99b2014530c\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" Oct 30 18:16:09.313091 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:09.312893 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m\" (UID: \"d9b6ff41-9b4a-43e1-8a09-b99b2014530c\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" Oct 30 18:16:09.313091 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:09.312956 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m\" (UID: \"d9b6ff41-9b4a-43e1-8a09-b99b2014530c\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" Oct 30 18:16:09.313250 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:09.313113 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m\" (UID: \"d9b6ff41-9b4a-43e1-8a09-b99b2014530c\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" Oct 30 18:16:09.313250 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:09.313181 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m\" (UID: \"d9b6ff41-9b4a-43e1-8a09-b99b2014530c\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" Oct 30 18:16:09.313594 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:09.313576 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m\" (UID: \"d9b6ff41-9b4a-43e1-8a09-b99b2014530c\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" Oct 30 18:16:09.315331 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:09.315309 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m\" (UID: \"d9b6ff41-9b4a-43e1-8a09-b99b2014530c\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" Oct 30 18:16:09.315418 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:09.315359 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m\" (UID: \"d9b6ff41-9b4a-43e1-8a09-b99b2014530c\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" Oct 30 18:16:09.359464 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:09.359386 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q92v4\" (UniqueName: \"kubernetes.io/projected/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-kube-api-access-q92v4\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m\" (UID: \"d9b6ff41-9b4a-43e1-8a09-b99b2014530c\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" Oct 30 18:16:09.363335 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:09.363304 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m\" (UID: \"d9b6ff41-9b4a-43e1-8a09-b99b2014530c\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" Oct 30 18:16:09.403276 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:09.403222 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" Oct 30 18:16:09.653669 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:09.652585 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-fkxzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:16:09.653669 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:09.652646 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" podUID="14c086ee-c4ba-48d2-a1b9-62cc8b09cc77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 18:16:09.752793 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:16:09.752759 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9b6ff41_9b4a_43e1_8a09_b99b2014530c.slice/crio-758513067df1c18571fd8b40d45ed3101e86a794e1d0615e3a62a886d0b3bad7 WatchSource:0}: Error finding container 758513067df1c18571fd8b40d45ed3101e86a794e1d0615e3a62a886d0b3bad7: Status 404 returned error can't find the container with id 758513067df1c18571fd8b40d45ed3101e86a794e1d0615e3a62a886d0b3bad7 Oct 30 18:16:09.754328 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:09.754303 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m"] Oct 30 18:16:09.755428 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:09.755399 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:16:09.755501 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:09.755461 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:16:09.755501 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:09.755492 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:16:10.218565 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:10.218535 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-df2cs"] Oct 30 18:16:10.235158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:10.227426 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-df2cs" Oct 30 18:16:10.259535 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:10.257251 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-0c6613ea-serving-cert\"" Oct 30 18:16:10.299541 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:10.291242 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-df2cs"] Oct 30 18:16:10.367237 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:10.365964 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5cjl\" (UniqueName: \"kubernetes.io/projected/521d2b6a-4cee-4ee9-9eab-14f864927808-kube-api-access-h5cjl\") pod \"maas-api-e2e-mt-0c6613ea-75cb5c7f6d-df2cs\" (UID: \"521d2b6a-4cee-4ee9-9eab-14f864927808\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-df2cs" Oct 30 18:16:10.367237 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:10.366026 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/521d2b6a-4cee-4ee9-9eab-14f864927808-metrics-certs\") pod \"maas-api-e2e-mt-0c6613ea-75cb5c7f6d-df2cs\" (UID: \"521d2b6a-4cee-4ee9-9eab-14f864927808\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-df2cs" Oct 30 18:16:10.367237 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:10.366079 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/521d2b6a-4cee-4ee9-9eab-14f864927808-openshift-service-ca\") pod \"maas-api-e2e-mt-0c6613ea-75cb5c7f6d-df2cs\" (UID: \"521d2b6a-4cee-4ee9-9eab-14f864927808\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-df2cs" Oct 30 18:16:10.367237 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:10.366162 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/521d2b6a-4cee-4ee9-9eab-14f864927808-maas-api-tls\") pod \"maas-api-e2e-mt-0c6613ea-75cb5c7f6d-df2cs\" (UID: \"521d2b6a-4cee-4ee9-9eab-14f864927808\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-df2cs" Oct 30 18:16:10.397773 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:10.397733 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" event={"ID":"d9b6ff41-9b4a-43e1-8a09-b99b2014530c","Type":"ContainerStarted","Data":"b04013d4b8552b82935627b34ab847c949a27d0038b27a31e64e4676e7c67653"} Oct 30 18:16:10.398107 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:10.397806 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" event={"ID":"d9b6ff41-9b4a-43e1-8a09-b99b2014530c","Type":"ContainerStarted","Data":"758513067df1c18571fd8b40d45ed3101e86a794e1d0615e3a62a886d0b3bad7"} Oct 30 18:16:10.403646 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:10.403447 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" Oct 30 18:16:10.405214 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:10.405168 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 18:16:10.405545 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:10.405494 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" podUID="d9b6ff41-9b4a-43e1-8a09-b99b2014530c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 18:16:10.476747 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:10.476653 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5cjl\" (UniqueName: \"kubernetes.io/projected/521d2b6a-4cee-4ee9-9eab-14f864927808-kube-api-access-h5cjl\") pod \"maas-api-e2e-mt-0c6613ea-75cb5c7f6d-df2cs\" (UID: \"521d2b6a-4cee-4ee9-9eab-14f864927808\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-df2cs" Oct 30 18:16:10.478028 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:10.477913 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/521d2b6a-4cee-4ee9-9eab-14f864927808-metrics-certs\") pod \"maas-api-e2e-mt-0c6613ea-75cb5c7f6d-df2cs\" (UID: \"521d2b6a-4cee-4ee9-9eab-14f864927808\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-df2cs" Oct 30 18:16:10.478354 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:10.478243 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/521d2b6a-4cee-4ee9-9eab-14f864927808-openshift-service-ca\") pod \"maas-api-e2e-mt-0c6613ea-75cb5c7f6d-df2cs\" (UID: \"521d2b6a-4cee-4ee9-9eab-14f864927808\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-df2cs" Oct 30 18:16:10.482216 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:10.482192 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/521d2b6a-4cee-4ee9-9eab-14f864927808-maas-api-tls\") pod \"maas-api-e2e-mt-0c6613ea-75cb5c7f6d-df2cs\" (UID: \"521d2b6a-4cee-4ee9-9eab-14f864927808\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-df2cs" Oct 30 18:16:10.503998 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:10.503966 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/521d2b6a-4cee-4ee9-9eab-14f864927808-openshift-service-ca\") pod \"maas-api-e2e-mt-0c6613ea-75cb5c7f6d-df2cs\" (UID: \"521d2b6a-4cee-4ee9-9eab-14f864927808\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-df2cs" Oct 30 18:16:10.504330 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:16:10.482546 4442 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-serving-cert: secret "maas-api-e2e-mt-0c6613ea-serving-cert" not found Oct 30 18:16:10.504885 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:16:10.504861 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/521d2b6a-4cee-4ee9-9eab-14f864927808-maas-api-tls podName:521d2b6a-4cee-4ee9-9eab-14f864927808 nodeName:}" failed. No retries permitted until 2026-10-30 18:16:11.004659343 +0000 UTC m=+1068.094721674 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/521d2b6a-4cee-4ee9-9eab-14f864927808-maas-api-tls") pod "maas-api-e2e-mt-0c6613ea-75cb5c7f6d-df2cs" (UID: "521d2b6a-4cee-4ee9-9eab-14f864927808") : secret "maas-api-e2e-mt-0c6613ea-serving-cert" not found Oct 30 18:16:10.521179 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:10.510069 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" podStartSLOduration=1.5100477589999999 podStartE2EDuration="1.510047759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:10.475553025 +0000 UTC m=+1067.565615363" watchObservedRunningTime="2026-10-30 18:16:10.510047759 +0000 UTC m=+1067.600110099" Oct 30 18:16:10.527281 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:10.527252 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/521d2b6a-4cee-4ee9-9eab-14f864927808-metrics-certs\") pod \"maas-api-e2e-mt-0c6613ea-75cb5c7f6d-df2cs\" (UID: \"521d2b6a-4cee-4ee9-9eab-14f864927808\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-df2cs" Oct 30 18:16:10.533176 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:10.531856 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5cjl\" (UniqueName: \"kubernetes.io/projected/521d2b6a-4cee-4ee9-9eab-14f864927808-kube-api-access-h5cjl\") pod \"maas-api-e2e-mt-0c6613ea-75cb5c7f6d-df2cs\" (UID: \"521d2b6a-4cee-4ee9-9eab-14f864927808\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-df2cs" Oct 30 18:16:10.650236 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:10.649797 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-fkxzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:16:10.650236 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:10.649855 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" podUID="14c086ee-c4ba-48d2-a1b9-62cc8b09cc77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 18:16:11.022837 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:11.022803 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/521d2b6a-4cee-4ee9-9eab-14f864927808-maas-api-tls\") pod \"maas-api-e2e-mt-0c6613ea-75cb5c7f6d-df2cs\" (UID: \"521d2b6a-4cee-4ee9-9eab-14f864927808\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-df2cs" Oct 30 18:16:11.070174 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:11.062183 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/521d2b6a-4cee-4ee9-9eab-14f864927808-maas-api-tls\") pod \"maas-api-e2e-mt-0c6613ea-75cb5c7f6d-df2cs\" (UID: \"521d2b6a-4cee-4ee9-9eab-14f864927808\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-df2cs" Oct 30 18:16:11.179160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:11.175396 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-df2cs" Oct 30 18:16:11.408623 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:11.407604 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 18:16:11.408623 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:11.407677 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" podUID="d9b6ff41-9b4a-43e1-8a09-b99b2014530c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 18:16:11.659612 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:11.659529 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-fkxzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:16:11.726307 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:11.726256 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" podUID="14c086ee-c4ba-48d2-a1b9-62cc8b09cc77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 18:16:11.926046 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:11.925923 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-df2cs"] Oct 30 18:16:11.981000 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:16:11.980964 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod521d2b6a_4cee_4ee9_9eab_14f864927808.slice/crio-f737a4fbad76ee123dfeebfb25102d2542012d445d306070d67d05f0d57e4ec0 WatchSource:0}: Error finding container f737a4fbad76ee123dfeebfb25102d2542012d445d306070d67d05f0d57e4ec0: Status 404 returned error can't find the container with id f737a4fbad76ee123dfeebfb25102d2542012d445d306070d67d05f0d57e4ec0 Oct 30 18:16:12.403710 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:12.403661 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 18:16:12.404110 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:12.403735 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" podUID="d9b6ff41-9b4a-43e1-8a09-b99b2014530c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 18:16:12.427232 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:12.427197 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-df2cs" event={"ID":"521d2b6a-4cee-4ee9-9eab-14f864927808","Type":"ContainerStarted","Data":"f737a4fbad76ee123dfeebfb25102d2542012d445d306070d67d05f0d57e4ec0"} Oct 30 18:16:12.562158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:12.561580 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4b709d-754cc6c5c8-fchc4"] Oct 30 18:16:12.581379 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:12.581351 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b709d-754cc6c5c8-fchc4" Oct 30 18:16:12.583803 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:12.583694 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4b709d-754cc6c5c8-fchc4"] Oct 30 18:16:12.601410 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:12.601384 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-4b709d-serving-cert\"" Oct 30 18:16:12.647984 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:12.647948 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-fkxzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:16:12.661399 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:12.661306 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" podUID="14c086ee-c4ba-48d2-a1b9-62cc8b09cc77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 18:16:12.662509 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:12.662459 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6db04ea0-e110-4d71-9a90-f37b4c431a3b-metrics-certs\") pod \"maas-api-e2e-api-a-4b709d-754cc6c5c8-fchc4\" (UID: \"6db04ea0-e110-4d71-9a90-f37b4c431a3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b709d-754cc6c5c8-fchc4" Oct 30 18:16:12.662743 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:12.662723 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6db04ea0-e110-4d71-9a90-f37b4c431a3b-maas-api-tls\") pod \"maas-api-e2e-api-a-4b709d-754cc6c5c8-fchc4\" (UID: \"6db04ea0-e110-4d71-9a90-f37b4c431a3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b709d-754cc6c5c8-fchc4" Oct 30 18:16:12.662859 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:12.662843 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7dlj\" (UniqueName: \"kubernetes.io/projected/6db04ea0-e110-4d71-9a90-f37b4c431a3b-kube-api-access-v7dlj\") pod \"maas-api-e2e-api-a-4b709d-754cc6c5c8-fchc4\" (UID: \"6db04ea0-e110-4d71-9a90-f37b4c431a3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b709d-754cc6c5c8-fchc4" Oct 30 18:16:12.663247 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:12.663225 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6db04ea0-e110-4d71-9a90-f37b4c431a3b-openshift-service-ca\") pod \"maas-api-e2e-api-a-4b709d-754cc6c5c8-fchc4\" (UID: \"6db04ea0-e110-4d71-9a90-f37b4c431a3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b709d-754cc6c5c8-fchc4" Oct 30 18:16:12.785829 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:12.785793 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6db04ea0-e110-4d71-9a90-f37b4c431a3b-openshift-service-ca\") pod \"maas-api-e2e-api-a-4b709d-754cc6c5c8-fchc4\" (UID: \"6db04ea0-e110-4d71-9a90-f37b4c431a3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b709d-754cc6c5c8-fchc4" Oct 30 18:16:12.786189 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:12.786163 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6db04ea0-e110-4d71-9a90-f37b4c431a3b-metrics-certs\") pod \"maas-api-e2e-api-a-4b709d-754cc6c5c8-fchc4\" (UID: \"6db04ea0-e110-4d71-9a90-f37b4c431a3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b709d-754cc6c5c8-fchc4" Oct 30 18:16:12.786382 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:12.786366 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6db04ea0-e110-4d71-9a90-f37b4c431a3b-maas-api-tls\") pod \"maas-api-e2e-api-a-4b709d-754cc6c5c8-fchc4\" (UID: \"6db04ea0-e110-4d71-9a90-f37b4c431a3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b709d-754cc6c5c8-fchc4" Oct 30 18:16:12.786489 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:12.786475 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7dlj\" (UniqueName: \"kubernetes.io/projected/6db04ea0-e110-4d71-9a90-f37b4c431a3b-kube-api-access-v7dlj\") pod \"maas-api-e2e-api-a-4b709d-754cc6c5c8-fchc4\" (UID: \"6db04ea0-e110-4d71-9a90-f37b4c431a3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b709d-754cc6c5c8-fchc4" Oct 30 18:16:12.792294 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:12.788365 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6db04ea0-e110-4d71-9a90-f37b4c431a3b-openshift-service-ca\") pod \"maas-api-e2e-api-a-4b709d-754cc6c5c8-fchc4\" (UID: \"6db04ea0-e110-4d71-9a90-f37b4c431a3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b709d-754cc6c5c8-fchc4" Oct 30 18:16:12.887160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:12.882732 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6db04ea0-e110-4d71-9a90-f37b4c431a3b-maas-api-tls\") pod \"maas-api-e2e-api-a-4b709d-754cc6c5c8-fchc4\" (UID: \"6db04ea0-e110-4d71-9a90-f37b4c431a3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b709d-754cc6c5c8-fchc4" Oct 30 18:16:12.911906 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:12.903803 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7dlj\" (UniqueName: \"kubernetes.io/projected/6db04ea0-e110-4d71-9a90-f37b4c431a3b-kube-api-access-v7dlj\") pod \"maas-api-e2e-api-a-4b709d-754cc6c5c8-fchc4\" (UID: \"6db04ea0-e110-4d71-9a90-f37b4c431a3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b709d-754cc6c5c8-fchc4" Oct 30 18:16:12.911906 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:12.911725 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6db04ea0-e110-4d71-9a90-f37b4c431a3b-metrics-certs\") pod \"maas-api-e2e-api-a-4b709d-754cc6c5c8-fchc4\" (UID: \"6db04ea0-e110-4d71-9a90-f37b4c431a3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b709d-754cc6c5c8-fchc4" Oct 30 18:16:13.199147 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:13.199052 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b709d-754cc6c5c8-fchc4" Oct 30 18:16:13.458654 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:13.458049 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 18:16:13.458654 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:13.458105 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" podUID="d9b6ff41-9b4a-43e1-8a09-b99b2014530c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 18:16:13.528173 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:13.523752 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-df2cs" event={"ID":"521d2b6a-4cee-4ee9-9eab-14f864927808","Type":"ContainerStarted","Data":"d28900c8301e3f6262b54f065a73638dc25948f6020c2a542da359710897fc2f"} Oct 30 18:16:13.528173 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:13.524403 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-df2cs" Oct 30 18:16:13.590161 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:16:13.590077 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6db04ea0_e110_4d71_9a90_f37b4c431a3b.slice/crio-0a6b5ebd2fa2dc933e2afa486cbb28ebf25ae19e2c45a57fa69ad06e6fa3851a WatchSource:0}: Error finding container 0a6b5ebd2fa2dc933e2afa486cbb28ebf25ae19e2c45a57fa69ad06e6fa3851a: Status 404 returned error can't find the container with id 0a6b5ebd2fa2dc933e2afa486cbb28ebf25ae19e2c45a57fa69ad06e6fa3851a Oct 30 18:16:13.601911 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:13.596954 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4b709d-754cc6c5c8-fchc4"] Oct 30 18:16:13.616438 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:13.616389 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-df2cs" podStartSLOduration=2.995608024 podStartE2EDuration="3.616373894s" totalImagesPullingTime="620.76587ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:13.612231695 +0000 UTC m=+1070.702294046" watchObservedRunningTime="2026-10-30 18:16:13.616373894 +0000 UTC m=+1070.706436233" Oct 30 18:16:13.650801 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:13.650723 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-fkxzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:16:13.651072 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:13.651004 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" podUID="14c086ee-c4ba-48d2-a1b9-62cc8b09cc77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 18:16:14.378165 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:14.377343 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-5jbdk" Oct 30 18:16:14.381073 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:14.380302 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-2zwb7" Oct 30 18:16:14.403997 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:14.403964 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 18:16:14.404173 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:14.404017 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" podUID="d9b6ff41-9b4a-43e1-8a09-b99b2014530c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 18:16:14.616261 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:14.616216 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b709d-754cc6c5c8-fchc4" event={"ID":"6db04ea0-e110-4d71-9a90-f37b4c431a3b","Type":"ContainerStarted","Data":"2e82e89fc8cf0d3b905dc3407f3fdd7676fef629d7426c0579d487cb265e4770"} Oct 30 18:16:14.616792 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:14.616276 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b709d-754cc6c5c8-fchc4" event={"ID":"6db04ea0-e110-4d71-9a90-f37b4c431a3b","Type":"ContainerStarted","Data":"0a6b5ebd2fa2dc933e2afa486cbb28ebf25ae19e2c45a57fa69ad06e6fa3851a"} Oct 30 18:16:14.616913 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:14.616892 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b709d-754cc6c5c8-fchc4" Oct 30 18:16:14.670397 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:14.668323 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-fkxzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:16:14.670397 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:14.668385 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" podUID="14c086ee-c4ba-48d2-a1b9-62cc8b09cc77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 18:16:14.676950 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:14.676118 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b709d-754cc6c5c8-fchc4" podStartSLOduration=2.113788006 podStartE2EDuration="2.676097354s" totalImagesPullingTime="562.309348ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:14.670925244 +0000 UTC m=+1071.760987606" watchObservedRunningTime="2026-10-30 18:16:14.676097354 +0000 UTC m=+1071.766159695" Oct 30 18:16:15.372160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:15.369386 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4b709d-5df8f677bc-b4srr" Oct 30 18:16:15.372160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:15.370736 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-4b709d-7d96656586-hxnbn" Oct 30 18:16:15.404457 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:15.404419 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 18:16:15.404602 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:15.404485 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" podUID="d9b6ff41-9b4a-43e1-8a09-b99b2014530c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 18:16:15.662216 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:15.650217 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-fkxzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:16:15.662216 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:15.650286 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" podUID="14c086ee-c4ba-48d2-a1b9-62cc8b09cc77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 18:16:16.404416 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:16.404268 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 18:16:16.404416 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:16.404336 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" podUID="d9b6ff41-9b4a-43e1-8a09-b99b2014530c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 18:16:17.403700 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:17.403662 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 18:16:17.404095 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:17.403735 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" podUID="d9b6ff41-9b4a-43e1-8a09-b99b2014530c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 18:16:17.542224 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:17.539715 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" Oct 30 18:16:17.543378 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:17.543188 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" Oct 30 18:16:17.545592 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:17.545543 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" Oct 30 18:16:17.585033 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:16:17.584995 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:16:18.474732 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:18.474380 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" Oct 30 18:16:18.475265 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:18.475064 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" Oct 30 18:16:18.475966 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:18.475945 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" Oct 30 18:16:19.622325 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:19.622298 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-df2cs" Oct 30 18:16:21.714158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:21.710322 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b709d-754cc6c5c8-fchc4" Oct 30 18:16:23.206161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:23.204274 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4" podUID="cf7d5f54-e091-4032-ab11-af2687e453c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:23.344161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:23.341504 4442 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8" podUID="1e2901e8-4919-4e5c-88c7-39126a7cca56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:23.722166 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:23.708575 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8" podUID="1e2901e8-4919-4e5c-88c7-39126a7cca56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Oct 30 18:16:24.784538 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:24.780382 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8" podUID="1e2901e8-4919-4e5c-88c7-39126a7cca56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:24.794171 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:24.785646 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8" Oct 30 18:16:25.108515 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:25.097962 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:16:25.108515 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:25.098066 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" podUID="e8ac1217-0b8a-4d72-907f-c70814b60a55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:25.296494 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:25.296448 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4" podUID="cf7d5f54-e091-4032-ab11-af2687e453c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:25.791161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:25.787421 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8" podUID="1e2901e8-4919-4e5c-88c7-39126a7cca56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:26.324160 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:16:26.323502 4442 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.258s" Oct 30 18:16:26.699046 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:26.699010 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8" Oct 30 18:16:26.835162 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:26.830277 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:16:26.835162 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:26.830388 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" podUID="8626d789-257d-46c2-8555-8af451ed6434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:39.227995 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:39.227951 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx"] Oct 30 18:16:39.234376 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:39.234342 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" Oct 30 18:16:39.245772 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:39.245749 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6a0e550-b203-433c-8991-387f2179e0de-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx\" (UID: \"f6a0e550-b203-433c-8991-387f2179e0de\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" Oct 30 18:16:39.245938 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:39.245917 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh5zd\" (UniqueName: \"kubernetes.io/projected/f6a0e550-b203-433c-8991-387f2179e0de-kube-api-access-wh5zd\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx\" (UID: \"f6a0e550-b203-433c-8991-387f2179e0de\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" Oct 30 18:16:39.246059 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:39.246045 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6a0e550-b203-433c-8991-387f2179e0de-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx\" (UID: \"f6a0e550-b203-433c-8991-387f2179e0de\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" Oct 30 18:16:39.246116 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:39.246096 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6a0e550-b203-433c-8991-387f2179e0de-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx\" (UID: \"f6a0e550-b203-433c-8991-387f2179e0de\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" Oct 30 18:16:39.246198 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:39.246173 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6a0e550-b203-433c-8991-387f2179e0de-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx\" (UID: \"f6a0e550-b203-433c-8991-387f2179e0de\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" Oct 30 18:16:39.246253 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:39.246197 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6a0e550-b203-433c-8991-387f2179e0de-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx\" (UID: \"f6a0e550-b203-433c-8991-387f2179e0de\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" Oct 30 18:16:39.246414 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:39.246388 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6a0e550-b203-433c-8991-387f2179e0de-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx\" (UID: \"f6a0e550-b203-433c-8991-387f2179e0de\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" Oct 30 18:16:39.246518 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:39.246476 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6a0e550-b203-433c-8991-387f2179e0de-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx\" (UID: \"f6a0e550-b203-433c-8991-387f2179e0de\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" Oct 30 18:16:39.246578 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:39.246517 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6a0e550-b203-433c-8991-387f2179e0de-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx\" (UID: \"f6a0e550-b203-433c-8991-387f2179e0de\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" Oct 30 18:16:39.254398 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:39.254372 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx"] Oct 30 18:16:39.347764 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:39.347727 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh5zd\" (UniqueName: \"kubernetes.io/projected/f6a0e550-b203-433c-8991-387f2179e0de-kube-api-access-wh5zd\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx\" (UID: \"f6a0e550-b203-433c-8991-387f2179e0de\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" Oct 30 18:16:39.347982 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:39.347818 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6a0e550-b203-433c-8991-387f2179e0de-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx\" (UID: \"f6a0e550-b203-433c-8991-387f2179e0de\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" Oct 30 18:16:39.347982 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:39.347835 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6a0e550-b203-433c-8991-387f2179e0de-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx\" (UID: \"f6a0e550-b203-433c-8991-387f2179e0de\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" Oct 30 18:16:39.347982 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:39.347863 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6a0e550-b203-433c-8991-387f2179e0de-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx\" (UID: \"f6a0e550-b203-433c-8991-387f2179e0de\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" Oct 30 18:16:39.347982 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:39.347887 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6a0e550-b203-433c-8991-387f2179e0de-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx\" (UID: \"f6a0e550-b203-433c-8991-387f2179e0de\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" Oct 30 18:16:39.347982 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:39.347970 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6a0e550-b203-433c-8991-387f2179e0de-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx\" (UID: \"f6a0e550-b203-433c-8991-387f2179e0de\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" Oct 30 18:16:39.348313 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:39.348030 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6a0e550-b203-433c-8991-387f2179e0de-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx\" (UID: \"f6a0e550-b203-433c-8991-387f2179e0de\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" Oct 30 18:16:39.348313 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:39.348073 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6a0e550-b203-433c-8991-387f2179e0de-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx\" (UID: \"f6a0e550-b203-433c-8991-387f2179e0de\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" Oct 30 18:16:39.348313 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:39.348181 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6a0e550-b203-433c-8991-387f2179e0de-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx\" (UID: \"f6a0e550-b203-433c-8991-387f2179e0de\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" Oct 30 18:16:39.348475 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:39.348306 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6a0e550-b203-433c-8991-387f2179e0de-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx\" (UID: \"f6a0e550-b203-433c-8991-387f2179e0de\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" Oct 30 18:16:39.348475 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:39.348321 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6a0e550-b203-433c-8991-387f2179e0de-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx\" (UID: \"f6a0e550-b203-433c-8991-387f2179e0de\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" Oct 30 18:16:39.348475 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:39.348423 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6a0e550-b203-433c-8991-387f2179e0de-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx\" (UID: \"f6a0e550-b203-433c-8991-387f2179e0de\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" Oct 30 18:16:39.348633 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:39.348509 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6a0e550-b203-433c-8991-387f2179e0de-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx\" (UID: \"f6a0e550-b203-433c-8991-387f2179e0de\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" Oct 30 18:16:39.348903 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:39.348885 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6a0e550-b203-433c-8991-387f2179e0de-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx\" (UID: \"f6a0e550-b203-433c-8991-387f2179e0de\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" Oct 30 18:16:39.351030 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:39.351003 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6a0e550-b203-433c-8991-387f2179e0de-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx\" (UID: \"f6a0e550-b203-433c-8991-387f2179e0de\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" Oct 30 18:16:39.351160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:39.351011 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6a0e550-b203-433c-8991-387f2179e0de-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx\" (UID: \"f6a0e550-b203-433c-8991-387f2179e0de\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" Oct 30 18:16:39.367069 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:39.367045 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6a0e550-b203-433c-8991-387f2179e0de-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx\" (UID: \"f6a0e550-b203-433c-8991-387f2179e0de\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" Oct 30 18:16:39.367230 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:39.367104 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh5zd\" (UniqueName: \"kubernetes.io/projected/f6a0e550-b203-433c-8991-387f2179e0de-kube-api-access-wh5zd\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx\" (UID: \"f6a0e550-b203-433c-8991-387f2179e0de\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" Oct 30 18:16:39.550059 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:39.549976 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" Oct 30 18:16:39.871787 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:16:39.871737 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6a0e550_b203_433c_8991_387f2179e0de.slice/crio-0cdfd330ac649e8573daf5611ee8c2af451c8e42f68bc780d20da3c7bb1d2165 WatchSource:0}: Error finding container 0cdfd330ac649e8573daf5611ee8c2af451c8e42f68bc780d20da3c7bb1d2165: Status 404 returned error can't find the container with id 0cdfd330ac649e8573daf5611ee8c2af451c8e42f68bc780d20da3c7bb1d2165 Oct 30 18:16:39.871970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:39.871890 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx"] Oct 30 18:16:39.877086 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:39.877055 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:16:39.877209 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:39.877117 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:16:39.877209 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:39.877160 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:16:40.856158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:40.855463 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" event={"ID":"f6a0e550-b203-433c-8991-387f2179e0de","Type":"ContainerStarted","Data":"4bed610050ec6cda902438aae0dcc950d53cc9ebee5302bf95a529cc8f48426f"} Oct 30 18:16:40.856158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:40.855574 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" event={"ID":"f6a0e550-b203-433c-8991-387f2179e0de","Type":"ContainerStarted","Data":"0cdfd330ac649e8573daf5611ee8c2af451c8e42f68bc780d20da3c7bb1d2165"} Oct 30 18:16:40.913513 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:40.913455 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" podStartSLOduration=1.913436515 podStartE2EDuration="1.913436515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:40.909125493 +0000 UTC m=+1097.999187856" watchObservedRunningTime="2026-10-30 18:16:40.913436515 +0000 UTC m=+1098.003498859" Oct 30 18:16:41.555528 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:41.555497 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" Oct 30 18:16:41.558770 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:41.558724 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 18:16:41.558927 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:41.558793 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" podUID="f6a0e550-b203-433c-8991-387f2179e0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 18:16:42.551731 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:42.551693 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 18:16:42.552360 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:42.552328 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" podUID="f6a0e550-b203-433c-8991-387f2179e0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 18:16:42.987539 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:42.987502 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f15587-7cccff6f5-89g5c"] Oct 30 18:16:42.993112 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:42.993086 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f15587-7cccff6f5-89g5c" Oct 30 18:16:43.073169 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:43.056811 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-f15587-dockercfg-jrxzb\"" Oct 30 18:16:43.073169 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:43.056928 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-f15587\"" Oct 30 18:16:43.089560 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:43.085229 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj4pq\" (UniqueName: \"kubernetes.io/projected/05f66220-a64e-4c45-866f-caa9b332e3ec-kube-api-access-lj4pq\") pod \"payload-pre-processing-e2e-api-a-f15587-7cccff6f5-89g5c\" (UID: \"05f66220-a64e-4c45-866f-caa9b332e3ec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f15587-7cccff6f5-89g5c" Oct 30 18:16:43.089560 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:43.085369 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05f66220-a64e-4c45-866f-caa9b332e3ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f15587-7cccff6f5-89g5c\" (UID: \"05f66220-a64e-4c45-866f-caa9b332e3ec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f15587-7cccff6f5-89g5c" Oct 30 18:16:43.309210 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:43.289702 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj4pq\" (UniqueName: \"kubernetes.io/projected/05f66220-a64e-4c45-866f-caa9b332e3ec-kube-api-access-lj4pq\") pod \"payload-pre-processing-e2e-api-a-f15587-7cccff6f5-89g5c\" (UID: \"05f66220-a64e-4c45-866f-caa9b332e3ec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f15587-7cccff6f5-89g5c" Oct 30 18:16:43.309210 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:43.289881 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05f66220-a64e-4c45-866f-caa9b332e3ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f15587-7cccff6f5-89g5c\" (UID: \"05f66220-a64e-4c45-866f-caa9b332e3ec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f15587-7cccff6f5-89g5c" Oct 30 18:16:43.309210 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:43.291094 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05f66220-a64e-4c45-866f-caa9b332e3ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f15587-7cccff6f5-89g5c\" (UID: \"05f66220-a64e-4c45-866f-caa9b332e3ec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f15587-7cccff6f5-89g5c" Oct 30 18:16:43.543173 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:43.543115 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f15587-7cccff6f5-89g5c"] Oct 30 18:16:43.640406 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:43.640364 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj4pq\" (UniqueName: \"kubernetes.io/projected/05f66220-a64e-4c45-866f-caa9b332e3ec-kube-api-access-lj4pq\") pod \"payload-pre-processing-e2e-api-a-f15587-7cccff6f5-89g5c\" (UID: \"05f66220-a64e-4c45-866f-caa9b332e3ec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f15587-7cccff6f5-89g5c" Oct 30 18:16:43.649297 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:43.649264 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 18:16:43.662687 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:43.649789 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" podUID="f6a0e550-b203-433c-8991-387f2179e0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 18:16:43.680002 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:43.672105 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f15587-7cccff6f5-89g5c" Oct 30 18:16:43.788237 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:43.754396 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f15587-674b5c966-d8pd8"] Oct 30 18:16:43.788237 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:43.770339 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f15587-674b5c966-d8pd8" Oct 30 18:16:43.884608 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:43.884570 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg62c\" (UniqueName: \"kubernetes.io/projected/0e1a7e8c-80ee-4549-b261-8cd5cb1ef16e-kube-api-access-fg62c\") pod \"payload-processing-e2e-api-a-f15587-674b5c966-d8pd8\" (UID: \"0e1a7e8c-80ee-4549-b261-8cd5cb1ef16e\") " pod="openshift-ingress/payload-processing-e2e-api-a-f15587-674b5c966-d8pd8" Oct 30 18:16:43.884982 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:43.884953 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e1a7e8c-80ee-4549-b261-8cd5cb1ef16e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f15587-674b5c966-d8pd8\" (UID: \"0e1a7e8c-80ee-4549-b261-8cd5cb1ef16e\") " pod="openshift-ingress/payload-processing-e2e-api-a-f15587-674b5c966-d8pd8" Oct 30 18:16:43.894626 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:43.894559 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f15587-674b5c966-d8pd8"] Oct 30 18:16:43.988623 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:43.986599 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg62c\" (UniqueName: \"kubernetes.io/projected/0e1a7e8c-80ee-4549-b261-8cd5cb1ef16e-kube-api-access-fg62c\") pod \"payload-processing-e2e-api-a-f15587-674b5c966-d8pd8\" (UID: \"0e1a7e8c-80ee-4549-b261-8cd5cb1ef16e\") " pod="openshift-ingress/payload-processing-e2e-api-a-f15587-674b5c966-d8pd8" Oct 30 18:16:43.988623 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:43.986857 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e1a7e8c-80ee-4549-b261-8cd5cb1ef16e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f15587-674b5c966-d8pd8\" (UID: \"0e1a7e8c-80ee-4549-b261-8cd5cb1ef16e\") " pod="openshift-ingress/payload-processing-e2e-api-a-f15587-674b5c966-d8pd8" Oct 30 18:16:43.988623 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:43.987875 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e1a7e8c-80ee-4549-b261-8cd5cb1ef16e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f15587-674b5c966-d8pd8\" (UID: \"0e1a7e8c-80ee-4549-b261-8cd5cb1ef16e\") " pod="openshift-ingress/payload-processing-e2e-api-a-f15587-674b5c966-d8pd8" Oct 30 18:16:44.007292 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:44.007265 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg62c\" (UniqueName: \"kubernetes.io/projected/0e1a7e8c-80ee-4549-b261-8cd5cb1ef16e-kube-api-access-fg62c\") pod \"payload-processing-e2e-api-a-f15587-674b5c966-d8pd8\" (UID: \"0e1a7e8c-80ee-4549-b261-8cd5cb1ef16e\") " pod="openshift-ingress/payload-processing-e2e-api-a-f15587-674b5c966-d8pd8" Oct 30 18:16:44.122773 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:44.122735 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f15587-674b5c966-d8pd8" Oct 30 18:16:44.198668 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:44.198641 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f15587-7cccff6f5-89g5c"] Oct 30 18:16:44.200229 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:16:44.200203 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05f66220_a64e_4c45_866f_caa9b332e3ec.slice/crio-39ad3a4463dc766c9631d26624276e0639d70f05ef706c421946345a97781cd0 WatchSource:0}: Error finding container 39ad3a4463dc766c9631d26624276e0639d70f05ef706c421946345a97781cd0: Status 404 returned error can't find the container with id 39ad3a4463dc766c9631d26624276e0639d70f05ef706c421946345a97781cd0 Oct 30 18:16:44.569156 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:44.568816 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 18:16:44.569156 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:44.568880 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" podUID="f6a0e550-b203-433c-8991-387f2179e0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 18:16:44.596537 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:44.591649 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f15587-674b5c966-d8pd8"] Oct 30 18:16:44.908643 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:44.908604 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f15587-7cccff6f5-89g5c" event={"ID":"05f66220-a64e-4c45-866f-caa9b332e3ec","Type":"ContainerStarted","Data":"0b9239a9d00486822b985281a34bb607f6667587fc251ed8beb02bf15800bbce"} Oct 30 18:16:44.909083 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:44.908666 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f15587-7cccff6f5-89g5c" event={"ID":"05f66220-a64e-4c45-866f-caa9b332e3ec","Type":"ContainerStarted","Data":"39ad3a4463dc766c9631d26624276e0639d70f05ef706c421946345a97781cd0"} Oct 30 18:16:44.909083 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:44.908730 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f15587-7cccff6f5-89g5c" Oct 30 18:16:44.910525 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:44.910499 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f15587-674b5c966-d8pd8" event={"ID":"0e1a7e8c-80ee-4549-b261-8cd5cb1ef16e","Type":"ContainerStarted","Data":"e4b48df9554f4a6e40c8853844b41e2979b1db08063f3196db5d097854f9f699"} Oct 30 18:16:44.910648 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:44.910547 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f15587-674b5c966-d8pd8" event={"ID":"0e1a7e8c-80ee-4549-b261-8cd5cb1ef16e","Type":"ContainerStarted","Data":"72c817689444b532df2c0e1dc3bccbc4e8e84f9d822d263ea964af9a4c64f95c"} Oct 30 18:16:44.911196 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:44.911170 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-f15587-674b5c966-d8pd8" Oct 30 18:16:44.956171 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:44.955576 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f15587-7cccff6f5-89g5c" podStartSLOduration=2.955556849 podStartE2EDuration="2.955556849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:44.952632719 +0000 UTC m=+1102.042695059" watchObservedRunningTime="2026-10-30 18:16:44.955556849 +0000 UTC m=+1102.045619189" Oct 30 18:16:44.996349 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:44.995786 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-f15587-674b5c966-d8pd8" podStartSLOduration=1.9957715120000001 podStartE2EDuration="1.995771512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:44.992961813 +0000 UTC m=+1102.083024153" watchObservedRunningTime="2026-10-30 18:16:44.995771512 +0000 UTC m=+1102.085833851" Oct 30 18:16:45.553167 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:45.550695 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 18:16:45.553167 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:45.550768 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" podUID="f6a0e550-b203-433c-8991-387f2179e0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 18:16:46.554770 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:46.554733 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 18:16:46.555492 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:46.555435 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" podUID="f6a0e550-b203-433c-8991-387f2179e0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 18:16:47.559158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:47.558290 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 18:16:47.559158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:47.558352 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" podUID="f6a0e550-b203-433c-8991-387f2179e0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 18:16:47.597880 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:16:47.597831 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:16:48.550656 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:48.550625 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 18:16:48.550996 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:48.550966 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" podUID="f6a0e550-b203-433c-8991-387f2179e0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 18:16:49.561393 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:49.561359 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" Oct 30 18:16:49.561850 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:49.561790 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" Oct 30 18:16:49.562792 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:49.562774 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" Oct 30 18:16:52.914061 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:52.914024 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b33a49-6f47646bf9-lpsfd"] Oct 30 18:16:52.919503 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:52.919480 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b33a49-6f47646bf9-lpsfd" Oct 30 18:16:52.938416 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:52.938391 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b33a49-6f47646bf9-lpsfd"] Oct 30 18:16:52.943440 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:52.943417 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-b33a49-dockercfg-h4z6x\"" Oct 30 18:16:52.943598 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:52.943457 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b33a49\"" Oct 30 18:16:53.236058 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:53.234218 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkgv6\" (UniqueName: \"kubernetes.io/projected/092c23c3-af50-48da-965a-8890dbe71a58-kube-api-access-lkgv6\") pod \"payload-pre-processing-e2e-api-b-b33a49-6f47646bf9-lpsfd\" (UID: \"092c23c3-af50-48da-965a-8890dbe71a58\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b33a49-6f47646bf9-lpsfd" Oct 30 18:16:53.236058 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:53.234311 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/092c23c3-af50-48da-965a-8890dbe71a58-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b33a49-6f47646bf9-lpsfd\" (UID: \"092c23c3-af50-48da-965a-8890dbe71a58\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b33a49-6f47646bf9-lpsfd" Oct 30 18:16:53.348684 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:53.341846 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkgv6\" (UniqueName: \"kubernetes.io/projected/092c23c3-af50-48da-965a-8890dbe71a58-kube-api-access-lkgv6\") pod \"payload-pre-processing-e2e-api-b-b33a49-6f47646bf9-lpsfd\" (UID: \"092c23c3-af50-48da-965a-8890dbe71a58\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b33a49-6f47646bf9-lpsfd" Oct 30 18:16:53.349165 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:53.349109 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/092c23c3-af50-48da-965a-8890dbe71a58-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b33a49-6f47646bf9-lpsfd\" (UID: \"092c23c3-af50-48da-965a-8890dbe71a58\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b33a49-6f47646bf9-lpsfd" Oct 30 18:16:53.350564 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:53.350537 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/092c23c3-af50-48da-965a-8890dbe71a58-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b33a49-6f47646bf9-lpsfd\" (UID: \"092c23c3-af50-48da-965a-8890dbe71a58\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b33a49-6f47646bf9-lpsfd" Oct 30 18:16:53.389304 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:53.389274 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b33a49-6d58bbc64d-49fnt"] Oct 30 18:16:53.393730 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:53.393695 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkgv6\" (UniqueName: \"kubernetes.io/projected/092c23c3-af50-48da-965a-8890dbe71a58-kube-api-access-lkgv6\") pod \"payload-pre-processing-e2e-api-b-b33a49-6f47646bf9-lpsfd\" (UID: \"092c23c3-af50-48da-965a-8890dbe71a58\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b33a49-6f47646bf9-lpsfd" Oct 30 18:16:53.395356 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:53.395336 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b33a49-6d58bbc64d-49fnt"] Oct 30 18:16:53.395540 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:53.395406 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b33a49-6d58bbc64d-49fnt" Oct 30 18:16:53.451767 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:53.451689 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpv2z\" (UniqueName: \"kubernetes.io/projected/7f87d5c3-1940-499d-b425-0a7ea416d250-kube-api-access-tpv2z\") pod \"payload-processing-e2e-api-b-b33a49-6d58bbc64d-49fnt\" (UID: \"7f87d5c3-1940-499d-b425-0a7ea416d250\") " pod="openshift-ingress/payload-processing-e2e-api-b-b33a49-6d58bbc64d-49fnt" Oct 30 18:16:53.451972 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:53.451828 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f87d5c3-1940-499d-b425-0a7ea416d250-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b33a49-6d58bbc64d-49fnt\" (UID: \"7f87d5c3-1940-499d-b425-0a7ea416d250\") " pod="openshift-ingress/payload-processing-e2e-api-b-b33a49-6d58bbc64d-49fnt" Oct 30 18:16:53.536996 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:53.536899 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b33a49-6f47646bf9-lpsfd" Oct 30 18:16:53.553425 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:53.553390 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpv2z\" (UniqueName: \"kubernetes.io/projected/7f87d5c3-1940-499d-b425-0a7ea416d250-kube-api-access-tpv2z\") pod \"payload-processing-e2e-api-b-b33a49-6d58bbc64d-49fnt\" (UID: \"7f87d5c3-1940-499d-b425-0a7ea416d250\") " pod="openshift-ingress/payload-processing-e2e-api-b-b33a49-6d58bbc64d-49fnt" Oct 30 18:16:53.553578 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:53.553501 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f87d5c3-1940-499d-b425-0a7ea416d250-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b33a49-6d58bbc64d-49fnt\" (UID: \"7f87d5c3-1940-499d-b425-0a7ea416d250\") " pod="openshift-ingress/payload-processing-e2e-api-b-b33a49-6d58bbc64d-49fnt" Oct 30 18:16:53.554509 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:53.554481 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f87d5c3-1940-499d-b425-0a7ea416d250-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b33a49-6d58bbc64d-49fnt\" (UID: \"7f87d5c3-1940-499d-b425-0a7ea416d250\") " pod="openshift-ingress/payload-processing-e2e-api-b-b33a49-6d58bbc64d-49fnt" Oct 30 18:16:53.574181 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:53.574149 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpv2z\" (UniqueName: \"kubernetes.io/projected/7f87d5c3-1940-499d-b425-0a7ea416d250-kube-api-access-tpv2z\") pod \"payload-processing-e2e-api-b-b33a49-6d58bbc64d-49fnt\" (UID: \"7f87d5c3-1940-499d-b425-0a7ea416d250\") " pod="openshift-ingress/payload-processing-e2e-api-b-b33a49-6d58bbc64d-49fnt" Oct 30 18:16:53.722660 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:53.722611 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b33a49-6d58bbc64d-49fnt" Oct 30 18:16:53.881214 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:53.878931 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b33a49-6f47646bf9-lpsfd"] Oct 30 18:16:53.881214 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:16:53.880849 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod092c23c3_af50_48da_965a_8890dbe71a58.slice/crio-2333cd74c834ee924e6f44182ead073c774cd5b10d0b21e55dc3162cdb7c82c5 WatchSource:0}: Error finding container 2333cd74c834ee924e6f44182ead073c774cd5b10d0b21e55dc3162cdb7c82c5: Status 404 returned error can't find the container with id 2333cd74c834ee924e6f44182ead073c774cd5b10d0b21e55dc3162cdb7c82c5 Oct 30 18:16:54.079225 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:16:54.079197 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f87d5c3_1940_499d_b425_0a7ea416d250.slice/crio-bdbac0a378351db034f27efedfeffd66e3be3a34f8ec60b325487b0a7b3e7076 WatchSource:0}: Error finding container bdbac0a378351db034f27efedfeffd66e3be3a34f8ec60b325487b0a7b3e7076: Status 404 returned error can't find the container with id bdbac0a378351db034f27efedfeffd66e3be3a34f8ec60b325487b0a7b3e7076 Oct 30 18:16:54.079569 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:54.079514 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b33a49-6d58bbc64d-49fnt"] Oct 30 18:16:54.414157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:54.413946 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b33a49-6d58bbc64d-49fnt" event={"ID":"7f87d5c3-1940-499d-b425-0a7ea416d250","Type":"ContainerStarted","Data":"9885972e104793d4a4807b934cb21ef2ae4b56985dd100df1f60e0c8f5920e2f"} Oct 30 18:16:54.414157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:54.414031 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b33a49-6d58bbc64d-49fnt" event={"ID":"7f87d5c3-1940-499d-b425-0a7ea416d250","Type":"ContainerStarted","Data":"bdbac0a378351db034f27efedfeffd66e3be3a34f8ec60b325487b0a7b3e7076"} Oct 30 18:16:54.417295 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:54.415694 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-b33a49-6d58bbc64d-49fnt" Oct 30 18:16:54.420548 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:54.418628 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b33a49-6f47646bf9-lpsfd" event={"ID":"092c23c3-af50-48da-965a-8890dbe71a58","Type":"ContainerStarted","Data":"3725fcb89f4742c0def9daa8dd7257f7e77bd67e9672e1af68e658e070eeb3f2"} Oct 30 18:16:54.420548 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:54.418706 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b33a49-6f47646bf9-lpsfd" event={"ID":"092c23c3-af50-48da-965a-8890dbe71a58","Type":"ContainerStarted","Data":"2333cd74c834ee924e6f44182ead073c774cd5b10d0b21e55dc3162cdb7c82c5"} Oct 30 18:16:54.421931 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:54.421723 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b33a49-6f47646bf9-lpsfd" Oct 30 18:16:54.469181 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:54.465232 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-b33a49-6d58bbc64d-49fnt" podStartSLOduration=2.465215733 podStartE2EDuration="2.465215733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:54.45951254 +0000 UTC m=+1111.549574879" watchObservedRunningTime="2026-10-30 18:16:54.465215733 +0000 UTC m=+1111.555278071" Oct 30 18:16:54.498717 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:54.498664 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b33a49-6f47646bf9-lpsfd" podStartSLOduration=2.49865044 podStartE2EDuration="2.49865044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:54.497327969 +0000 UTC m=+1111.587390309" watchObservedRunningTime="2026-10-30 18:16:54.49865044 +0000 UTC m=+1111.588712778" Oct 30 18:16:55.723339 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:55.723303 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f15587-55b68687b-xfwqd"] Oct 30 18:16:55.727951 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:55.727926 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f15587-55b68687b-xfwqd" Oct 30 18:16:55.748812 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:55.747014 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f15587-55b68687b-xfwqd"] Oct 30 18:16:55.751501 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:55.751478 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f15587-serving-cert\"" Oct 30 18:16:55.810448 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:55.808471 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/907f83bb-f7b8-4074-9abc-e7ce23e1b665-metrics-certs\") pod \"maas-api-e2e-api-a-f15587-55b68687b-xfwqd\" (UID: \"907f83bb-f7b8-4074-9abc-e7ce23e1b665\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f15587-55b68687b-xfwqd" Oct 30 18:16:55.810448 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:55.808535 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/907f83bb-f7b8-4074-9abc-e7ce23e1b665-openshift-service-ca\") pod \"maas-api-e2e-api-a-f15587-55b68687b-xfwqd\" (UID: \"907f83bb-f7b8-4074-9abc-e7ce23e1b665\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f15587-55b68687b-xfwqd" Oct 30 18:16:55.810448 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:55.808639 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/907f83bb-f7b8-4074-9abc-e7ce23e1b665-maas-api-tls\") pod \"maas-api-e2e-api-a-f15587-55b68687b-xfwqd\" (UID: \"907f83bb-f7b8-4074-9abc-e7ce23e1b665\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f15587-55b68687b-xfwqd" Oct 30 18:16:55.810448 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:55.808699 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjn5m\" (UniqueName: \"kubernetes.io/projected/907f83bb-f7b8-4074-9abc-e7ce23e1b665-kube-api-access-mjn5m\") pod \"maas-api-e2e-api-a-f15587-55b68687b-xfwqd\" (UID: \"907f83bb-f7b8-4074-9abc-e7ce23e1b665\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f15587-55b68687b-xfwqd" Oct 30 18:16:55.910166 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:55.910093 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/907f83bb-f7b8-4074-9abc-e7ce23e1b665-metrics-certs\") pod \"maas-api-e2e-api-a-f15587-55b68687b-xfwqd\" (UID: \"907f83bb-f7b8-4074-9abc-e7ce23e1b665\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f15587-55b68687b-xfwqd" Oct 30 18:16:55.910166 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:55.910164 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/907f83bb-f7b8-4074-9abc-e7ce23e1b665-openshift-service-ca\") pod \"maas-api-e2e-api-a-f15587-55b68687b-xfwqd\" (UID: \"907f83bb-f7b8-4074-9abc-e7ce23e1b665\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f15587-55b68687b-xfwqd" Oct 30 18:16:55.910415 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:55.910261 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/907f83bb-f7b8-4074-9abc-e7ce23e1b665-maas-api-tls\") pod \"maas-api-e2e-api-a-f15587-55b68687b-xfwqd\" (UID: \"907f83bb-f7b8-4074-9abc-e7ce23e1b665\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f15587-55b68687b-xfwqd" Oct 30 18:16:55.910415 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:55.910317 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjn5m\" (UniqueName: \"kubernetes.io/projected/907f83bb-f7b8-4074-9abc-e7ce23e1b665-kube-api-access-mjn5m\") pod \"maas-api-e2e-api-a-f15587-55b68687b-xfwqd\" (UID: \"907f83bb-f7b8-4074-9abc-e7ce23e1b665\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f15587-55b68687b-xfwqd" Oct 30 18:16:55.910876 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:55.910849 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/907f83bb-f7b8-4074-9abc-e7ce23e1b665-openshift-service-ca\") pod \"maas-api-e2e-api-a-f15587-55b68687b-xfwqd\" (UID: \"907f83bb-f7b8-4074-9abc-e7ce23e1b665\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f15587-55b68687b-xfwqd" Oct 30 18:16:55.912873 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:55.912852 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/907f83bb-f7b8-4074-9abc-e7ce23e1b665-maas-api-tls\") pod \"maas-api-e2e-api-a-f15587-55b68687b-xfwqd\" (UID: \"907f83bb-f7b8-4074-9abc-e7ce23e1b665\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f15587-55b68687b-xfwqd" Oct 30 18:16:55.913190 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:55.913127 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/907f83bb-f7b8-4074-9abc-e7ce23e1b665-metrics-certs\") pod \"maas-api-e2e-api-a-f15587-55b68687b-xfwqd\" (UID: \"907f83bb-f7b8-4074-9abc-e7ce23e1b665\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f15587-55b68687b-xfwqd" Oct 30 18:16:55.922002 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:55.921975 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f15587-7cccff6f5-89g5c" Oct 30 18:16:55.931829 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:55.931806 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjn5m\" (UniqueName: \"kubernetes.io/projected/907f83bb-f7b8-4074-9abc-e7ce23e1b665-kube-api-access-mjn5m\") pod \"maas-api-e2e-api-a-f15587-55b68687b-xfwqd\" (UID: \"907f83bb-f7b8-4074-9abc-e7ce23e1b665\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f15587-55b68687b-xfwqd" Oct 30 18:16:56.054519 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:56.054427 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f15587-55b68687b-xfwqd" Oct 30 18:16:56.378640 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:56.378299 4442 patch_prober.go:29] interesting pod/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:16:56.378640 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:56.378368 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" podUID="059512a2-ce79-4426-bd6d-b11fede6864f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:56.510981 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:56.510952 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f15587-55b68687b-xfwqd"] Oct 30 18:16:56.513953 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:16:56.513919 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod907f83bb_f7b8_4074_9abc_e7ce23e1b665.slice/crio-e1636e53397276b5898d4be45683223d3e35b32f04eefbb9d6325af5572ef0c1 WatchSource:0}: Error finding container e1636e53397276b5898d4be45683223d3e35b32f04eefbb9d6325af5572ef0c1: Status 404 returned error can't find the container with id e1636e53397276b5898d4be45683223d3e35b32f04eefbb9d6325af5572ef0c1 Oct 30 18:16:57.020416 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:57.015508 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-f15587-674b5c966-d8pd8" Oct 30 18:16:57.445486 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:57.445438 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f15587-55b68687b-xfwqd" event={"ID":"907f83bb-f7b8-4074-9abc-e7ce23e1b665","Type":"ContainerStarted","Data":"a2eaa882f313b5a1ebc21b041d4af990ac83619e7411c7931ec7d84073174d95"} Oct 30 18:16:57.445723 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:57.445543 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f15587-55b68687b-xfwqd" event={"ID":"907f83bb-f7b8-4074-9abc-e7ce23e1b665","Type":"ContainerStarted","Data":"e1636e53397276b5898d4be45683223d3e35b32f04eefbb9d6325af5572ef0c1"} Oct 30 18:16:57.445723 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:57.445572 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f15587-55b68687b-xfwqd" Oct 30 18:16:57.447485 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:57.447459 4442 generic.go:356] "Generic (PLEG): container finished" podID="059512a2-ce79-4426-bd6d-b11fede6864f" containerID="7fd2dc80a09a09123b64a0e7047b73ebd91c5533354165648e6f964709503e5e" exitCode=0 Oct 30 18:16:57.447642 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:57.447536 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" event={"ID":"059512a2-ce79-4426-bd6d-b11fede6864f","Type":"ContainerDied","Data":"7fd2dc80a09a09123b64a0e7047b73ebd91c5533354165648e6f964709503e5e"} Oct 30 18:16:57.447929 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:57.447914 4442 scope.go:122] "RemoveContainer" containerID="7fd2dc80a09a09123b64a0e7047b73ebd91c5533354165648e6f964709503e5e" Oct 30 18:16:57.449396 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:57.449367 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:16:57.449510 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:57.449432 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:16:57.449510 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:57.449464 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:16:57.488415 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:57.488368 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f15587-55b68687b-xfwqd" podStartSLOduration=2.165906276 podStartE2EDuration="2.48835328s" totalImagesPullingTime="322.447004ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:57.48449389 +0000 UTC m=+1114.574556228" watchObservedRunningTime="2026-10-30 18:16:57.48835328 +0000 UTC m=+1114.578415620" Oct 30 18:16:58.457771 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:58.457728 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" event={"ID":"059512a2-ce79-4426-bd6d-b11fede6864f","Type":"ContainerStarted","Data":"d6b06b6275d295c62b3ca4b3b9cf9fd50ffe482c8a49f60856911ebbfb870515"} Oct 30 18:16:58.548665 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:58.548539 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" Oct 30 18:16:58.548855 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:58.548812 4442 patch_prober.go:29] interesting pod/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 18:16:58.548930 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:58.548892 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" podUID="059512a2-ce79-4426-bd6d-b11fede6864f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 18:16:59.536935 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:59.535973 4442 patch_prober.go:29] interesting pod/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 18:16:59.536935 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:16:59.536057 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" podUID="059512a2-ce79-4426-bd6d-b11fede6864f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 18:17:00.539158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:00.536553 4442 patch_prober.go:29] interesting pod/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 18:17:00.539158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:00.536615 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" podUID="059512a2-ce79-4426-bd6d-b11fede6864f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 18:17:01.544161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:01.540751 4442 patch_prober.go:29] interesting pod/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 18:17:01.544161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:01.540819 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" podUID="059512a2-ce79-4426-bd6d-b11fede6864f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 18:17:02.542339 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:02.542294 4442 patch_prober.go:29] interesting pod/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 18:17:02.542641 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:02.542609 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" podUID="059512a2-ce79-4426-bd6d-b11fede6864f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 18:17:03.463756 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:03.463726 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f15587-55b68687b-xfwqd" Oct 30 18:17:03.534355 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:03.533863 4442 patch_prober.go:29] interesting pod/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 18:17:03.534355 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:03.533918 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" podUID="059512a2-ce79-4426-bd6d-b11fede6864f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 18:17:04.538898 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:04.538820 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" Oct 30 18:17:04.539375 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:04.539356 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" Oct 30 18:17:04.540124 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:04.540103 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" Oct 30 18:17:06.042882 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:06.042851 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b33a49-6b5fd95565-fvshv"] Oct 30 18:17:06.049019 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:06.048988 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b33a49-6b5fd95565-fvshv" Oct 30 18:17:06.066032 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:06.065996 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b33a49-6b5fd95565-fvshv"] Oct 30 18:17:06.071363 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:06.071332 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-b33a49-serving-cert\"" Oct 30 18:17:06.125918 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:06.125887 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlmfm\" (UniqueName: \"kubernetes.io/projected/999b71c3-39d4-4b0a-8a38-3495d0f11180-kube-api-access-nlmfm\") pod \"maas-api-e2e-api-b-b33a49-6b5fd95565-fvshv\" (UID: \"999b71c3-39d4-4b0a-8a38-3495d0f11180\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b33a49-6b5fd95565-fvshv" Oct 30 18:17:06.126180 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:06.126030 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/999b71c3-39d4-4b0a-8a38-3495d0f11180-maas-api-tls\") pod \"maas-api-e2e-api-b-b33a49-6b5fd95565-fvshv\" (UID: \"999b71c3-39d4-4b0a-8a38-3495d0f11180\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b33a49-6b5fd95565-fvshv" Oct 30 18:17:06.126180 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:06.126054 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/999b71c3-39d4-4b0a-8a38-3495d0f11180-metrics-certs\") pod \"maas-api-e2e-api-b-b33a49-6b5fd95565-fvshv\" (UID: \"999b71c3-39d4-4b0a-8a38-3495d0f11180\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b33a49-6b5fd95565-fvshv" Oct 30 18:17:06.126312 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:06.126276 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/999b71c3-39d4-4b0a-8a38-3495d0f11180-openshift-service-ca\") pod \"maas-api-e2e-api-b-b33a49-6b5fd95565-fvshv\" (UID: \"999b71c3-39d4-4b0a-8a38-3495d0f11180\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b33a49-6b5fd95565-fvshv" Oct 30 18:17:06.241693 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:06.234926 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/999b71c3-39d4-4b0a-8a38-3495d0f11180-openshift-service-ca\") pod \"maas-api-e2e-api-b-b33a49-6b5fd95565-fvshv\" (UID: \"999b71c3-39d4-4b0a-8a38-3495d0f11180\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b33a49-6b5fd95565-fvshv" Oct 30 18:17:06.248225 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:06.248177 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlmfm\" (UniqueName: \"kubernetes.io/projected/999b71c3-39d4-4b0a-8a38-3495d0f11180-kube-api-access-nlmfm\") pod \"maas-api-e2e-api-b-b33a49-6b5fd95565-fvshv\" (UID: \"999b71c3-39d4-4b0a-8a38-3495d0f11180\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b33a49-6b5fd95565-fvshv" Oct 30 18:17:06.266424 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:06.266386 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/999b71c3-39d4-4b0a-8a38-3495d0f11180-maas-api-tls\") pod \"maas-api-e2e-api-b-b33a49-6b5fd95565-fvshv\" (UID: \"999b71c3-39d4-4b0a-8a38-3495d0f11180\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b33a49-6b5fd95565-fvshv" Oct 30 18:17:06.274264 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:06.274228 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/999b71c3-39d4-4b0a-8a38-3495d0f11180-metrics-certs\") pod \"maas-api-e2e-api-b-b33a49-6b5fd95565-fvshv\" (UID: \"999b71c3-39d4-4b0a-8a38-3495d0f11180\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b33a49-6b5fd95565-fvshv" Oct 30 18:17:06.284605 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:06.284569 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/999b71c3-39d4-4b0a-8a38-3495d0f11180-maas-api-tls\") pod \"maas-api-e2e-api-b-b33a49-6b5fd95565-fvshv\" (UID: \"999b71c3-39d4-4b0a-8a38-3495d0f11180\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b33a49-6b5fd95565-fvshv" Oct 30 18:17:06.285019 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:06.242805 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/999b71c3-39d4-4b0a-8a38-3495d0f11180-openshift-service-ca\") pod \"maas-api-e2e-api-b-b33a49-6b5fd95565-fvshv\" (UID: \"999b71c3-39d4-4b0a-8a38-3495d0f11180\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b33a49-6b5fd95565-fvshv" Oct 30 18:17:06.318125 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:06.310441 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlmfm\" (UniqueName: \"kubernetes.io/projected/999b71c3-39d4-4b0a-8a38-3495d0f11180-kube-api-access-nlmfm\") pod \"maas-api-e2e-api-b-b33a49-6b5fd95565-fvshv\" (UID: \"999b71c3-39d4-4b0a-8a38-3495d0f11180\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b33a49-6b5fd95565-fvshv" Oct 30 18:17:06.318125 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:06.311857 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/999b71c3-39d4-4b0a-8a38-3495d0f11180-metrics-certs\") pod \"maas-api-e2e-api-b-b33a49-6b5fd95565-fvshv\" (UID: \"999b71c3-39d4-4b0a-8a38-3495d0f11180\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b33a49-6b5fd95565-fvshv" Oct 30 18:17:06.383166 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:06.376836 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b33a49-6b5fd95565-fvshv" Oct 30 18:17:06.440410 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:06.440385 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-b33a49-6d58bbc64d-49fnt" Oct 30 18:17:06.440599 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:06.440513 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b33a49-6f47646bf9-lpsfd" Oct 30 18:17:06.725903 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:06.725871 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b33a49-6b5fd95565-fvshv"] Oct 30 18:17:06.725903 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:17:06.725887 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod999b71c3_39d4_4b0a_8a38_3495d0f11180.slice/crio-875b848998b87e0103655da877fdf82b10e860a345a189bc1870f420be84b28b WatchSource:0}: Error finding container 875b848998b87e0103655da877fdf82b10e860a345a189bc1870f420be84b28b: Status 404 returned error can't find the container with id 875b848998b87e0103655da877fdf82b10e860a345a189bc1870f420be84b28b Oct 30 18:17:07.543991 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:07.543189 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b33a49-6b5fd95565-fvshv" event={"ID":"999b71c3-39d4-4b0a-8a38-3495d0f11180","Type":"ContainerStarted","Data":"43d3be194f1b20eae2899669c68076ef792aecb0914162c65cafb14d662781bf"} Oct 30 18:17:07.543991 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:07.543252 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b33a49-6b5fd95565-fvshv" event={"ID":"999b71c3-39d4-4b0a-8a38-3495d0f11180","Type":"ContainerStarted","Data":"875b848998b87e0103655da877fdf82b10e860a345a189bc1870f420be84b28b"} Oct 30 18:17:07.543991 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:07.543283 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b33a49-6b5fd95565-fvshv" Oct 30 18:17:07.586649 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:07.586598 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b33a49-6b5fd95565-fvshv" podStartSLOduration=1.251950632 podStartE2EDuration="1.586583369s" totalImagesPullingTime="334.632737ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:07.581903954 +0000 UTC m=+1124.671966294" watchObservedRunningTime="2026-10-30 18:17:07.586583369 +0000 UTC m=+1124.676645707" Oct 30 18:17:13.560269 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:13.560241 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b33a49-6b5fd95565-fvshv" Oct 30 18:17:17.564650 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:17:17.564604 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:17:23.416238 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:23.416202 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8dc43c-5bc899bbc9-m89jr"] Oct 30 18:17:23.422200 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:23.422179 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8dc43c-5bc899bbc9-m89jr" Oct 30 18:17:23.441450 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:23.441416 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8dc43c-5bc899bbc9-m89jr"] Oct 30 18:17:23.445224 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:23.445196 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-8dc43c\"" Oct 30 18:17:23.491990 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:23.476896 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6w6b\" (UniqueName: \"kubernetes.io/projected/5b6343c8-e782-48b4-87d3-ee8637dc8f9e-kube-api-access-d6w6b\") pod \"payload-pre-processing-e2e-api-b-8dc43c-5bc899bbc9-m89jr\" (UID: \"5b6343c8-e782-48b4-87d3-ee8637dc8f9e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8dc43c-5bc899bbc9-m89jr" Oct 30 18:17:23.491990 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:23.477101 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b6343c8-e782-48b4-87d3-ee8637dc8f9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8dc43c-5bc899bbc9-m89jr\" (UID: \"5b6343c8-e782-48b4-87d3-ee8637dc8f9e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8dc43c-5bc899bbc9-m89jr" Oct 30 18:17:23.528513 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:23.528480 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-8dc43c-dockercfg-cmb4d\"" Oct 30 18:17:23.709312 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:23.647665 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b6343c8-e782-48b4-87d3-ee8637dc8f9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8dc43c-5bc899bbc9-m89jr\" (UID: \"5b6343c8-e782-48b4-87d3-ee8637dc8f9e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8dc43c-5bc899bbc9-m89jr" Oct 30 18:17:23.742158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:23.713553 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6w6b\" (UniqueName: \"kubernetes.io/projected/5b6343c8-e782-48b4-87d3-ee8637dc8f9e-kube-api-access-d6w6b\") pod \"payload-pre-processing-e2e-api-b-8dc43c-5bc899bbc9-m89jr\" (UID: \"5b6343c8-e782-48b4-87d3-ee8637dc8f9e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8dc43c-5bc899bbc9-m89jr" Oct 30 18:17:23.742666 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:23.696278 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b6343c8-e782-48b4-87d3-ee8637dc8f9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8dc43c-5bc899bbc9-m89jr\" (UID: \"5b6343c8-e782-48b4-87d3-ee8637dc8f9e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8dc43c-5bc899bbc9-m89jr" Oct 30 18:17:23.786517 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:23.771437 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6w6b\" (UniqueName: \"kubernetes.io/projected/5b6343c8-e782-48b4-87d3-ee8637dc8f9e-kube-api-access-d6w6b\") pod \"payload-pre-processing-e2e-api-b-8dc43c-5bc899bbc9-m89jr\" (UID: \"5b6343c8-e782-48b4-87d3-ee8637dc8f9e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8dc43c-5bc899bbc9-m89jr" Oct 30 18:17:24.043285 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:24.043190 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8dc43c-5bc899bbc9-m89jr" Oct 30 18:17:24.415935 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:24.415897 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8dc43c-5bc899bbc9-m89jr"] Oct 30 18:17:24.784567 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:24.784478 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8dc43c-5bc899bbc9-m89jr" event={"ID":"5b6343c8-e782-48b4-87d3-ee8637dc8f9e","Type":"ContainerStarted","Data":"5ad36ced1c830af6a745f653337c9513093cb8a9317dca5cfc82bd9cda368fdc"} Oct 30 18:17:24.786449 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:24.786423 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8dc43c-5bc899bbc9-m89jr" Oct 30 18:17:24.787362 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:24.787338 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8dc43c-5bc899bbc9-m89jr" event={"ID":"5b6343c8-e782-48b4-87d3-ee8637dc8f9e","Type":"ContainerStarted","Data":"4006f79a0467e3723d1f411fe40d25f570c9a1046d8a4756f5ed6ace145c6ab4"} Oct 30 18:17:24.855170 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:24.850990 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8dc43c-5bc899bbc9-m89jr" podStartSLOduration=1.850965386 podStartE2EDuration="1.850965386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:24.827714963 +0000 UTC m=+1141.917777303" watchObservedRunningTime="2026-10-30 18:17:24.850965386 +0000 UTC m=+1141.941027728" Oct 30 18:17:36.842608 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:36.842575 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8dc43c-5bc899bbc9-m89jr" Oct 30 18:17:38.142096 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.142057 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8dc43c-884559b4-hgppv"] Oct 30 18:17:38.148248 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.148224 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8dc43c-884559b4-hgppv" Oct 30 18:17:38.168697 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.168661 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8dc43c-884559b4-hgppv"] Oct 30 18:17:38.174492 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.174467 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-8dc43c-serving-cert\"" Oct 30 18:17:38.180336 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.180308 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t"] Oct 30 18:17:38.185619 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.185591 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" Oct 30 18:17:38.215894 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.215859 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t"] Oct 30 18:17:38.266611 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.266575 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-0c6613ea-openshift-default-dockercfg-twcmm\"" Oct 30 18:17:38.316921 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.316888 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44gjh\" (UniqueName: \"kubernetes.io/projected/6db69e2c-9e56-4b22-b8e4-3d346b92c062-kube-api-access-44gjh\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-f257t\" (UID: \"6db69e2c-9e56-4b22-b8e4-3d346b92c062\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" Oct 30 18:17:38.317114 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.316929 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drxdw\" (UniqueName: \"kubernetes.io/projected/d859a2eb-5149-4c88-a99b-85e08c7aaf2b-kube-api-access-drxdw\") pod \"maas-api-e2e-api-b-8dc43c-884559b4-hgppv\" (UID: \"d859a2eb-5149-4c88-a99b-85e08c7aaf2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8dc43c-884559b4-hgppv" Oct 30 18:17:38.317114 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.317014 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db69e2c-9e56-4b22-b8e4-3d346b92c062-workload-socket\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-f257t\" (UID: \"6db69e2c-9e56-4b22-b8e4-3d346b92c062\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" Oct 30 18:17:38.317114 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.317062 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db69e2c-9e56-4b22-b8e4-3d346b92c062-istiod-ca-cert\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-f257t\" (UID: \"6db69e2c-9e56-4b22-b8e4-3d346b92c062\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" Oct 30 18:17:38.317280 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.317164 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db69e2c-9e56-4b22-b8e4-3d346b92c062-istio-data\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-f257t\" (UID: \"6db69e2c-9e56-4b22-b8e4-3d346b92c062\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" Oct 30 18:17:38.317280 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.317202 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6db69e2c-9e56-4b22-b8e4-3d346b92c062-istio-token\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-f257t\" (UID: \"6db69e2c-9e56-4b22-b8e4-3d346b92c062\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" Oct 30 18:17:38.317280 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.317233 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d859a2eb-5149-4c88-a99b-85e08c7aaf2b-openshift-service-ca\") pod \"maas-api-e2e-api-b-8dc43c-884559b4-hgppv\" (UID: \"d859a2eb-5149-4c88-a99b-85e08c7aaf2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8dc43c-884559b4-hgppv" Oct 30 18:17:38.317383 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.317320 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d859a2eb-5149-4c88-a99b-85e08c7aaf2b-metrics-certs\") pod \"maas-api-e2e-api-b-8dc43c-884559b4-hgppv\" (UID: \"d859a2eb-5149-4c88-a99b-85e08c7aaf2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8dc43c-884559b4-hgppv" Oct 30 18:17:38.317383 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.317351 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db69e2c-9e56-4b22-b8e4-3d346b92c062-istio-envoy\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-f257t\" (UID: \"6db69e2c-9e56-4b22-b8e4-3d346b92c062\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" Oct 30 18:17:38.317451 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.317428 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d859a2eb-5149-4c88-a99b-85e08c7aaf2b-maas-api-tls\") pod \"maas-api-e2e-api-b-8dc43c-884559b4-hgppv\" (UID: \"d859a2eb-5149-4c88-a99b-85e08c7aaf2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8dc43c-884559b4-hgppv" Oct 30 18:17:38.317451 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.317445 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db69e2c-9e56-4b22-b8e4-3d346b92c062-credential-socket\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-f257t\" (UID: \"6db69e2c-9e56-4b22-b8e4-3d346b92c062\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" Oct 30 18:17:38.317528 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.317502 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db69e2c-9e56-4b22-b8e4-3d346b92c062-istio-podinfo\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-f257t\" (UID: \"6db69e2c-9e56-4b22-b8e4-3d346b92c062\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" Oct 30 18:17:38.317569 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.317548 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db69e2c-9e56-4b22-b8e4-3d346b92c062-workload-certs\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-f257t\" (UID: \"6db69e2c-9e56-4b22-b8e4-3d346b92c062\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" Oct 30 18:17:38.427274 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.427174 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db69e2c-9e56-4b22-b8e4-3d346b92c062-istio-envoy\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-f257t\" (UID: \"6db69e2c-9e56-4b22-b8e4-3d346b92c062\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" Oct 30 18:17:38.442159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.428022 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d859a2eb-5149-4c88-a99b-85e08c7aaf2b-maas-api-tls\") pod \"maas-api-e2e-api-b-8dc43c-884559b4-hgppv\" (UID: \"d859a2eb-5149-4c88-a99b-85e08c7aaf2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8dc43c-884559b4-hgppv" Oct 30 18:17:38.442644 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.442564 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db69e2c-9e56-4b22-b8e4-3d346b92c062-credential-socket\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-f257t\" (UID: \"6db69e2c-9e56-4b22-b8e4-3d346b92c062\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" Oct 30 18:17:38.442966 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.442946 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db69e2c-9e56-4b22-b8e4-3d346b92c062-istio-podinfo\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-f257t\" (UID: \"6db69e2c-9e56-4b22-b8e4-3d346b92c062\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" Oct 30 18:17:38.443193 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.443172 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db69e2c-9e56-4b22-b8e4-3d346b92c062-workload-certs\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-f257t\" (UID: \"6db69e2c-9e56-4b22-b8e4-3d346b92c062\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" Oct 30 18:17:38.443528 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.443507 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44gjh\" (UniqueName: \"kubernetes.io/projected/6db69e2c-9e56-4b22-b8e4-3d346b92c062-kube-api-access-44gjh\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-f257t\" (UID: \"6db69e2c-9e56-4b22-b8e4-3d346b92c062\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" Oct 30 18:17:38.443681 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.443660 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drxdw\" (UniqueName: \"kubernetes.io/projected/d859a2eb-5149-4c88-a99b-85e08c7aaf2b-kube-api-access-drxdw\") pod \"maas-api-e2e-api-b-8dc43c-884559b4-hgppv\" (UID: \"d859a2eb-5149-4c88-a99b-85e08c7aaf2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8dc43c-884559b4-hgppv" Oct 30 18:17:38.467997 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.445998 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db69e2c-9e56-4b22-b8e4-3d346b92c062-workload-certs\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-f257t\" (UID: \"6db69e2c-9e56-4b22-b8e4-3d346b92c062\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" Oct 30 18:17:38.468296 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.446645 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db69e2c-9e56-4b22-b8e4-3d346b92c062-credential-socket\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-f257t\" (UID: \"6db69e2c-9e56-4b22-b8e4-3d346b92c062\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" Oct 30 18:17:38.468450 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.453750 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db69e2c-9e56-4b22-b8e4-3d346b92c062-istio-envoy\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-f257t\" (UID: \"6db69e2c-9e56-4b22-b8e4-3d346b92c062\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" Oct 30 18:17:38.468558 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.467981 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db69e2c-9e56-4b22-b8e4-3d346b92c062-workload-socket\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-f257t\" (UID: \"6db69e2c-9e56-4b22-b8e4-3d346b92c062\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" Oct 30 18:17:38.468716 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.468290 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db69e2c-9e56-4b22-b8e4-3d346b92c062-workload-socket\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-f257t\" (UID: \"6db69e2c-9e56-4b22-b8e4-3d346b92c062\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" Oct 30 18:17:38.468796 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.468695 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db69e2c-9e56-4b22-b8e4-3d346b92c062-istiod-ca-cert\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-f257t\" (UID: \"6db69e2c-9e56-4b22-b8e4-3d346b92c062\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" Oct 30 18:17:38.468852 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.468814 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db69e2c-9e56-4b22-b8e4-3d346b92c062-istio-data\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-f257t\" (UID: \"6db69e2c-9e56-4b22-b8e4-3d346b92c062\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" Oct 30 18:17:38.468909 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.468857 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6db69e2c-9e56-4b22-b8e4-3d346b92c062-istio-token\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-f257t\" (UID: \"6db69e2c-9e56-4b22-b8e4-3d346b92c062\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" Oct 30 18:17:38.468961 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.468907 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d859a2eb-5149-4c88-a99b-85e08c7aaf2b-openshift-service-ca\") pod \"maas-api-e2e-api-b-8dc43c-884559b4-hgppv\" (UID: \"d859a2eb-5149-4c88-a99b-85e08c7aaf2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8dc43c-884559b4-hgppv" Oct 30 18:17:38.469011 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.468966 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d859a2eb-5149-4c88-a99b-85e08c7aaf2b-metrics-certs\") pod \"maas-api-e2e-api-b-8dc43c-884559b4-hgppv\" (UID: \"d859a2eb-5149-4c88-a99b-85e08c7aaf2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8dc43c-884559b4-hgppv" Oct 30 18:17:38.485160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.473837 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db69e2c-9e56-4b22-b8e4-3d346b92c062-istiod-ca-cert\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-f257t\" (UID: \"6db69e2c-9e56-4b22-b8e4-3d346b92c062\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" Oct 30 18:17:38.485411 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.482635 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d859a2eb-5149-4c88-a99b-85e08c7aaf2b-maas-api-tls\") pod \"maas-api-e2e-api-b-8dc43c-884559b4-hgppv\" (UID: \"d859a2eb-5149-4c88-a99b-85e08c7aaf2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8dc43c-884559b4-hgppv" Oct 30 18:17:38.485585 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.484091 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db69e2c-9e56-4b22-b8e4-3d346b92c062-istio-data\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-f257t\" (UID: \"6db69e2c-9e56-4b22-b8e4-3d346b92c062\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" Oct 30 18:17:38.485693 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.484758 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d859a2eb-5149-4c88-a99b-85e08c7aaf2b-openshift-service-ca\") pod \"maas-api-e2e-api-b-8dc43c-884559b4-hgppv\" (UID: \"d859a2eb-5149-4c88-a99b-85e08c7aaf2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8dc43c-884559b4-hgppv" Oct 30 18:17:38.489071 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.489044 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d859a2eb-5149-4c88-a99b-85e08c7aaf2b-metrics-certs\") pod \"maas-api-e2e-api-b-8dc43c-884559b4-hgppv\" (UID: \"d859a2eb-5149-4c88-a99b-85e08c7aaf2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8dc43c-884559b4-hgppv" Oct 30 18:17:38.496929 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.496900 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44gjh\" (UniqueName: \"kubernetes.io/projected/6db69e2c-9e56-4b22-b8e4-3d346b92c062-kube-api-access-44gjh\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-f257t\" (UID: \"6db69e2c-9e56-4b22-b8e4-3d346b92c062\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" Oct 30 18:17:38.506995 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.502986 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drxdw\" (UniqueName: \"kubernetes.io/projected/d859a2eb-5149-4c88-a99b-85e08c7aaf2b-kube-api-access-drxdw\") pod \"maas-api-e2e-api-b-8dc43c-884559b4-hgppv\" (UID: \"d859a2eb-5149-4c88-a99b-85e08c7aaf2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8dc43c-884559b4-hgppv" Oct 30 18:17:38.506995 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.506100 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db69e2c-9e56-4b22-b8e4-3d346b92c062-istio-podinfo\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-f257t\" (UID: \"6db69e2c-9e56-4b22-b8e4-3d346b92c062\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" Oct 30 18:17:38.506995 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.506725 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6db69e2c-9e56-4b22-b8e4-3d346b92c062-istio-token\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-f257t\" (UID: \"6db69e2c-9e56-4b22-b8e4-3d346b92c062\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" Oct 30 18:17:38.525111 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.524792 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" Oct 30 18:17:38.774687 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.774591 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8dc43c-884559b4-hgppv" Oct 30 18:17:38.897888 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.897854 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t"] Oct 30 18:17:38.908556 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:17:38.908522 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6db69e2c_9e56_4b22_b8e4_3d346b92c062.slice/crio-2a6d224e7181600427ae783ea6df3fe7b119e8ed4ee6f83d52ef56179b3ec418 WatchSource:0}: Error finding container 2a6d224e7181600427ae783ea6df3fe7b119e8ed4ee6f83d52ef56179b3ec418: Status 404 returned error can't find the container with id 2a6d224e7181600427ae783ea6df3fe7b119e8ed4ee6f83d52ef56179b3ec418 Oct 30 18:17:38.911753 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.911721 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:17:38.911851 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.911807 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:17:38.911908 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:38.911852 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:17:39.075017 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:39.074984 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" event={"ID":"6db69e2c-9e56-4b22-b8e4-3d346b92c062","Type":"ContainerStarted","Data":"2a6d224e7181600427ae783ea6df3fe7b119e8ed4ee6f83d52ef56179b3ec418"} Oct 30 18:17:39.101353 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:39.101326 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8dc43c-884559b4-hgppv"] Oct 30 18:17:39.102600 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:17:39.102572 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd859a2eb_5149_4c88_a99b_85e08c7aaf2b.slice/crio-e0f2cceffe8d477b716c78b880673e06100fa90fcc24943f6acf055f15bdc75e WatchSource:0}: Error finding container e0f2cceffe8d477b716c78b880673e06100fa90fcc24943f6acf055f15bdc75e: Status 404 returned error can't find the container with id e0f2cceffe8d477b716c78b880673e06100fa90fcc24943f6acf055f15bdc75e Oct 30 18:17:40.085290 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:40.085250 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" event={"ID":"6db69e2c-9e56-4b22-b8e4-3d346b92c062","Type":"ContainerStarted","Data":"33a4c4b4c4c56e01791bc62d5a1538254f534d969eec70d702ab02df5b99ab52"} Oct 30 18:17:40.087190 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:40.087128 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8dc43c-884559b4-hgppv" event={"ID":"d859a2eb-5149-4c88-a99b-85e08c7aaf2b","Type":"ContainerStarted","Data":"21dae01a25b325f96079a234df55bd1c05725d845a5b039e0244b22e0b4a8b75"} Oct 30 18:17:40.087329 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:40.087212 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8dc43c-884559b4-hgppv" event={"ID":"d859a2eb-5149-4c88-a99b-85e08c7aaf2b","Type":"ContainerStarted","Data":"e0f2cceffe8d477b716c78b880673e06100fa90fcc24943f6acf055f15bdc75e"} Oct 30 18:17:40.087329 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:40.087239 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8dc43c-884559b4-hgppv" Oct 30 18:17:40.171090 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:40.171029 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" podStartSLOduration=2.171008621 podStartE2EDuration="2.171008621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:40.139818912 +0000 UTC m=+1157.229881257" watchObservedRunningTime="2026-10-30 18:17:40.171008621 +0000 UTC m=+1157.261070965" Oct 30 18:17:40.546529 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:40.544310 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" Oct 30 18:17:40.554163 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:40.551277 4442 patch_prober.go:29] interesting pod/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 18:17:40.554163 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:40.551340 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" podUID="6db69e2c-9e56-4b22-b8e4-3d346b92c062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 18:17:41.547159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:41.546249 4442 patch_prober.go:29] interesting pod/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 18:17:41.547159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:41.546314 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" podUID="6db69e2c-9e56-4b22-b8e4-3d346b92c062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 18:17:42.544718 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:42.544506 4442 patch_prober.go:29] interesting pod/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 18:17:42.544718 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:42.544566 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" podUID="6db69e2c-9e56-4b22-b8e4-3d346b92c062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 18:17:43.542160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:43.541587 4442 patch_prober.go:29] interesting pod/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 18:17:43.542160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:43.541650 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" podUID="6db69e2c-9e56-4b22-b8e4-3d346b92c062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 18:17:44.544099 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:44.544063 4442 patch_prober.go:29] interesting pod/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 18:17:44.544791 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:44.544761 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" podUID="6db69e2c-9e56-4b22-b8e4-3d346b92c062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 18:17:44.755728 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:44.755665 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8dc43c-884559b4-hgppv" podStartSLOduration=6.280978377 podStartE2EDuration="6.755646758s" totalImagesPullingTime="474.668381ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:40.245944865 +0000 UTC m=+1157.336007205" watchObservedRunningTime="2026-10-30 18:17:44.755646758 +0000 UTC m=+1161.845709096" Oct 30 18:17:44.758157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:44.758111 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4b709d-7d96656586-hxnbn"] Oct 30 18:17:44.785542 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:44.785506 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4b709d-5df8f677bc-b4srr"] Oct 30 18:17:45.548178 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:45.540578 4442 patch_prober.go:29] interesting pod/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 18:17:45.548178 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:45.540641 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" podUID="6db69e2c-9e56-4b22-b8e4-3d346b92c062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 18:17:46.102574 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:46.102544 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8dc43c-884559b4-hgppv" Oct 30 18:17:46.544831 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:46.544798 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" Oct 30 18:17:46.545552 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:46.545310 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" Oct 30 18:17:46.546829 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:46.546804 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" Oct 30 18:17:47.635245 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:17:47.635213 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:17:49.759120 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:49.759077 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-4b709d-7d96656586-hxnbn" podUID="0cc9578c-0f6a-48a4-a71c-c0a30dc75171" containerName="payload-processing" containerID="cri-o://596a8c7fed6de35c869328364dd06b27ea5d79038fc3fd05af12ece017583ebb" gracePeriod=25 Oct 30 18:17:49.786679 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:49.786645 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4b709d-5df8f677bc-b4srr" podUID="361a8ea8-96e6-404d-b694-052dbb6fb070" containerName="payload-pre-processing" containerID="cri-o://731ee324d9f8808cabfe7b5e56c20227c13224411bc2470b75001129c093d403" gracePeriod=25 Oct 30 18:17:50.113302 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:50.113271 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4b709d-7d96656586-hxnbn" Oct 30 18:17:50.200399 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:50.200371 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4b709d-5df8f677bc-b4srr" Oct 30 18:17:50.250023 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:50.249989 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/361a8ea8-96e6-404d-b694-052dbb6fb070-kube-api-access-tbq96\" (UniqueName: \"kubernetes.io/projected/361a8ea8-96e6-404d-b694-052dbb6fb070-kube-api-access-tbq96\") pod \"361a8ea8-96e6-404d-b694-052dbb6fb070\" (UID: \"361a8ea8-96e6-404d-b694-052dbb6fb070\") " Oct 30 18:17:50.250235 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:50.250088 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/361a8ea8-96e6-404d-b694-052dbb6fb070-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/361a8ea8-96e6-404d-b694-052dbb6fb070-plugins-config-volume\") pod \"361a8ea8-96e6-404d-b694-052dbb6fb070\" (UID: \"361a8ea8-96e6-404d-b694-052dbb6fb070\") " Oct 30 18:17:50.250235 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:50.250159 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cc9578c-0f6a-48a4-a71c-c0a30dc75171-kube-api-access-nsg5q\" (UniqueName: \"kubernetes.io/projected/0cc9578c-0f6a-48a4-a71c-c0a30dc75171-kube-api-access-nsg5q\") pod \"0cc9578c-0f6a-48a4-a71c-c0a30dc75171\" (UID: \"0cc9578c-0f6a-48a4-a71c-c0a30dc75171\") " Oct 30 18:17:50.250235 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:50.250203 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cc9578c-0f6a-48a4-a71c-c0a30dc75171-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cc9578c-0f6a-48a4-a71c-c0a30dc75171-plugins-config-volume\") pod \"0cc9578c-0f6a-48a4-a71c-c0a30dc75171\" (UID: \"0cc9578c-0f6a-48a4-a71c-c0a30dc75171\") " Oct 30 18:17:50.250981 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:50.250790 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/361a8ea8-96e6-404d-b694-052dbb6fb070-plugins-config-volume" pod "361a8ea8-96e6-404d-b694-052dbb6fb070" (UID: "361a8ea8-96e6-404d-b694-052dbb6fb070"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:17:50.251114 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:50.251007 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cc9578c-0f6a-48a4-a71c-c0a30dc75171-plugins-config-volume" pod "0cc9578c-0f6a-48a4-a71c-c0a30dc75171" (UID: "0cc9578c-0f6a-48a4-a71c-c0a30dc75171"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:17:50.262211 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:50.253804 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc9578c-0f6a-48a4-a71c-c0a30dc75171-kube-api-access-nsg5q" pod "0cc9578c-0f6a-48a4-a71c-c0a30dc75171" (UID: "0cc9578c-0f6a-48a4-a71c-c0a30dc75171"). InnerVolumeSpecName "kube-api-access-nsg5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:17:50.262211 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:50.254289 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/361a8ea8-96e6-404d-b694-052dbb6fb070-kube-api-access-tbq96" pod "361a8ea8-96e6-404d-b694-052dbb6fb070" (UID: "361a8ea8-96e6-404d-b694-052dbb6fb070"). InnerVolumeSpecName "kube-api-access-tbq96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:17:50.294753 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:50.294659 4442 generic.go:356] "Generic (PLEG): container finished" podID="0cc9578c-0f6a-48a4-a71c-c0a30dc75171" containerID="596a8c7fed6de35c869328364dd06b27ea5d79038fc3fd05af12ece017583ebb" exitCode=0 Oct 30 18:17:50.294753 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:50.294715 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4b709d-7d96656586-hxnbn" Oct 30 18:17:50.294957 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:50.294754 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4b709d-7d96656586-hxnbn" event={"ID":"0cc9578c-0f6a-48a4-a71c-c0a30dc75171","Type":"ContainerDied","Data":"596a8c7fed6de35c869328364dd06b27ea5d79038fc3fd05af12ece017583ebb"} Oct 30 18:17:50.294957 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:50.294823 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4b709d-7d96656586-hxnbn" event={"ID":"0cc9578c-0f6a-48a4-a71c-c0a30dc75171","Type":"ContainerDied","Data":"ab3061310ae5f9e8f7108496979e54b35da899a091e0f2fc91d9c47bcc005c62"} Oct 30 18:17:50.294957 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:50.294853 4442 scope.go:122] "RemoveContainer" containerID="596a8c7fed6de35c869328364dd06b27ea5d79038fc3fd05af12ece017583ebb" Oct 30 18:17:50.296926 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:50.296887 4442 generic.go:356] "Generic (PLEG): container finished" podID="361a8ea8-96e6-404d-b694-052dbb6fb070" containerID="731ee324d9f8808cabfe7b5e56c20227c13224411bc2470b75001129c093d403" exitCode=0 Oct 30 18:17:50.297031 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:50.296942 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4b709d-5df8f677bc-b4srr" event={"ID":"361a8ea8-96e6-404d-b694-052dbb6fb070","Type":"ContainerDied","Data":"731ee324d9f8808cabfe7b5e56c20227c13224411bc2470b75001129c093d403"} Oct 30 18:17:50.297031 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:50.296984 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4b709d-5df8f677bc-b4srr" event={"ID":"361a8ea8-96e6-404d-b694-052dbb6fb070","Type":"ContainerDied","Data":"1fc4417deef02e4c5f6a2c83fea22010a2986be951ce04520f6e379f8ec727cf"} Oct 30 18:17:50.297209 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:50.297058 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4b709d-5df8f677bc-b4srr" Oct 30 18:17:50.309809 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:50.309795 4442 scope.go:122] "RemoveContainer" containerID="596a8c7fed6de35c869328364dd06b27ea5d79038fc3fd05af12ece017583ebb" Oct 30 18:17:50.310144 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:17:50.310113 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"596a8c7fed6de35c869328364dd06b27ea5d79038fc3fd05af12ece017583ebb\": container with ID starting with 596a8c7fed6de35c869328364dd06b27ea5d79038fc3fd05af12ece017583ebb not found: ID does not exist" containerID="596a8c7fed6de35c869328364dd06b27ea5d79038fc3fd05af12ece017583ebb" Oct 30 18:17:50.310214 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:50.310159 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"596a8c7fed6de35c869328364dd06b27ea5d79038fc3fd05af12ece017583ebb"} err="failed to get container status \"596a8c7fed6de35c869328364dd06b27ea5d79038fc3fd05af12ece017583ebb\": rpc error: code = NotFound desc = could not find container \"596a8c7fed6de35c869328364dd06b27ea5d79038fc3fd05af12ece017583ebb\": container with ID starting with 596a8c7fed6de35c869328364dd06b27ea5d79038fc3fd05af12ece017583ebb not found: ID does not exist" Oct 30 18:17:50.310214 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:50.310177 4442 scope.go:122] "RemoveContainer" containerID="731ee324d9f8808cabfe7b5e56c20227c13224411bc2470b75001129c093d403" Oct 30 18:17:50.319985 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:50.319971 4442 scope.go:122] "RemoveContainer" containerID="731ee324d9f8808cabfe7b5e56c20227c13224411bc2470b75001129c093d403" Oct 30 18:17:50.320281 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:17:50.320260 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"731ee324d9f8808cabfe7b5e56c20227c13224411bc2470b75001129c093d403\": container with ID starting with 731ee324d9f8808cabfe7b5e56c20227c13224411bc2470b75001129c093d403 not found: ID does not exist" containerID="731ee324d9f8808cabfe7b5e56c20227c13224411bc2470b75001129c093d403" Oct 30 18:17:50.320356 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:50.320286 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"731ee324d9f8808cabfe7b5e56c20227c13224411bc2470b75001129c093d403"} err="failed to get container status \"731ee324d9f8808cabfe7b5e56c20227c13224411bc2470b75001129c093d403\": rpc error: code = NotFound desc = could not find container \"731ee324d9f8808cabfe7b5e56c20227c13224411bc2470b75001129c093d403\": container with ID starting with 731ee324d9f8808cabfe7b5e56c20227c13224411bc2470b75001129c093d403 not found: ID does not exist" Oct 30 18:17:50.352012 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:50.351976 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/361a8ea8-96e6-404d-b694-052dbb6fb070-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:17:50.352012 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:50.352008 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsg5q\" (UniqueName: \"kubernetes.io/projected/0cc9578c-0f6a-48a4-a71c-c0a30dc75171-kube-api-access-nsg5q\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:17:50.352012 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:50.352017 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cc9578c-0f6a-48a4-a71c-c0a30dc75171-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:17:50.352303 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:50.352026 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbq96\" (UniqueName: \"kubernetes.io/projected/361a8ea8-96e6-404d-b694-052dbb6fb070-kube-api-access-tbq96\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:17:50.360948 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:50.360914 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4b709d-7d96656586-hxnbn"] Oct 30 18:17:50.374842 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:50.374806 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4b709d-7d96656586-hxnbn"] Oct 30 18:17:50.401940 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:50.401901 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4b709d-5df8f677bc-b4srr"] Oct 30 18:17:50.406486 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:50.406465 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4b709d-5df8f677bc-b4srr"] Oct 30 18:17:51.069900 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:51.069825 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cc9578c-0f6a-48a4-a71c-c0a30dc75171" path="/var/lib/kubelet/pods/0cc9578c-0f6a-48a4-a71c-c0a30dc75171/volumes" Oct 30 18:17:51.070298 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:51.070265 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="361a8ea8-96e6-404d-b694-052dbb6fb070" path="/var/lib/kubelet/pods/361a8ea8-96e6-404d-b694-052dbb6fb070/volumes" Oct 30 18:17:55.492243 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:55.489287 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-x2rr2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:17:55.492243 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:17:55.489346 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" podUID="e03523d4-1b6f-4a80-9006-26b0345d809d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:15.726088 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:15.726052 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4b709d-754cc6c5c8-fchc4"] Oct 30 18:18:15.726563 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:15.726349 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b709d-754cc6c5c8-fchc4" podUID="6db04ea0-e110-4d71-9a90-f37b4c431a3b" containerName="maas-api" containerID="cri-o://2e82e89fc8cf0d3b905dc3407f3fdd7676fef629d7426c0579d487cb265e4770" gracePeriod=30 Oct 30 18:18:16.245412 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:16.244995 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b709d-754cc6c5c8-fchc4" Oct 30 18:18:16.379529 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:16.375736 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6db04ea0-e110-4d71-9a90-f37b4c431a3b-kube-api-access-v7dlj\" (UniqueName: \"kubernetes.io/projected/6db04ea0-e110-4d71-9a90-f37b4c431a3b-kube-api-access-v7dlj\") pod \"6db04ea0-e110-4d71-9a90-f37b4c431a3b\" (UID: \"6db04ea0-e110-4d71-9a90-f37b4c431a3b\") " Oct 30 18:18:16.379529 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:16.376013 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6db04ea0-e110-4d71-9a90-f37b4c431a3b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6db04ea0-e110-4d71-9a90-f37b4c431a3b-maas-api-tls\") pod \"6db04ea0-e110-4d71-9a90-f37b4c431a3b\" (UID: \"6db04ea0-e110-4d71-9a90-f37b4c431a3b\") " Oct 30 18:18:16.379529 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:16.376065 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6db04ea0-e110-4d71-9a90-f37b4c431a3b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6db04ea0-e110-4d71-9a90-f37b4c431a3b-openshift-service-ca\") pod \"6db04ea0-e110-4d71-9a90-f37b4c431a3b\" (UID: \"6db04ea0-e110-4d71-9a90-f37b4c431a3b\") " Oct 30 18:18:16.379529 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:16.376180 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6db04ea0-e110-4d71-9a90-f37b4c431a3b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6db04ea0-e110-4d71-9a90-f37b4c431a3b-metrics-certs\") pod \"6db04ea0-e110-4d71-9a90-f37b4c431a3b\" (UID: \"6db04ea0-e110-4d71-9a90-f37b4c431a3b\") " Oct 30 18:18:16.455438 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:16.401937 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6db04ea0-e110-4d71-9a90-f37b4c431a3b-openshift-service-ca" pod "6db04ea0-e110-4d71-9a90-f37b4c431a3b" (UID: "6db04ea0-e110-4d71-9a90-f37b4c431a3b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:16.455438 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:16.413320 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6db04ea0-e110-4d71-9a90-f37b4c431a3b-metrics-certs" pod "6db04ea0-e110-4d71-9a90-f37b4c431a3b" (UID: "6db04ea0-e110-4d71-9a90-f37b4c431a3b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:16.477208 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:16.474742 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db04ea0-e110-4d71-9a90-f37b4c431a3b-kube-api-access-v7dlj" pod "6db04ea0-e110-4d71-9a90-f37b4c431a3b" (UID: "6db04ea0-e110-4d71-9a90-f37b4c431a3b"). InnerVolumeSpecName "kube-api-access-v7dlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:16.499176 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:16.483119 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6db04ea0-e110-4d71-9a90-f37b4c431a3b-openshift-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:18:16.499176 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:16.483169 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6db04ea0-e110-4d71-9a90-f37b4c431a3b-metrics-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:18:16.499176 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:16.483186 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7dlj\" (UniqueName: \"kubernetes.io/projected/6db04ea0-e110-4d71-9a90-f37b4c431a3b-kube-api-access-v7dlj\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:18:16.500195 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:16.500161 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6db04ea0-e110-4d71-9a90-f37b4c431a3b-maas-api-tls" pod "6db04ea0-e110-4d71-9a90-f37b4c431a3b" (UID: "6db04ea0-e110-4d71-9a90-f37b4c431a3b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:16.614439 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:16.608939 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6db04ea0-e110-4d71-9a90-f37b4c431a3b-maas-api-tls\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:18:16.795632 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:16.795572 4442 generic.go:356] "Generic (PLEG): container finished" podID="6db04ea0-e110-4d71-9a90-f37b4c431a3b" containerID="2e82e89fc8cf0d3b905dc3407f3fdd7676fef629d7426c0579d487cb265e4770" exitCode=0 Oct 30 18:18:16.796301 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:16.796276 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b709d-754cc6c5c8-fchc4" Oct 30 18:18:16.796434 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:16.796412 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b709d-754cc6c5c8-fchc4" event={"ID":"6db04ea0-e110-4d71-9a90-f37b4c431a3b","Type":"ContainerDied","Data":"2e82e89fc8cf0d3b905dc3407f3fdd7676fef629d7426c0579d487cb265e4770"} Oct 30 18:18:16.796506 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:16.796475 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b709d-754cc6c5c8-fchc4" event={"ID":"6db04ea0-e110-4d71-9a90-f37b4c431a3b","Type":"ContainerDied","Data":"0a6b5ebd2fa2dc933e2afa486cbb28ebf25ae19e2c45a57fa69ad06e6fa3851a"} Oct 30 18:18:16.796561 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:16.796509 4442 scope.go:122] "RemoveContainer" containerID="2e82e89fc8cf0d3b905dc3407f3fdd7676fef629d7426c0579d487cb265e4770" Oct 30 18:18:16.814297 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:16.814278 4442 scope.go:122] "RemoveContainer" containerID="2e82e89fc8cf0d3b905dc3407f3fdd7676fef629d7426c0579d487cb265e4770" Oct 30 18:18:16.815656 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:18:16.815630 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e82e89fc8cf0d3b905dc3407f3fdd7676fef629d7426c0579d487cb265e4770\": container with ID starting with 2e82e89fc8cf0d3b905dc3407f3fdd7676fef629d7426c0579d487cb265e4770 not found: ID does not exist" containerID="2e82e89fc8cf0d3b905dc3407f3fdd7676fef629d7426c0579d487cb265e4770" Oct 30 18:18:16.815756 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:16.815663 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e82e89fc8cf0d3b905dc3407f3fdd7676fef629d7426c0579d487cb265e4770"} err="failed to get container status \"2e82e89fc8cf0d3b905dc3407f3fdd7676fef629d7426c0579d487cb265e4770\": rpc error: code = NotFound desc = could not find container \"2e82e89fc8cf0d3b905dc3407f3fdd7676fef629d7426c0579d487cb265e4770\": container with ID starting with 2e82e89fc8cf0d3b905dc3407f3fdd7676fef629d7426c0579d487cb265e4770 not found: ID does not exist" Oct 30 18:18:16.851672 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:16.851635 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4b709d-754cc6c5c8-fchc4"] Oct 30 18:18:16.862691 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:16.862657 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4b709d-754cc6c5c8-fchc4"] Oct 30 18:18:17.108541 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:17.108421 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6db04ea0-e110-4d71-9a90-f37b4c431a3b" path="/var/lib/kubelet/pods/6db04ea0-e110-4d71-9a90-f37b4c431a3b/volumes" Oct 30 18:18:17.563666 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:18:17.563629 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:18:23.244923 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:23.244892 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52rn6_516c6b6c-fc40-49cf-93b5-1c4471b02cf0/ovn-acl-logging/0.log" Oct 30 18:18:23.247268 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:23.247246 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52rn6_516c6b6c-fc40-49cf-93b5-1c4471b02cf0/ovn-acl-logging/0.log" Oct 30 18:18:23.672861 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:23.671307 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2"] Oct 30 18:18:23.672861 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:23.671599 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" podUID="059512a2-ce79-4426-bd6d-b11fede6864f" containerName="istio-proxy" containerID="cri-o://d6b06b6275d295c62b3ca4b3b9cf9fd50ffe482c8a49f60856911ebbfb870515" gracePeriod=30 Oct 30 18:18:23.895152 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:23.895094 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f15587-674b5c966-d8pd8"] Oct 30 18:18:24.165073 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:24.164361 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f15587-7cccff6f5-89g5c"] Oct 30 18:18:28.907887 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:28.907762 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-f15587-674b5c966-d8pd8" podUID="0e1a7e8c-80ee-4549-b261-8cd5cb1ef16e" containerName="payload-processing" containerID="cri-o://e4b48df9554f4a6e40c8853844b41e2979b1db08063f3196db5d097854f9f699" gracePeriod=25 Oct 30 18:18:29.051572 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.051265 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" Oct 30 18:18:29.188432 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.187830 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/059512a2-ce79-4426-bd6d-b11fede6864f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/059512a2-ce79-4426-bd6d-b11fede6864f-credential-socket\") pod \"059512a2-ce79-4426-bd6d-b11fede6864f\" (UID: \"059512a2-ce79-4426-bd6d-b11fede6864f\") " Oct 30 18:18:29.188432 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.187931 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/059512a2-ce79-4426-bd6d-b11fede6864f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/059512a2-ce79-4426-bd6d-b11fede6864f-workload-certs\") pod \"059512a2-ce79-4426-bd6d-b11fede6864f\" (UID: \"059512a2-ce79-4426-bd6d-b11fede6864f\") " Oct 30 18:18:29.188432 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.188073 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/059512a2-ce79-4426-bd6d-b11fede6864f-kube-api-access-wnrvz\" (UniqueName: \"kubernetes.io/projected/059512a2-ce79-4426-bd6d-b11fede6864f-kube-api-access-wnrvz\") pod \"059512a2-ce79-4426-bd6d-b11fede6864f\" (UID: \"059512a2-ce79-4426-bd6d-b11fede6864f\") " Oct 30 18:18:29.188432 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.188168 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/059512a2-ce79-4426-bd6d-b11fede6864f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/059512a2-ce79-4426-bd6d-b11fede6864f-workload-socket\") pod \"059512a2-ce79-4426-bd6d-b11fede6864f\" (UID: \"059512a2-ce79-4426-bd6d-b11fede6864f\") " Oct 30 18:18:29.190181 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.188916 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/059512a2-ce79-4426-bd6d-b11fede6864f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/059512a2-ce79-4426-bd6d-b11fede6864f-istio-data\") pod \"059512a2-ce79-4426-bd6d-b11fede6864f\" (UID: \"059512a2-ce79-4426-bd6d-b11fede6864f\") " Oct 30 18:18:29.190181 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.189008 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/059512a2-ce79-4426-bd6d-b11fede6864f-istio-token\" (UniqueName: \"kubernetes.io/projected/059512a2-ce79-4426-bd6d-b11fede6864f-istio-token\") pod \"059512a2-ce79-4426-bd6d-b11fede6864f\" (UID: \"059512a2-ce79-4426-bd6d-b11fede6864f\") " Oct 30 18:18:29.190181 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.189061 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/059512a2-ce79-4426-bd6d-b11fede6864f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/059512a2-ce79-4426-bd6d-b11fede6864f-istio-podinfo\") pod \"059512a2-ce79-4426-bd6d-b11fede6864f\" (UID: \"059512a2-ce79-4426-bd6d-b11fede6864f\") " Oct 30 18:18:29.190181 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.189163 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/059512a2-ce79-4426-bd6d-b11fede6864f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/059512a2-ce79-4426-bd6d-b11fede6864f-istio-envoy\") pod \"059512a2-ce79-4426-bd6d-b11fede6864f\" (UID: \"059512a2-ce79-4426-bd6d-b11fede6864f\") " Oct 30 18:18:29.190181 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.189218 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/059512a2-ce79-4426-bd6d-b11fede6864f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/059512a2-ce79-4426-bd6d-b11fede6864f-istiod-ca-cert\") pod \"059512a2-ce79-4426-bd6d-b11fede6864f\" (UID: \"059512a2-ce79-4426-bd6d-b11fede6864f\") " Oct 30 18:18:29.191819 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.191607 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/059512a2-ce79-4426-bd6d-b11fede6864f-credential-socket" pod "059512a2-ce79-4426-bd6d-b11fede6864f" (UID: "059512a2-ce79-4426-bd6d-b11fede6864f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:18:29.192346 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.191990 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/059512a2-ce79-4426-bd6d-b11fede6864f-workload-certs" pod "059512a2-ce79-4426-bd6d-b11fede6864f" (UID: "059512a2-ce79-4426-bd6d-b11fede6864f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:18:29.194432 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.194192 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/059512a2-ce79-4426-bd6d-b11fede6864f-istio-data" pod "059512a2-ce79-4426-bd6d-b11fede6864f" (UID: "059512a2-ce79-4426-bd6d-b11fede6864f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:18:29.194916 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.194869 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/059512a2-ce79-4426-bd6d-b11fede6864f-istiod-ca-cert" pod "059512a2-ce79-4426-bd6d-b11fede6864f" (UID: "059512a2-ce79-4426-bd6d-b11fede6864f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:29.205798 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.205700 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/059512a2-ce79-4426-bd6d-b11fede6864f-kube-api-access-wnrvz" pod "059512a2-ce79-4426-bd6d-b11fede6864f" (UID: "059512a2-ce79-4426-bd6d-b11fede6864f"). InnerVolumeSpecName "kube-api-access-wnrvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:29.223171 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.218254 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/059512a2-ce79-4426-bd6d-b11fede6864f-workload-socket" pod "059512a2-ce79-4426-bd6d-b11fede6864f" (UID: "059512a2-ce79-4426-bd6d-b11fede6864f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:18:29.253258 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.249300 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/059512a2-ce79-4426-bd6d-b11fede6864f-istio-envoy" pod "059512a2-ce79-4426-bd6d-b11fede6864f" (UID: "059512a2-ce79-4426-bd6d-b11fede6864f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:18:29.253258 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.249436 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/059512a2-ce79-4426-bd6d-b11fede6864f-istio-token" pod "059512a2-ce79-4426-bd6d-b11fede6864f" (UID: "059512a2-ce79-4426-bd6d-b11fede6864f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:29.263472 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.263229 4442 generic.go:356] "Generic (PLEG): container finished" podID="0e1a7e8c-80ee-4549-b261-8cd5cb1ef16e" containerID="e4b48df9554f4a6e40c8853844b41e2979b1db08063f3196db5d097854f9f699" exitCode=0 Oct 30 18:18:29.263472 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.263391 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f15587-674b5c966-d8pd8" event={"ID":"0e1a7e8c-80ee-4549-b261-8cd5cb1ef16e","Type":"ContainerDied","Data":"e4b48df9554f4a6e40c8853844b41e2979b1db08063f3196db5d097854f9f699"} Oct 30 18:18:29.268080 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.266287 4442 generic.go:356] "Generic (PLEG): container finished" podID="059512a2-ce79-4426-bd6d-b11fede6864f" containerID="d6b06b6275d295c62b3ca4b3b9cf9fd50ffe482c8a49f60856911ebbfb870515" exitCode=0 Oct 30 18:18:29.268080 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.266417 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" event={"ID":"059512a2-ce79-4426-bd6d-b11fede6864f","Type":"ContainerDied","Data":"d6b06b6275d295c62b3ca4b3b9cf9fd50ffe482c8a49f60856911ebbfb870515"} Oct 30 18:18:29.268080 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.266466 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" event={"ID":"059512a2-ce79-4426-bd6d-b11fede6864f","Type":"ContainerDied","Data":"4f677f5a6ae28cebcb12542779ac9d9c1be15797bfcaf34739d087e662b618f8"} Oct 30 18:18:29.268080 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.266495 4442 scope.go:122] "RemoveContainer" containerID="d6b06b6275d295c62b3ca4b3b9cf9fd50ffe482c8a49f60856911ebbfb870515" Oct 30 18:18:29.268080 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.266675 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2" Oct 30 18:18:29.292759 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.292563 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/059512a2-ce79-4426-bd6d-b11fede6864f-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:18:29.292759 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.292599 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/059512a2-ce79-4426-bd6d-b11fede6864f-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:18:29.292759 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.292619 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnrvz\" (UniqueName: \"kubernetes.io/projected/059512a2-ce79-4426-bd6d-b11fede6864f-kube-api-access-wnrvz\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:18:29.292759 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.292634 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/059512a2-ce79-4426-bd6d-b11fede6864f-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:18:29.292759 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.292649 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/059512a2-ce79-4426-bd6d-b11fede6864f-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:18:29.292759 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.292662 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/059512a2-ce79-4426-bd6d-b11fede6864f-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:18:29.292759 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.292675 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/059512a2-ce79-4426-bd6d-b11fede6864f-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:18:29.292759 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.292687 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/059512a2-ce79-4426-bd6d-b11fede6864f-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:18:29.307903 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.305258 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/059512a2-ce79-4426-bd6d-b11fede6864f-istio-podinfo" pod "059512a2-ce79-4426-bd6d-b11fede6864f" (UID: "059512a2-ce79-4426-bd6d-b11fede6864f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:18:29.307903 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.305712 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f15587-7cccff6f5-89g5c" podUID="05f66220-a64e-4c45-866f-caa9b332e3ec" containerName="payload-pre-processing" containerID="cri-o://0b9239a9d00486822b985281a34bb607f6667587fc251ed8beb02bf15800bbce" gracePeriod=25 Oct 30 18:18:29.320351 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.320234 4442 scope.go:122] "RemoveContainer" containerID="7fd2dc80a09a09123b64a0e7047b73ebd91c5533354165648e6f964709503e5e" Oct 30 18:18:29.324902 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.324641 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f15587-674b5c966-d8pd8" Oct 30 18:18:29.342323 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.341984 4442 scope.go:122] "RemoveContainer" containerID="d6b06b6275d295c62b3ca4b3b9cf9fd50ffe482c8a49f60856911ebbfb870515" Oct 30 18:18:29.342875 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:18:29.342564 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6b06b6275d295c62b3ca4b3b9cf9fd50ffe482c8a49f60856911ebbfb870515\": container with ID starting with d6b06b6275d295c62b3ca4b3b9cf9fd50ffe482c8a49f60856911ebbfb870515 not found: ID does not exist" containerID="d6b06b6275d295c62b3ca4b3b9cf9fd50ffe482c8a49f60856911ebbfb870515" Oct 30 18:18:29.342875 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.342601 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6b06b6275d295c62b3ca4b3b9cf9fd50ffe482c8a49f60856911ebbfb870515"} err="failed to get container status \"d6b06b6275d295c62b3ca4b3b9cf9fd50ffe482c8a49f60856911ebbfb870515\": rpc error: code = NotFound desc = could not find container \"d6b06b6275d295c62b3ca4b3b9cf9fd50ffe482c8a49f60856911ebbfb870515\": container with ID starting with d6b06b6275d295c62b3ca4b3b9cf9fd50ffe482c8a49f60856911ebbfb870515 not found: ID does not exist" Oct 30 18:18:29.342875 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.342625 4442 scope.go:122] "RemoveContainer" containerID="7fd2dc80a09a09123b64a0e7047b73ebd91c5533354165648e6f964709503e5e" Oct 30 18:18:29.343302 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:18:29.343217 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fd2dc80a09a09123b64a0e7047b73ebd91c5533354165648e6f964709503e5e\": container with ID starting with 7fd2dc80a09a09123b64a0e7047b73ebd91c5533354165648e6f964709503e5e not found: ID does not exist" containerID="7fd2dc80a09a09123b64a0e7047b73ebd91c5533354165648e6f964709503e5e" Oct 30 18:18:29.343302 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.343249 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fd2dc80a09a09123b64a0e7047b73ebd91c5533354165648e6f964709503e5e"} err="failed to get container status \"7fd2dc80a09a09123b64a0e7047b73ebd91c5533354165648e6f964709503e5e\": rpc error: code = NotFound desc = could not find container \"7fd2dc80a09a09123b64a0e7047b73ebd91c5533354165648e6f964709503e5e\": container with ID starting with 7fd2dc80a09a09123b64a0e7047b73ebd91c5533354165648e6f964709503e5e not found: ID does not exist" Oct 30 18:18:29.393713 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.393668 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e1a7e8c-80ee-4549-b261-8cd5cb1ef16e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e1a7e8c-80ee-4549-b261-8cd5cb1ef16e-plugins-config-volume\") pod \"0e1a7e8c-80ee-4549-b261-8cd5cb1ef16e\" (UID: \"0e1a7e8c-80ee-4549-b261-8cd5cb1ef16e\") " Oct 30 18:18:29.393996 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.393975 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e1a7e8c-80ee-4549-b261-8cd5cb1ef16e-kube-api-access-fg62c\" (UniqueName: \"kubernetes.io/projected/0e1a7e8c-80ee-4549-b261-8cd5cb1ef16e-kube-api-access-fg62c\") pod \"0e1a7e8c-80ee-4549-b261-8cd5cb1ef16e\" (UID: \"0e1a7e8c-80ee-4549-b261-8cd5cb1ef16e\") " Oct 30 18:18:29.394713 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.394693 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/059512a2-ce79-4426-bd6d-b11fede6864f-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:18:29.395177 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.395115 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e1a7e8c-80ee-4549-b261-8cd5cb1ef16e-plugins-config-volume" pod "0e1a7e8c-80ee-4549-b261-8cd5cb1ef16e" (UID: "0e1a7e8c-80ee-4549-b261-8cd5cb1ef16e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:29.397292 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.397267 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e1a7e8c-80ee-4549-b261-8cd5cb1ef16e-kube-api-access-fg62c" pod "0e1a7e8c-80ee-4549-b261-8cd5cb1ef16e" (UID: "0e1a7e8c-80ee-4549-b261-8cd5cb1ef16e"). InnerVolumeSpecName "kube-api-access-fg62c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:29.496887 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.496843 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e1a7e8c-80ee-4549-b261-8cd5cb1ef16e-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:18:29.496887 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.496887 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg62c\" (UniqueName: \"kubernetes.io/projected/0e1a7e8c-80ee-4549-b261-8cd5cb1ef16e-kube-api-access-fg62c\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:18:29.618990 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.618965 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2"] Oct 30 18:18:29.624018 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.623985 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-4b709d-openshift-default-54cc45486f-rqjh2"] Oct 30 18:18:29.628091 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.628063 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f15587-7cccff6f5-89g5c" Oct 30 18:18:29.699744 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.699704 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05f66220-a64e-4c45-866f-caa9b332e3ec-kube-api-access-lj4pq\" (UniqueName: \"kubernetes.io/projected/05f66220-a64e-4c45-866f-caa9b332e3ec-kube-api-access-lj4pq\") pod \"05f66220-a64e-4c45-866f-caa9b332e3ec\" (UID: \"05f66220-a64e-4c45-866f-caa9b332e3ec\") " Oct 30 18:18:29.699905 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.699850 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05f66220-a64e-4c45-866f-caa9b332e3ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05f66220-a64e-4c45-866f-caa9b332e3ec-plugins-config-volume\") pod \"05f66220-a64e-4c45-866f-caa9b332e3ec\" (UID: \"05f66220-a64e-4c45-866f-caa9b332e3ec\") " Oct 30 18:18:29.700333 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.700303 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05f66220-a64e-4c45-866f-caa9b332e3ec-plugins-config-volume" pod "05f66220-a64e-4c45-866f-caa9b332e3ec" (UID: "05f66220-a64e-4c45-866f-caa9b332e3ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:29.700569 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.700551 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05f66220-a64e-4c45-866f-caa9b332e3ec-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:18:29.702902 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.702789 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f66220-a64e-4c45-866f-caa9b332e3ec-kube-api-access-lj4pq" pod "05f66220-a64e-4c45-866f-caa9b332e3ec" (UID: "05f66220-a64e-4c45-866f-caa9b332e3ec"). InnerVolumeSpecName "kube-api-access-lj4pq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:29.801551 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:29.801475 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj4pq\" (UniqueName: \"kubernetes.io/projected/05f66220-a64e-4c45-866f-caa9b332e3ec-kube-api-access-lj4pq\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:18:30.276979 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:30.276945 4442 generic.go:356] "Generic (PLEG): container finished" podID="05f66220-a64e-4c45-866f-caa9b332e3ec" containerID="0b9239a9d00486822b985281a34bb607f6667587fc251ed8beb02bf15800bbce" exitCode=0 Oct 30 18:18:30.277420 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:30.276995 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f15587-7cccff6f5-89g5c" Oct 30 18:18:30.277420 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:30.277030 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f15587-7cccff6f5-89g5c" event={"ID":"05f66220-a64e-4c45-866f-caa9b332e3ec","Type":"ContainerDied","Data":"0b9239a9d00486822b985281a34bb607f6667587fc251ed8beb02bf15800bbce"} Oct 30 18:18:30.277420 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:30.277082 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f15587-7cccff6f5-89g5c" event={"ID":"05f66220-a64e-4c45-866f-caa9b332e3ec","Type":"ContainerDied","Data":"39ad3a4463dc766c9631d26624276e0639d70f05ef706c421946345a97781cd0"} Oct 30 18:18:30.277420 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:30.277105 4442 scope.go:122] "RemoveContainer" containerID="0b9239a9d00486822b985281a34bb607f6667587fc251ed8beb02bf15800bbce" Oct 30 18:18:30.278983 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:30.278949 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f15587-674b5c966-d8pd8" Oct 30 18:18:30.279098 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:30.278979 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f15587-674b5c966-d8pd8" event={"ID":"0e1a7e8c-80ee-4549-b261-8cd5cb1ef16e","Type":"ContainerDied","Data":"72c817689444b532df2c0e1dc3bccbc4e8e84f9d822d263ea964af9a4c64f95c"} Oct 30 18:18:30.289567 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:30.289551 4442 scope.go:122] "RemoveContainer" containerID="0b9239a9d00486822b985281a34bb607f6667587fc251ed8beb02bf15800bbce" Oct 30 18:18:30.289857 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:18:30.289832 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b9239a9d00486822b985281a34bb607f6667587fc251ed8beb02bf15800bbce\": container with ID starting with 0b9239a9d00486822b985281a34bb607f6667587fc251ed8beb02bf15800bbce not found: ID does not exist" containerID="0b9239a9d00486822b985281a34bb607f6667587fc251ed8beb02bf15800bbce" Oct 30 18:18:30.289978 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:30.289863 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b9239a9d00486822b985281a34bb607f6667587fc251ed8beb02bf15800bbce"} err="failed to get container status \"0b9239a9d00486822b985281a34bb607f6667587fc251ed8beb02bf15800bbce\": rpc error: code = NotFound desc = could not find container \"0b9239a9d00486822b985281a34bb607f6667587fc251ed8beb02bf15800bbce\": container with ID starting with 0b9239a9d00486822b985281a34bb607f6667587fc251ed8beb02bf15800bbce not found: ID does not exist" Oct 30 18:18:30.289978 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:30.289883 4442 scope.go:122] "RemoveContainer" containerID="e4b48df9554f4a6e40c8853844b41e2979b1db08063f3196db5d097854f9f699" Oct 30 18:18:30.344445 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:30.344412 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f15587-7cccff6f5-89g5c"] Oct 30 18:18:30.354100 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:30.354069 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f15587-7cccff6f5-89g5c"] Oct 30 18:18:30.400786 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:30.400759 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f15587-674b5c966-d8pd8"] Oct 30 18:18:30.404970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:30.404945 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f15587-674b5c966-d8pd8"] Oct 30 18:18:31.067206 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:31.067176 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="059512a2-ce79-4426-bd6d-b11fede6864f" path="/var/lib/kubelet/pods/059512a2-ce79-4426-bd6d-b11fede6864f/volumes" Oct 30 18:18:31.067674 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:31.067660 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05f66220-a64e-4c45-866f-caa9b332e3ec" path="/var/lib/kubelet/pods/05f66220-a64e-4c45-866f-caa9b332e3ec/volumes" Oct 30 18:18:31.068031 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:31.068017 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e1a7e8c-80ee-4549-b261-8cd5cb1ef16e" path="/var/lib/kubelet/pods/0e1a7e8c-80ee-4549-b261-8cd5cb1ef16e/volumes" Oct 30 18:18:34.841449 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:34.841371 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b33a49-6d58bbc64d-49fnt"] Oct 30 18:18:34.875412 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:34.875375 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b33a49-6f47646bf9-lpsfd"] Oct 30 18:18:35.657504 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:35.649045 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-2zwb7"] Oct 30 18:18:35.701158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:35.700730 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-5jbdk"] Oct 30 18:18:39.841883 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:39.841812 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-b33a49-6d58bbc64d-49fnt" podUID="7f87d5c3-1940-499d-b425-0a7ea416d250" containerName="payload-processing" containerID="cri-o://9885972e104793d4a4807b934cb21ef2ae4b56985dd100df1f60e0c8f5920e2f" gracePeriod=25 Oct 30 18:18:39.876354 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:39.876314 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b33a49-6f47646bf9-lpsfd" podUID="092c23c3-af50-48da-965a-8890dbe71a58" containerName="payload-pre-processing" containerID="cri-o://3725fcb89f4742c0def9daa8dd7257f7e77bd67e9672e1af68e658e070eeb3f2" gracePeriod=25 Oct 30 18:18:40.192908 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:40.192879 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b33a49-6d58bbc64d-49fnt" Oct 30 18:18:40.227578 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:40.227549 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b33a49-6f47646bf9-lpsfd" Oct 30 18:18:40.279093 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:40.279064 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/092c23c3-af50-48da-965a-8890dbe71a58-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/092c23c3-af50-48da-965a-8890dbe71a58-plugins-config-volume\") pod \"092c23c3-af50-48da-965a-8890dbe71a58\" (UID: \"092c23c3-af50-48da-965a-8890dbe71a58\") " Oct 30 18:18:40.279306 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:40.279287 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f87d5c3-1940-499d-b425-0a7ea416d250-kube-api-access-tpv2z\" (UniqueName: \"kubernetes.io/projected/7f87d5c3-1940-499d-b425-0a7ea416d250-kube-api-access-tpv2z\") pod \"7f87d5c3-1940-499d-b425-0a7ea416d250\" (UID: \"7f87d5c3-1940-499d-b425-0a7ea416d250\") " Oct 30 18:18:40.279384 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:40.279371 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/092c23c3-af50-48da-965a-8890dbe71a58-kube-api-access-lkgv6\" (UniqueName: \"kubernetes.io/projected/092c23c3-af50-48da-965a-8890dbe71a58-kube-api-access-lkgv6\") pod \"092c23c3-af50-48da-965a-8890dbe71a58\" (UID: \"092c23c3-af50-48da-965a-8890dbe71a58\") " Oct 30 18:18:40.279477 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:40.279463 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f87d5c3-1940-499d-b425-0a7ea416d250-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f87d5c3-1940-499d-b425-0a7ea416d250-plugins-config-volume\") pod \"7f87d5c3-1940-499d-b425-0a7ea416d250\" (UID: \"7f87d5c3-1940-499d-b425-0a7ea416d250\") " Oct 30 18:18:40.279554 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:40.279478 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/092c23c3-af50-48da-965a-8890dbe71a58-plugins-config-volume" pod "092c23c3-af50-48da-965a-8890dbe71a58" (UID: "092c23c3-af50-48da-965a-8890dbe71a58"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:40.279888 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:40.279860 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f87d5c3-1940-499d-b425-0a7ea416d250-plugins-config-volume" pod "7f87d5c3-1940-499d-b425-0a7ea416d250" (UID: "7f87d5c3-1940-499d-b425-0a7ea416d250"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:40.280210 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:40.280182 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f87d5c3-1940-499d-b425-0a7ea416d250-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:18:40.280210 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:40.280210 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/092c23c3-af50-48da-965a-8890dbe71a58-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:18:40.282115 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:40.282085 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f87d5c3-1940-499d-b425-0a7ea416d250-kube-api-access-tpv2z" pod "7f87d5c3-1940-499d-b425-0a7ea416d250" (UID: "7f87d5c3-1940-499d-b425-0a7ea416d250"). InnerVolumeSpecName "kube-api-access-tpv2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:40.282472 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:40.282451 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/092c23c3-af50-48da-965a-8890dbe71a58-kube-api-access-lkgv6" pod "092c23c3-af50-48da-965a-8890dbe71a58" (UID: "092c23c3-af50-48da-965a-8890dbe71a58"). InnerVolumeSpecName "kube-api-access-lkgv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:40.382317 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:40.382230 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpv2z\" (UniqueName: \"kubernetes.io/projected/7f87d5c3-1940-499d-b425-0a7ea416d250-kube-api-access-tpv2z\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:18:40.382317 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:40.382263 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkgv6\" (UniqueName: \"kubernetes.io/projected/092c23c3-af50-48da-965a-8890dbe71a58-kube-api-access-lkgv6\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:18:40.469242 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:40.469211 4442 generic.go:356] "Generic (PLEG): container finished" podID="7f87d5c3-1940-499d-b425-0a7ea416d250" containerID="9885972e104793d4a4807b934cb21ef2ae4b56985dd100df1f60e0c8f5920e2f" exitCode=0 Oct 30 18:18:40.469433 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:40.469261 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b33a49-6d58bbc64d-49fnt" Oct 30 18:18:40.469433 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:40.469297 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b33a49-6d58bbc64d-49fnt" event={"ID":"7f87d5c3-1940-499d-b425-0a7ea416d250","Type":"ContainerDied","Data":"9885972e104793d4a4807b934cb21ef2ae4b56985dd100df1f60e0c8f5920e2f"} Oct 30 18:18:40.469433 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:40.469363 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b33a49-6d58bbc64d-49fnt" event={"ID":"7f87d5c3-1940-499d-b425-0a7ea416d250","Type":"ContainerDied","Data":"bdbac0a378351db034f27efedfeffd66e3be3a34f8ec60b325487b0a7b3e7076"} Oct 30 18:18:40.469433 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:40.469392 4442 scope.go:122] "RemoveContainer" containerID="9885972e104793d4a4807b934cb21ef2ae4b56985dd100df1f60e0c8f5920e2f" Oct 30 18:18:40.471299 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:40.471272 4442 generic.go:356] "Generic (PLEG): container finished" podID="092c23c3-af50-48da-965a-8890dbe71a58" containerID="3725fcb89f4742c0def9daa8dd7257f7e77bd67e9672e1af68e658e070eeb3f2" exitCode=0 Oct 30 18:18:40.471416 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:40.471321 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b33a49-6f47646bf9-lpsfd" Oct 30 18:18:40.471416 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:40.471358 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b33a49-6f47646bf9-lpsfd" event={"ID":"092c23c3-af50-48da-965a-8890dbe71a58","Type":"ContainerDied","Data":"3725fcb89f4742c0def9daa8dd7257f7e77bd67e9672e1af68e658e070eeb3f2"} Oct 30 18:18:40.471416 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:40.471406 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b33a49-6f47646bf9-lpsfd" event={"ID":"092c23c3-af50-48da-965a-8890dbe71a58","Type":"ContainerDied","Data":"2333cd74c834ee924e6f44182ead073c774cd5b10d0b21e55dc3162cdb7c82c5"} Oct 30 18:18:40.491540 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:40.491514 4442 scope.go:122] "RemoveContainer" containerID="9885972e104793d4a4807b934cb21ef2ae4b56985dd100df1f60e0c8f5920e2f" Oct 30 18:18:40.491883 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:18:40.491863 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9885972e104793d4a4807b934cb21ef2ae4b56985dd100df1f60e0c8f5920e2f\": container with ID starting with 9885972e104793d4a4807b934cb21ef2ae4b56985dd100df1f60e0c8f5920e2f not found: ID does not exist" containerID="9885972e104793d4a4807b934cb21ef2ae4b56985dd100df1f60e0c8f5920e2f" Oct 30 18:18:40.491942 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:40.491890 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9885972e104793d4a4807b934cb21ef2ae4b56985dd100df1f60e0c8f5920e2f"} err="failed to get container status \"9885972e104793d4a4807b934cb21ef2ae4b56985dd100df1f60e0c8f5920e2f\": rpc error: code = NotFound desc = could not find container \"9885972e104793d4a4807b934cb21ef2ae4b56985dd100df1f60e0c8f5920e2f\": container with ID starting with 9885972e104793d4a4807b934cb21ef2ae4b56985dd100df1f60e0c8f5920e2f not found: ID does not exist" Oct 30 18:18:40.491942 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:40.491907 4442 scope.go:122] "RemoveContainer" containerID="3725fcb89f4742c0def9daa8dd7257f7e77bd67e9672e1af68e658e070eeb3f2" Oct 30 18:18:40.504731 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:40.504695 4442 scope.go:122] "RemoveContainer" containerID="3725fcb89f4742c0def9daa8dd7257f7e77bd67e9672e1af68e658e070eeb3f2" Oct 30 18:18:40.505027 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:18:40.505004 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3725fcb89f4742c0def9daa8dd7257f7e77bd67e9672e1af68e658e070eeb3f2\": container with ID starting with 3725fcb89f4742c0def9daa8dd7257f7e77bd67e9672e1af68e658e070eeb3f2 not found: ID does not exist" containerID="3725fcb89f4742c0def9daa8dd7257f7e77bd67e9672e1af68e658e070eeb3f2" Oct 30 18:18:40.505106 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:40.505041 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3725fcb89f4742c0def9daa8dd7257f7e77bd67e9672e1af68e658e070eeb3f2"} err="failed to get container status \"3725fcb89f4742c0def9daa8dd7257f7e77bd67e9672e1af68e658e070eeb3f2\": rpc error: code = NotFound desc = could not find container \"3725fcb89f4742c0def9daa8dd7257f7e77bd67e9672e1af68e658e070eeb3f2\": container with ID starting with 3725fcb89f4742c0def9daa8dd7257f7e77bd67e9672e1af68e658e070eeb3f2 not found: ID does not exist" Oct 30 18:18:40.535055 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:40.535027 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b33a49-6d58bbc64d-49fnt"] Oct 30 18:18:40.540527 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:40.540502 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b33a49-6d58bbc64d-49fnt"] Oct 30 18:18:40.572515 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:40.572488 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b33a49-6f47646bf9-lpsfd"] Oct 30 18:18:40.576633 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:40.576607 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b33a49-6f47646bf9-lpsfd"] Oct 30 18:18:40.649668 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:40.649580 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-2zwb7" podUID="e938b627-f318-4c80-abd3-8b4634431ec1" containerName="payload-processing" containerID="cri-o://37a9a36934e6c724131ecc40decf8dd03833ed45270d1d99cf84dff06210f3d6" gracePeriod=25 Oct 30 18:18:40.701994 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:40.701947 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-5jbdk" podUID="5b5cc5ed-a8f3-434e-ae4b-57217ad6b320" containerName="payload-pre-processing" containerID="cri-o://8bd8ae9df8b5006b2e67d5c9d6eb109835d0b60aa777fec9ac71ff8a826c7093" gracePeriod=25 Oct 30 18:18:40.719565 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:40.719529 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f15587-55b68687b-xfwqd"] Oct 30 18:18:40.719834 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:40.719800 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f15587-55b68687b-xfwqd" podUID="907f83bb-f7b8-4074-9abc-e7ce23e1b665" containerName="maas-api" containerID="cri-o://a2eaa882f313b5a1ebc21b041d4af990ac83619e7411c7931ec7d84073174d95" gracePeriod=30 Oct 30 18:18:41.140493 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:41.137979 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="092c23c3-af50-48da-965a-8890dbe71a58" path="/var/lib/kubelet/pods/092c23c3-af50-48da-965a-8890dbe71a58/volumes" Oct 30 18:18:41.140493 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:41.138772 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f87d5c3-1940-499d-b425-0a7ea416d250" path="/var/lib/kubelet/pods/7f87d5c3-1940-499d-b425-0a7ea416d250/volumes" Oct 30 18:18:41.287203 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:41.286399 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f15587-55b68687b-xfwqd" Oct 30 18:18:41.567915 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:41.567799 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/907f83bb-f7b8-4074-9abc-e7ce23e1b665-kube-api-access-mjn5m\" (UniqueName: \"kubernetes.io/projected/907f83bb-f7b8-4074-9abc-e7ce23e1b665-kube-api-access-mjn5m\") pod \"907f83bb-f7b8-4074-9abc-e7ce23e1b665\" (UID: \"907f83bb-f7b8-4074-9abc-e7ce23e1b665\") " Oct 30 18:18:41.571321 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:41.571294 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/907f83bb-f7b8-4074-9abc-e7ce23e1b665-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/907f83bb-f7b8-4074-9abc-e7ce23e1b665-maas-api-tls\") pod \"907f83bb-f7b8-4074-9abc-e7ce23e1b665\" (UID: \"907f83bb-f7b8-4074-9abc-e7ce23e1b665\") " Oct 30 18:18:41.571761 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:41.571718 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/907f83bb-f7b8-4074-9abc-e7ce23e1b665-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/907f83bb-f7b8-4074-9abc-e7ce23e1b665-openshift-service-ca\") pod \"907f83bb-f7b8-4074-9abc-e7ce23e1b665\" (UID: \"907f83bb-f7b8-4074-9abc-e7ce23e1b665\") " Oct 30 18:18:41.572006 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:41.571980 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/907f83bb-f7b8-4074-9abc-e7ce23e1b665-metrics-certs\" (UniqueName: \"kubernetes.io/secret/907f83bb-f7b8-4074-9abc-e7ce23e1b665-metrics-certs\") pod \"907f83bb-f7b8-4074-9abc-e7ce23e1b665\" (UID: \"907f83bb-f7b8-4074-9abc-e7ce23e1b665\") " Oct 30 18:18:41.578188 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:41.578115 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/907f83bb-f7b8-4074-9abc-e7ce23e1b665-openshift-service-ca" pod "907f83bb-f7b8-4074-9abc-e7ce23e1b665" (UID: "907f83bb-f7b8-4074-9abc-e7ce23e1b665"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:41.578389 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:41.578328 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/907f83bb-f7b8-4074-9abc-e7ce23e1b665-maas-api-tls" pod "907f83bb-f7b8-4074-9abc-e7ce23e1b665" (UID: "907f83bb-f7b8-4074-9abc-e7ce23e1b665"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:41.599222 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:41.599160 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/907f83bb-f7b8-4074-9abc-e7ce23e1b665-kube-api-access-mjn5m" pod "907f83bb-f7b8-4074-9abc-e7ce23e1b665" (UID: "907f83bb-f7b8-4074-9abc-e7ce23e1b665"). InnerVolumeSpecName "kube-api-access-mjn5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:41.606918 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:41.606881 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/907f83bb-f7b8-4074-9abc-e7ce23e1b665-metrics-certs" pod "907f83bb-f7b8-4074-9abc-e7ce23e1b665" (UID: "907f83bb-f7b8-4074-9abc-e7ce23e1b665"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:41.631163 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:41.628349 4442 generic.go:356] "Generic (PLEG): container finished" podID="907f83bb-f7b8-4074-9abc-e7ce23e1b665" containerID="a2eaa882f313b5a1ebc21b041d4af990ac83619e7411c7931ec7d84073174d95" exitCode=0 Oct 30 18:18:41.631163 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:41.628471 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f15587-55b68687b-xfwqd" event={"ID":"907f83bb-f7b8-4074-9abc-e7ce23e1b665","Type":"ContainerDied","Data":"a2eaa882f313b5a1ebc21b041d4af990ac83619e7411c7931ec7d84073174d95"} Oct 30 18:18:41.631163 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:41.628517 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f15587-55b68687b-xfwqd" event={"ID":"907f83bb-f7b8-4074-9abc-e7ce23e1b665","Type":"ContainerDied","Data":"e1636e53397276b5898d4be45683223d3e35b32f04eefbb9d6325af5572ef0c1"} Oct 30 18:18:41.631163 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:41.628541 4442 scope.go:122] "RemoveContainer" containerID="a2eaa882f313b5a1ebc21b041d4af990ac83619e7411c7931ec7d84073174d95" Oct 30 18:18:41.631163 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:41.628648 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f15587-55b68687b-xfwqd" Oct 30 18:18:41.653954 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:41.653925 4442 scope.go:122] "RemoveContainer" containerID="a2eaa882f313b5a1ebc21b041d4af990ac83619e7411c7931ec7d84073174d95" Oct 30 18:18:41.654374 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:18:41.654350 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2eaa882f313b5a1ebc21b041d4af990ac83619e7411c7931ec7d84073174d95\": container with ID starting with a2eaa882f313b5a1ebc21b041d4af990ac83619e7411c7931ec7d84073174d95 not found: ID does not exist" containerID="a2eaa882f313b5a1ebc21b041d4af990ac83619e7411c7931ec7d84073174d95" Oct 30 18:18:41.654479 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:41.654382 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2eaa882f313b5a1ebc21b041d4af990ac83619e7411c7931ec7d84073174d95"} err="failed to get container status \"a2eaa882f313b5a1ebc21b041d4af990ac83619e7411c7931ec7d84073174d95\": rpc error: code = NotFound desc = could not find container \"a2eaa882f313b5a1ebc21b041d4af990ac83619e7411c7931ec7d84073174d95\": container with ID starting with a2eaa882f313b5a1ebc21b041d4af990ac83619e7411c7931ec7d84073174d95 not found: ID does not exist" Oct 30 18:18:41.674094 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:41.674064 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjn5m\" (UniqueName: \"kubernetes.io/projected/907f83bb-f7b8-4074-9abc-e7ce23e1b665-kube-api-access-mjn5m\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:18:41.674238 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:41.674100 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/907f83bb-f7b8-4074-9abc-e7ce23e1b665-maas-api-tls\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:18:41.674238 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:41.674112 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/907f83bb-f7b8-4074-9abc-e7ce23e1b665-openshift-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:18:41.674238 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:41.674120 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/907f83bb-f7b8-4074-9abc-e7ce23e1b665-metrics-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:18:41.691405 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:41.691379 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f15587-55b68687b-xfwqd"] Oct 30 18:18:41.702892 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:41.702851 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f15587-55b68687b-xfwqd"] Oct 30 18:18:43.074290 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:43.074248 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="907f83bb-f7b8-4074-9abc-e7ce23e1b665" path="/var/lib/kubelet/pods/907f83bb-f7b8-4074-9abc-e7ce23e1b665/volumes" Oct 30 18:18:44.338996 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:44.338954 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-5jbdk" podUID="5b5cc5ed-a8f3-434e-ae4b-57217ad6b320" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:18:44.339478 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:44.338956 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-2zwb7" podUID="e938b627-f318-4c80-abd3-8b4634431ec1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:18:47.562519 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:18:47.562480 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:18:54.337606 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:54.337563 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-2zwb7" podUID="e938b627-f318-4c80-abd3-8b4634431ec1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:18:54.337606 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:54.337585 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-5jbdk" podUID="5b5cc5ed-a8f3-434e-ae4b-57217ad6b320" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:18:57.604160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:57.601511 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-df2cs"] Oct 30 18:18:57.604160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:57.601760 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-df2cs" podUID="521d2b6a-4cee-4ee9-9eab-14f864927808" containerName="maas-api" containerID="cri-o://d28900c8301e3f6262b54f065a73638dc25948f6020c2a542da359710897fc2f" gracePeriod=30 Oct 30 18:18:59.165933 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:59.165898 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-df2cs" Oct 30 18:18:59.191954 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:59.191910 4442 generic.go:356] "Generic (PLEG): container finished" podID="521d2b6a-4cee-4ee9-9eab-14f864927808" containerID="d28900c8301e3f6262b54f065a73638dc25948f6020c2a542da359710897fc2f" exitCode=0 Oct 30 18:18:59.192173 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:59.192023 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-df2cs" Oct 30 18:18:59.192173 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:59.192045 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-df2cs" event={"ID":"521d2b6a-4cee-4ee9-9eab-14f864927808","Type":"ContainerDied","Data":"d28900c8301e3f6262b54f065a73638dc25948f6020c2a542da359710897fc2f"} Oct 30 18:18:59.192173 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:59.192110 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-df2cs" event={"ID":"521d2b6a-4cee-4ee9-9eab-14f864927808","Type":"ContainerDied","Data":"f737a4fbad76ee123dfeebfb25102d2542012d445d306070d67d05f0d57e4ec0"} Oct 30 18:18:59.192304 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:59.192126 4442 scope.go:122] "RemoveContainer" containerID="d28900c8301e3f6262b54f065a73638dc25948f6020c2a542da359710897fc2f" Oct 30 18:18:59.205248 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:59.205223 4442 scope.go:122] "RemoveContainer" containerID="d28900c8301e3f6262b54f065a73638dc25948f6020c2a542da359710897fc2f" Oct 30 18:18:59.205952 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:18:59.205644 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d28900c8301e3f6262b54f065a73638dc25948f6020c2a542da359710897fc2f\": container with ID starting with d28900c8301e3f6262b54f065a73638dc25948f6020c2a542da359710897fc2f not found: ID does not exist" containerID="d28900c8301e3f6262b54f065a73638dc25948f6020c2a542da359710897fc2f" Oct 30 18:18:59.205952 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:59.205683 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d28900c8301e3f6262b54f065a73638dc25948f6020c2a542da359710897fc2f"} err="failed to get container status \"d28900c8301e3f6262b54f065a73638dc25948f6020c2a542da359710897fc2f\": rpc error: code = NotFound desc = could not find container \"d28900c8301e3f6262b54f065a73638dc25948f6020c2a542da359710897fc2f\": container with ID starting with d28900c8301e3f6262b54f065a73638dc25948f6020c2a542da359710897fc2f not found: ID does not exist" Oct 30 18:18:59.319879 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:59.319758 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/521d2b6a-4cee-4ee9-9eab-14f864927808-metrics-certs\" (UniqueName: \"kubernetes.io/secret/521d2b6a-4cee-4ee9-9eab-14f864927808-metrics-certs\") pod \"521d2b6a-4cee-4ee9-9eab-14f864927808\" (UID: \"521d2b6a-4cee-4ee9-9eab-14f864927808\") " Oct 30 18:18:59.320046 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:59.319883 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/521d2b6a-4cee-4ee9-9eab-14f864927808-kube-api-access-h5cjl\" (UniqueName: \"kubernetes.io/projected/521d2b6a-4cee-4ee9-9eab-14f864927808-kube-api-access-h5cjl\") pod \"521d2b6a-4cee-4ee9-9eab-14f864927808\" (UID: \"521d2b6a-4cee-4ee9-9eab-14f864927808\") " Oct 30 18:18:59.320046 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:59.319948 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/521d2b6a-4cee-4ee9-9eab-14f864927808-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/521d2b6a-4cee-4ee9-9eab-14f864927808-openshift-service-ca\") pod \"521d2b6a-4cee-4ee9-9eab-14f864927808\" (UID: \"521d2b6a-4cee-4ee9-9eab-14f864927808\") " Oct 30 18:18:59.320046 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:59.320035 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/521d2b6a-4cee-4ee9-9eab-14f864927808-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/521d2b6a-4cee-4ee9-9eab-14f864927808-maas-api-tls\") pod \"521d2b6a-4cee-4ee9-9eab-14f864927808\" (UID: \"521d2b6a-4cee-4ee9-9eab-14f864927808\") " Oct 30 18:18:59.322009 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:59.321977 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/521d2b6a-4cee-4ee9-9eab-14f864927808-openshift-service-ca" pod "521d2b6a-4cee-4ee9-9eab-14f864927808" (UID: "521d2b6a-4cee-4ee9-9eab-14f864927808"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:59.338163 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:59.331460 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/521d2b6a-4cee-4ee9-9eab-14f864927808-metrics-certs" pod "521d2b6a-4cee-4ee9-9eab-14f864927808" (UID: "521d2b6a-4cee-4ee9-9eab-14f864927808"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:59.345379 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:59.332970 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/521d2b6a-4cee-4ee9-9eab-14f864927808-maas-api-tls" pod "521d2b6a-4cee-4ee9-9eab-14f864927808" (UID: "521d2b6a-4cee-4ee9-9eab-14f864927808"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:59.354312 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:59.354274 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/521d2b6a-4cee-4ee9-9eab-14f864927808-kube-api-access-h5cjl" pod "521d2b6a-4cee-4ee9-9eab-14f864927808" (UID: "521d2b6a-4cee-4ee9-9eab-14f864927808"). InnerVolumeSpecName "kube-api-access-h5cjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:59.422761 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:59.422730 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5cjl\" (UniqueName: \"kubernetes.io/projected/521d2b6a-4cee-4ee9-9eab-14f864927808-kube-api-access-h5cjl\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:18:59.422761 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:59.422759 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/521d2b6a-4cee-4ee9-9eab-14f864927808-openshift-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:18:59.422761 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:59.422772 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/521d2b6a-4cee-4ee9-9eab-14f864927808-maas-api-tls\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:18:59.423076 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:59.422780 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/521d2b6a-4cee-4ee9-9eab-14f864927808-metrics-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:18:59.542089 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:59.542052 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-df2cs"] Oct 30 18:18:59.548523 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:59.548477 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-df2cs"] Oct 30 18:18:59.683085 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:18:59.683053 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8dc43c-5bc899bbc9-m89jr"] Oct 30 18:19:01.072658 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:01.072042 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="521d2b6a-4cee-4ee9-9eab-14f864927808" path="/var/lib/kubelet/pods/521d2b6a-4cee-4ee9-9eab-14f864927808/volumes" Oct 30 18:19:01.140016 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:01.139987 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-2zwb7" Oct 30 18:19:01.187534 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:01.187474 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-5jbdk" Oct 30 18:19:01.211752 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:01.211705 4442 generic.go:356] "Generic (PLEG): container finished" podID="5b5cc5ed-a8f3-434e-ae4b-57217ad6b320" containerID="8bd8ae9df8b5006b2e67d5c9d6eb109835d0b60aa777fec9ac71ff8a826c7093" exitCode=0 Oct 30 18:19:01.211938 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:01.211762 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-5jbdk" Oct 30 18:19:01.211938 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:01.211769 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-5jbdk" event={"ID":"5b5cc5ed-a8f3-434e-ae4b-57217ad6b320","Type":"ContainerDied","Data":"8bd8ae9df8b5006b2e67d5c9d6eb109835d0b60aa777fec9ac71ff8a826c7093"} Oct 30 18:19:01.211938 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:01.211832 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-5jbdk" event={"ID":"5b5cc5ed-a8f3-434e-ae4b-57217ad6b320","Type":"ContainerDied","Data":"db50ddefcd21b0b5b6e1361e465de45e58e33f70ab69458c24162ea37995d959"} Oct 30 18:19:01.211938 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:01.211862 4442 scope.go:122] "RemoveContainer" containerID="8bd8ae9df8b5006b2e67d5c9d6eb109835d0b60aa777fec9ac71ff8a826c7093" Oct 30 18:19:01.213899 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:01.213829 4442 generic.go:356] "Generic (PLEG): container finished" podID="e938b627-f318-4c80-abd3-8b4634431ec1" containerID="37a9a36934e6c724131ecc40decf8dd03833ed45270d1d99cf84dff06210f3d6" exitCode=0 Oct 30 18:19:01.213899 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:01.213864 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-2zwb7" Oct 30 18:19:01.213899 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:01.213907 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-2zwb7" event={"ID":"e938b627-f318-4c80-abd3-8b4634431ec1","Type":"ContainerDied","Data":"37a9a36934e6c724131ecc40decf8dd03833ed45270d1d99cf84dff06210f3d6"} Oct 30 18:19:01.214185 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:01.213948 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-2zwb7" event={"ID":"e938b627-f318-4c80-abd3-8b4634431ec1","Type":"ContainerDied","Data":"33cdacf131f0308600aafce376dd8b7042fbb74e6706b486f8b2c986dea89393"} Oct 30 18:19:01.223978 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:01.223934 4442 scope.go:122] "RemoveContainer" containerID="8bd8ae9df8b5006b2e67d5c9d6eb109835d0b60aa777fec9ac71ff8a826c7093" Oct 30 18:19:01.224327 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:19:01.224304 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bd8ae9df8b5006b2e67d5c9d6eb109835d0b60aa777fec9ac71ff8a826c7093\": container with ID starting with 8bd8ae9df8b5006b2e67d5c9d6eb109835d0b60aa777fec9ac71ff8a826c7093 not found: ID does not exist" containerID="8bd8ae9df8b5006b2e67d5c9d6eb109835d0b60aa777fec9ac71ff8a826c7093" Oct 30 18:19:01.224415 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:01.224340 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bd8ae9df8b5006b2e67d5c9d6eb109835d0b60aa777fec9ac71ff8a826c7093"} err="failed to get container status \"8bd8ae9df8b5006b2e67d5c9d6eb109835d0b60aa777fec9ac71ff8a826c7093\": rpc error: code = NotFound desc = could not find container \"8bd8ae9df8b5006b2e67d5c9d6eb109835d0b60aa777fec9ac71ff8a826c7093\": container with ID starting with 8bd8ae9df8b5006b2e67d5c9d6eb109835d0b60aa777fec9ac71ff8a826c7093 not found: ID does not exist" Oct 30 18:19:01.224415 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:01.224365 4442 scope.go:122] "RemoveContainer" containerID="37a9a36934e6c724131ecc40decf8dd03833ed45270d1d99cf84dff06210f3d6" Oct 30 18:19:01.238573 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:01.238541 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e938b627-f318-4c80-abd3-8b4634431ec1-kube-api-access-pdf9w\" (UniqueName: \"kubernetes.io/projected/e938b627-f318-4c80-abd3-8b4634431ec1-kube-api-access-pdf9w\") pod \"e938b627-f318-4c80-abd3-8b4634431ec1\" (UID: \"e938b627-f318-4c80-abd3-8b4634431ec1\") " Oct 30 18:19:01.238734 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:01.238619 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b5cc5ed-a8f3-434e-ae4b-57217ad6b320-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b5cc5ed-a8f3-434e-ae4b-57217ad6b320-plugins-config-volume\") pod \"5b5cc5ed-a8f3-434e-ae4b-57217ad6b320\" (UID: \"5b5cc5ed-a8f3-434e-ae4b-57217ad6b320\") " Oct 30 18:19:01.238734 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:01.238676 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b5cc5ed-a8f3-434e-ae4b-57217ad6b320-kube-api-access-dz2fs\" (UniqueName: \"kubernetes.io/projected/5b5cc5ed-a8f3-434e-ae4b-57217ad6b320-kube-api-access-dz2fs\") pod \"5b5cc5ed-a8f3-434e-ae4b-57217ad6b320\" (UID: \"5b5cc5ed-a8f3-434e-ae4b-57217ad6b320\") " Oct 30 18:19:01.238858 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:01.238754 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e938b627-f318-4c80-abd3-8b4634431ec1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e938b627-f318-4c80-abd3-8b4634431ec1-plugins-config-volume\") pod \"e938b627-f318-4c80-abd3-8b4634431ec1\" (UID: \"e938b627-f318-4c80-abd3-8b4634431ec1\") " Oct 30 18:19:01.239677 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:01.239651 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b5cc5ed-a8f3-434e-ae4b-57217ad6b320-plugins-config-volume" pod "5b5cc5ed-a8f3-434e-ae4b-57217ad6b320" (UID: "5b5cc5ed-a8f3-434e-ae4b-57217ad6b320"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:01.239842 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:01.239681 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e938b627-f318-4c80-abd3-8b4634431ec1-plugins-config-volume" pod "e938b627-f318-4c80-abd3-8b4634431ec1" (UID: "e938b627-f318-4c80-abd3-8b4634431ec1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:01.241282 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:01.241259 4442 scope.go:122] "RemoveContainer" containerID="37a9a36934e6c724131ecc40decf8dd03833ed45270d1d99cf84dff06210f3d6" Oct 30 18:19:01.241819 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:19:01.241779 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37a9a36934e6c724131ecc40decf8dd03833ed45270d1d99cf84dff06210f3d6\": container with ID starting with 37a9a36934e6c724131ecc40decf8dd03833ed45270d1d99cf84dff06210f3d6 not found: ID does not exist" containerID="37a9a36934e6c724131ecc40decf8dd03833ed45270d1d99cf84dff06210f3d6" Oct 30 18:19:01.241966 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:01.241826 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37a9a36934e6c724131ecc40decf8dd03833ed45270d1d99cf84dff06210f3d6"} err="failed to get container status \"37a9a36934e6c724131ecc40decf8dd03833ed45270d1d99cf84dff06210f3d6\": rpc error: code = NotFound desc = could not find container \"37a9a36934e6c724131ecc40decf8dd03833ed45270d1d99cf84dff06210f3d6\": container with ID starting with 37a9a36934e6c724131ecc40decf8dd03833ed45270d1d99cf84dff06210f3d6 not found: ID does not exist" Oct 30 18:19:01.242911 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:01.242885 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b5cc5ed-a8f3-434e-ae4b-57217ad6b320-kube-api-access-dz2fs" pod "5b5cc5ed-a8f3-434e-ae4b-57217ad6b320" (UID: "5b5cc5ed-a8f3-434e-ae4b-57217ad6b320"). InnerVolumeSpecName "kube-api-access-dz2fs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:01.244686 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:01.244660 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e938b627-f318-4c80-abd3-8b4634431ec1-kube-api-access-pdf9w" pod "e938b627-f318-4c80-abd3-8b4634431ec1" (UID: "e938b627-f318-4c80-abd3-8b4634431ec1"). InnerVolumeSpecName "kube-api-access-pdf9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:01.340766 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:01.340731 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdf9w\" (UniqueName: \"kubernetes.io/projected/e938b627-f318-4c80-abd3-8b4634431ec1-kube-api-access-pdf9w\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:19:01.340766 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:01.340762 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b5cc5ed-a8f3-434e-ae4b-57217ad6b320-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:19:01.340766 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:01.340773 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz2fs\" (UniqueName: \"kubernetes.io/projected/5b5cc5ed-a8f3-434e-ae4b-57217ad6b320-kube-api-access-dz2fs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:19:01.341029 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:01.340782 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e938b627-f318-4c80-abd3-8b4634431ec1-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:19:01.559657 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:01.559628 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-5jbdk"] Oct 30 18:19:01.570817 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:01.570782 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-5jbdk"] Oct 30 18:19:01.603426 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:01.603390 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-2zwb7"] Oct 30 18:19:01.607549 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:01.607524 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-2zwb7"] Oct 30 18:19:02.206418 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:02.206385 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b33a49-6b5fd95565-fvshv"] Oct 30 18:19:02.207283 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:02.207233 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b33a49-6b5fd95565-fvshv" podUID="999b71c3-39d4-4b0a-8a38-3495d0f11180" containerName="maas-api" containerID="cri-o://43d3be194f1b20eae2899669c68076ef792aecb0914162c65cafb14d662781bf" gracePeriod=30 Oct 30 18:19:02.768158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:02.761921 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b33a49-6b5fd95565-fvshv" Oct 30 18:19:02.836206 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:02.833456 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/999b71c3-39d4-4b0a-8a38-3495d0f11180-kube-api-access-nlmfm\" (UniqueName: \"kubernetes.io/projected/999b71c3-39d4-4b0a-8a38-3495d0f11180-kube-api-access-nlmfm\") pod \"999b71c3-39d4-4b0a-8a38-3495d0f11180\" (UID: \"999b71c3-39d4-4b0a-8a38-3495d0f11180\") " Oct 30 18:19:02.836206 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:02.833644 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/999b71c3-39d4-4b0a-8a38-3495d0f11180-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/999b71c3-39d4-4b0a-8a38-3495d0f11180-openshift-service-ca\") pod \"999b71c3-39d4-4b0a-8a38-3495d0f11180\" (UID: \"999b71c3-39d4-4b0a-8a38-3495d0f11180\") " Oct 30 18:19:02.836206 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:02.833703 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/999b71c3-39d4-4b0a-8a38-3495d0f11180-metrics-certs\" (UniqueName: \"kubernetes.io/secret/999b71c3-39d4-4b0a-8a38-3495d0f11180-metrics-certs\") pod \"999b71c3-39d4-4b0a-8a38-3495d0f11180\" (UID: \"999b71c3-39d4-4b0a-8a38-3495d0f11180\") " Oct 30 18:19:02.836206 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:02.833759 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/999b71c3-39d4-4b0a-8a38-3495d0f11180-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/999b71c3-39d4-4b0a-8a38-3495d0f11180-maas-api-tls\") pod \"999b71c3-39d4-4b0a-8a38-3495d0f11180\" (UID: \"999b71c3-39d4-4b0a-8a38-3495d0f11180\") " Oct 30 18:19:02.836206 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:02.835801 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/999b71c3-39d4-4b0a-8a38-3495d0f11180-openshift-service-ca" pod "999b71c3-39d4-4b0a-8a38-3495d0f11180" (UID: "999b71c3-39d4-4b0a-8a38-3495d0f11180"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:02.925254 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:02.922302 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/999b71c3-39d4-4b0a-8a38-3495d0f11180-maas-api-tls" pod "999b71c3-39d4-4b0a-8a38-3495d0f11180" (UID: "999b71c3-39d4-4b0a-8a38-3495d0f11180"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:02.925254 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:02.922395 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/999b71c3-39d4-4b0a-8a38-3495d0f11180-metrics-certs" pod "999b71c3-39d4-4b0a-8a38-3495d0f11180" (UID: "999b71c3-39d4-4b0a-8a38-3495d0f11180"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:02.925254 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:02.922487 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/999b71c3-39d4-4b0a-8a38-3495d0f11180-kube-api-access-nlmfm" pod "999b71c3-39d4-4b0a-8a38-3495d0f11180" (UID: "999b71c3-39d4-4b0a-8a38-3495d0f11180"). InnerVolumeSpecName "kube-api-access-nlmfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:02.967158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:02.962472 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/999b71c3-39d4-4b0a-8a38-3495d0f11180-openshift-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:19:02.967158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:02.962511 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/999b71c3-39d4-4b0a-8a38-3495d0f11180-metrics-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:19:02.967158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:02.962528 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/999b71c3-39d4-4b0a-8a38-3495d0f11180-maas-api-tls\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:19:02.967158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:02.962542 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlmfm\" (UniqueName: \"kubernetes.io/projected/999b71c3-39d4-4b0a-8a38-3495d0f11180-kube-api-access-nlmfm\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:19:03.248313 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:03.248232 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b5cc5ed-a8f3-434e-ae4b-57217ad6b320" path="/var/lib/kubelet/pods/5b5cc5ed-a8f3-434e-ae4b-57217ad6b320/volumes" Oct 30 18:19:03.248991 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:03.248970 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e938b627-f318-4c80-abd3-8b4634431ec1" path="/var/lib/kubelet/pods/e938b627-f318-4c80-abd3-8b4634431ec1/volumes" Oct 30 18:19:03.296526 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:03.296486 4442 generic.go:356] "Generic (PLEG): container finished" podID="999b71c3-39d4-4b0a-8a38-3495d0f11180" containerID="43d3be194f1b20eae2899669c68076ef792aecb0914162c65cafb14d662781bf" exitCode=0 Oct 30 18:19:03.308417 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:03.308371 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b33a49-6b5fd95565-fvshv" event={"ID":"999b71c3-39d4-4b0a-8a38-3495d0f11180","Type":"ContainerDied","Data":"43d3be194f1b20eae2899669c68076ef792aecb0914162c65cafb14d662781bf"} Oct 30 18:19:03.308724 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:03.308693 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b33a49-6b5fd95565-fvshv" event={"ID":"999b71c3-39d4-4b0a-8a38-3495d0f11180","Type":"ContainerDied","Data":"875b848998b87e0103655da877fdf82b10e860a345a189bc1870f420be84b28b"} Oct 30 18:19:03.309070 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:03.308937 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b33a49-6b5fd95565-fvshv" Oct 30 18:19:03.315820 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:03.308955 4442 scope.go:122] "RemoveContainer" containerID="43d3be194f1b20eae2899669c68076ef792aecb0914162c65cafb14d662781bf" Oct 30 18:19:03.373749 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:03.373318 4442 scope.go:122] "RemoveContainer" containerID="43d3be194f1b20eae2899669c68076ef792aecb0914162c65cafb14d662781bf" Oct 30 18:19:03.374194 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:19:03.374080 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43d3be194f1b20eae2899669c68076ef792aecb0914162c65cafb14d662781bf\": container with ID starting with 43d3be194f1b20eae2899669c68076ef792aecb0914162c65cafb14d662781bf not found: ID does not exist" containerID="43d3be194f1b20eae2899669c68076ef792aecb0914162c65cafb14d662781bf" Oct 30 18:19:03.374194 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:03.374123 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43d3be194f1b20eae2899669c68076ef792aecb0914162c65cafb14d662781bf"} err="failed to get container status \"43d3be194f1b20eae2899669c68076ef792aecb0914162c65cafb14d662781bf\": rpc error: code = NotFound desc = could not find container \"43d3be194f1b20eae2899669c68076ef792aecb0914162c65cafb14d662781bf\": container with ID starting with 43d3be194f1b20eae2899669c68076ef792aecb0914162c65cafb14d662781bf not found: ID does not exist" Oct 30 18:19:03.495534 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:03.495311 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b33a49-6b5fd95565-fvshv"] Oct 30 18:19:03.498830 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:03.498744 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b33a49-6b5fd95565-fvshv"] Oct 30 18:19:04.683889 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:04.683850 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8dc43c-5bc899bbc9-m89jr" podUID="5b6343c8-e782-48b4-87d3-ee8637dc8f9e" containerName="payload-pre-processing" containerID="cri-o://5ad36ced1c830af6a745f653337c9513093cb8a9317dca5cfc82bd9cda368fdc" gracePeriod=25 Oct 30 18:19:04.985730 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:04.985703 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8dc43c-5bc899bbc9-m89jr" Oct 30 18:19:05.067828 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:05.067791 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="999b71c3-39d4-4b0a-8a38-3495d0f11180" path="/var/lib/kubelet/pods/999b71c3-39d4-4b0a-8a38-3495d0f11180/volumes" Oct 30 18:19:05.107801 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:05.107766 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b6343c8-e782-48b4-87d3-ee8637dc8f9e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b6343c8-e782-48b4-87d3-ee8637dc8f9e-plugins-config-volume\") pod \"5b6343c8-e782-48b4-87d3-ee8637dc8f9e\" (UID: \"5b6343c8-e782-48b4-87d3-ee8637dc8f9e\") " Oct 30 18:19:05.107995 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:05.107835 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b6343c8-e782-48b4-87d3-ee8637dc8f9e-kube-api-access-d6w6b\" (UniqueName: \"kubernetes.io/projected/5b6343c8-e782-48b4-87d3-ee8637dc8f9e-kube-api-access-d6w6b\") pod \"5b6343c8-e782-48b4-87d3-ee8637dc8f9e\" (UID: \"5b6343c8-e782-48b4-87d3-ee8637dc8f9e\") " Oct 30 18:19:05.108310 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:05.108260 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b6343c8-e782-48b4-87d3-ee8637dc8f9e-plugins-config-volume" pod "5b6343c8-e782-48b4-87d3-ee8637dc8f9e" (UID: "5b6343c8-e782-48b4-87d3-ee8637dc8f9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:05.110254 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:05.110228 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b6343c8-e782-48b4-87d3-ee8637dc8f9e-kube-api-access-d6w6b" pod "5b6343c8-e782-48b4-87d3-ee8637dc8f9e" (UID: "5b6343c8-e782-48b4-87d3-ee8637dc8f9e"). InnerVolumeSpecName "kube-api-access-d6w6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:05.211002 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:05.210903 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6w6b\" (UniqueName: \"kubernetes.io/projected/5b6343c8-e782-48b4-87d3-ee8637dc8f9e-kube-api-access-d6w6b\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:19:05.211002 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:05.210930 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b6343c8-e782-48b4-87d3-ee8637dc8f9e-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:19:05.340082 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:05.340044 4442 generic.go:356] "Generic (PLEG): container finished" podID="5b6343c8-e782-48b4-87d3-ee8637dc8f9e" containerID="5ad36ced1c830af6a745f653337c9513093cb8a9317dca5cfc82bd9cda368fdc" exitCode=0 Oct 30 18:19:05.340302 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:05.340105 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8dc43c-5bc899bbc9-m89jr" Oct 30 18:19:05.340302 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:05.340168 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8dc43c-5bc899bbc9-m89jr" event={"ID":"5b6343c8-e782-48b4-87d3-ee8637dc8f9e","Type":"ContainerDied","Data":"5ad36ced1c830af6a745f653337c9513093cb8a9317dca5cfc82bd9cda368fdc"} Oct 30 18:19:05.340302 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:05.340242 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8dc43c-5bc899bbc9-m89jr" event={"ID":"5b6343c8-e782-48b4-87d3-ee8637dc8f9e","Type":"ContainerDied","Data":"4006f79a0467e3723d1f411fe40d25f570c9a1046d8a4756f5ed6ace145c6ab4"} Oct 30 18:19:05.340302 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:05.340269 4442 scope.go:122] "RemoveContainer" containerID="5ad36ced1c830af6a745f653337c9513093cb8a9317dca5cfc82bd9cda368fdc" Oct 30 18:19:05.351036 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:05.351016 4442 scope.go:122] "RemoveContainer" containerID="5ad36ced1c830af6a745f653337c9513093cb8a9317dca5cfc82bd9cda368fdc" Oct 30 18:19:05.351410 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:19:05.351382 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ad36ced1c830af6a745f653337c9513093cb8a9317dca5cfc82bd9cda368fdc\": container with ID starting with 5ad36ced1c830af6a745f653337c9513093cb8a9317dca5cfc82bd9cda368fdc not found: ID does not exist" containerID="5ad36ced1c830af6a745f653337c9513093cb8a9317dca5cfc82bd9cda368fdc" Oct 30 18:19:05.351484 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:05.351418 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ad36ced1c830af6a745f653337c9513093cb8a9317dca5cfc82bd9cda368fdc"} err="failed to get container status \"5ad36ced1c830af6a745f653337c9513093cb8a9317dca5cfc82bd9cda368fdc\": rpc error: code = NotFound desc = could not find container \"5ad36ced1c830af6a745f653337c9513093cb8a9317dca5cfc82bd9cda368fdc\": container with ID starting with 5ad36ced1c830af6a745f653337c9513093cb8a9317dca5cfc82bd9cda368fdc not found: ID does not exist" Oct 30 18:19:05.392610 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:05.392575 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8dc43c-5bc899bbc9-m89jr"] Oct 30 18:19:05.403188 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:05.403051 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8dc43c-5bc899bbc9-m89jr"] Oct 30 18:19:07.067298 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.067264 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b6343c8-e782-48b4-87d3-ee8637dc8f9e" path="/var/lib/kubelet/pods/5b6343c8-e782-48b4-87d3-ee8637dc8f9e/volumes" Oct 30 18:19:07.245970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.235588 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b"] Oct 30 18:19:07.245970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.236566 4442 cpu_manager.go:414] "containerMap: removing container" podUID="092c23c3-af50-48da-965a-8890dbe71a58" containerName="payload-pre-processing" Oct 30 18:19:07.245970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.236586 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="092c23c3-af50-48da-965a-8890dbe71a58" containerName="payload-pre-processing" Oct 30 18:19:07.245970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.236603 4442 cpu_manager.go:414] "containerMap: removing container" podUID="e938b627-f318-4c80-abd3-8b4634431ec1" containerName="payload-processing" Oct 30 18:19:07.245970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.236611 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e938b627-f318-4c80-abd3-8b4634431ec1" containerName="payload-processing" Oct 30 18:19:07.245970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.236630 4442 cpu_manager.go:414] "containerMap: removing container" podUID="0cc9578c-0f6a-48a4-a71c-c0a30dc75171" containerName="payload-processing" Oct 30 18:19:07.245970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.236637 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cc9578c-0f6a-48a4-a71c-c0a30dc75171" containerName="payload-processing" Oct 30 18:19:07.245970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.236649 4442 cpu_manager.go:414] "containerMap: removing container" podUID="0e1a7e8c-80ee-4549-b261-8cd5cb1ef16e" containerName="payload-processing" Oct 30 18:19:07.245970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.236655 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e1a7e8c-80ee-4549-b261-8cd5cb1ef16e" containerName="payload-processing" Oct 30 18:19:07.245970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.236666 4442 cpu_manager.go:414] "containerMap: removing container" podUID="999b71c3-39d4-4b0a-8a38-3495d0f11180" containerName="maas-api" Oct 30 18:19:07.245970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.236672 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="999b71c3-39d4-4b0a-8a38-3495d0f11180" containerName="maas-api" Oct 30 18:19:07.245970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.236683 4442 cpu_manager.go:414] "containerMap: removing container" podUID="059512a2-ce79-4426-bd6d-b11fede6864f" containerName="istio-proxy" Oct 30 18:19:07.245970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.236689 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="059512a2-ce79-4426-bd6d-b11fede6864f" containerName="istio-proxy" Oct 30 18:19:07.245970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.236699 4442 cpu_manager.go:414] "containerMap: removing container" podUID="059512a2-ce79-4426-bd6d-b11fede6864f" containerName="istio-proxy" Oct 30 18:19:07.245970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.236706 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="059512a2-ce79-4426-bd6d-b11fede6864f" containerName="istio-proxy" Oct 30 18:19:07.245970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.236724 4442 cpu_manager.go:414] "containerMap: removing container" podUID="5b6343c8-e782-48b4-87d3-ee8637dc8f9e" containerName="payload-pre-processing" Oct 30 18:19:07.245970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.236730 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b6343c8-e782-48b4-87d3-ee8637dc8f9e" containerName="payload-pre-processing" Oct 30 18:19:07.245970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.236740 4442 cpu_manager.go:414] "containerMap: removing container" podUID="6db04ea0-e110-4d71-9a90-f37b4c431a3b" containerName="maas-api" Oct 30 18:19:07.245970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.236746 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6db04ea0-e110-4d71-9a90-f37b4c431a3b" containerName="maas-api" Oct 30 18:19:07.245970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.236757 4442 cpu_manager.go:414] "containerMap: removing container" podUID="5b5cc5ed-a8f3-434e-ae4b-57217ad6b320" containerName="payload-pre-processing" Oct 30 18:19:07.245970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.236764 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b5cc5ed-a8f3-434e-ae4b-57217ad6b320" containerName="payload-pre-processing" Oct 30 18:19:07.245970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.236784 4442 cpu_manager.go:414] "containerMap: removing container" podUID="05f66220-a64e-4c45-866f-caa9b332e3ec" containerName="payload-pre-processing" Oct 30 18:19:07.245970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.236790 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05f66220-a64e-4c45-866f-caa9b332e3ec" containerName="payload-pre-processing" Oct 30 18:19:07.245970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.236803 4442 cpu_manager.go:414] "containerMap: removing container" podUID="521d2b6a-4cee-4ee9-9eab-14f864927808" containerName="maas-api" Oct 30 18:19:07.245970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.236810 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="521d2b6a-4cee-4ee9-9eab-14f864927808" containerName="maas-api" Oct 30 18:19:07.245970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.236820 4442 cpu_manager.go:414] "containerMap: removing container" podUID="7f87d5c3-1940-499d-b425-0a7ea416d250" containerName="payload-processing" Oct 30 18:19:07.245970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.236826 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f87d5c3-1940-499d-b425-0a7ea416d250" containerName="payload-processing" Oct 30 18:19:07.245970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.236837 4442 cpu_manager.go:414] "containerMap: removing container" podUID="907f83bb-f7b8-4074-9abc-e7ce23e1b665" containerName="maas-api" Oct 30 18:19:07.245970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.236843 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="907f83bb-f7b8-4074-9abc-e7ce23e1b665" containerName="maas-api" Oct 30 18:19:07.245970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.236858 4442 cpu_manager.go:414] "containerMap: removing container" podUID="361a8ea8-96e6-404d-b694-052dbb6fb070" containerName="payload-pre-processing" Oct 30 18:19:07.245970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.236865 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="361a8ea8-96e6-404d-b694-052dbb6fb070" containerName="payload-pre-processing" Oct 30 18:19:07.245970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.236997 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="05f66220-a64e-4c45-866f-caa9b332e3ec" containerName="payload-pre-processing" Oct 30 18:19:07.245970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.237007 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="521d2b6a-4cee-4ee9-9eab-14f864927808" containerName="maas-api" Oct 30 18:19:07.245970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.237015 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f87d5c3-1940-499d-b425-0a7ea416d250" containerName="payload-processing" Oct 30 18:19:07.245970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.237024 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="907f83bb-f7b8-4074-9abc-e7ce23e1b665" containerName="maas-api" Oct 30 18:19:07.245970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.237037 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="361a8ea8-96e6-404d-b694-052dbb6fb070" containerName="payload-pre-processing" Oct 30 18:19:07.245970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.237048 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="092c23c3-af50-48da-965a-8890dbe71a58" containerName="payload-pre-processing" Oct 30 18:19:07.245970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.237058 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="e938b627-f318-4c80-abd3-8b4634431ec1" containerName="payload-processing" Oct 30 18:19:07.245970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.237069 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cc9578c-0f6a-48a4-a71c-c0a30dc75171" containerName="payload-processing" Oct 30 18:19:07.245970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.237075 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e1a7e8c-80ee-4549-b261-8cd5cb1ef16e" containerName="payload-processing" Oct 30 18:19:07.245970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.237083 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="999b71c3-39d4-4b0a-8a38-3495d0f11180" containerName="maas-api" Oct 30 18:19:07.245970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.237090 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="059512a2-ce79-4426-bd6d-b11fede6864f" containerName="istio-proxy" Oct 30 18:19:07.245970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.237098 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="059512a2-ce79-4426-bd6d-b11fede6864f" containerName="istio-proxy" Oct 30 18:19:07.245970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.237105 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b6343c8-e782-48b4-87d3-ee8637dc8f9e" containerName="payload-pre-processing" Oct 30 18:19:07.245970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.237112 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="6db04ea0-e110-4d71-9a90-f37b4c431a3b" containerName="maas-api" Oct 30 18:19:07.245970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.237120 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b5cc5ed-a8f3-434e-ae4b-57217ad6b320" containerName="payload-pre-processing" Oct 30 18:19:07.253881 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.253852 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" Oct 30 18:19:07.275948 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.275428 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-2ee874-openshift-default-dockercfg-bbffq\"" Oct 30 18:19:07.307470 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.306519 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b"] Oct 30 18:19:07.368715 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.359069 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-istiod-ca-cert\") pod \"e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b\" (UID: \"64bde5b5-0bc5-42b8-9aa9-fc6495afca20\") " pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" Oct 30 18:19:07.372018 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.371840 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-istio-token\") pod \"e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b\" (UID: \"64bde5b5-0bc5-42b8-9aa9-fc6495afca20\") " pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" Oct 30 18:19:07.372388 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.372368 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-workload-socket\") pod \"e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b\" (UID: \"64bde5b5-0bc5-42b8-9aa9-fc6495afca20\") " pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" Oct 30 18:19:07.372719 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.372703 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-workload-certs\") pod \"e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b\" (UID: \"64bde5b5-0bc5-42b8-9aa9-fc6495afca20\") " pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" Oct 30 18:19:07.372877 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.372863 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-istio-data\") pod \"e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b\" (UID: \"64bde5b5-0bc5-42b8-9aa9-fc6495afca20\") " pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" Oct 30 18:19:07.373038 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.373023 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-istio-podinfo\") pod \"e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b\" (UID: \"64bde5b5-0bc5-42b8-9aa9-fc6495afca20\") " pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" Oct 30 18:19:07.373339 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.373320 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-istio-envoy\") pod \"e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b\" (UID: \"64bde5b5-0bc5-42b8-9aa9-fc6495afca20\") " pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" Oct 30 18:19:07.380266 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.380234 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vbjg\" (UniqueName: \"kubernetes.io/projected/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-kube-api-access-5vbjg\") pod \"e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b\" (UID: \"64bde5b5-0bc5-42b8-9aa9-fc6495afca20\") " pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" Oct 30 18:19:07.398200 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.390242 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-credential-socket\") pod \"e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b\" (UID: \"64bde5b5-0bc5-42b8-9aa9-fc6495afca20\") " pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" Oct 30 18:19:07.491761 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.491728 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-workload-certs\") pod \"e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b\" (UID: \"64bde5b5-0bc5-42b8-9aa9-fc6495afca20\") " pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" Oct 30 18:19:07.491935 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.491771 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-istio-data\") pod \"e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b\" (UID: \"64bde5b5-0bc5-42b8-9aa9-fc6495afca20\") " pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" Oct 30 18:19:07.491935 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.491822 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-istio-podinfo\") pod \"e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b\" (UID: \"64bde5b5-0bc5-42b8-9aa9-fc6495afca20\") " pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" Oct 30 18:19:07.491935 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.491903 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-istio-envoy\") pod \"e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b\" (UID: \"64bde5b5-0bc5-42b8-9aa9-fc6495afca20\") " pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" Oct 30 18:19:07.492101 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.491944 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vbjg\" (UniqueName: \"kubernetes.io/projected/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-kube-api-access-5vbjg\") pod \"e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b\" (UID: \"64bde5b5-0bc5-42b8-9aa9-fc6495afca20\") " pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" Oct 30 18:19:07.492101 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.491986 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-credential-socket\") pod \"e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b\" (UID: \"64bde5b5-0bc5-42b8-9aa9-fc6495afca20\") " pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" Oct 30 18:19:07.492101 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.492067 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-istiod-ca-cert\") pod \"e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b\" (UID: \"64bde5b5-0bc5-42b8-9aa9-fc6495afca20\") " pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" Oct 30 18:19:07.492101 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.492096 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-istio-token\") pod \"e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b\" (UID: \"64bde5b5-0bc5-42b8-9aa9-fc6495afca20\") " pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" Oct 30 18:19:07.492349 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.492196 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-workload-socket\") pod \"e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b\" (UID: \"64bde5b5-0bc5-42b8-9aa9-fc6495afca20\") " pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" Oct 30 18:19:07.492349 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.492228 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-workload-certs\") pod \"e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b\" (UID: \"64bde5b5-0bc5-42b8-9aa9-fc6495afca20\") " pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" Oct 30 18:19:07.492349 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.492236 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-istio-data\") pod \"e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b\" (UID: \"64bde5b5-0bc5-42b8-9aa9-fc6495afca20\") " pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" Oct 30 18:19:07.492642 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.492620 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-workload-socket\") pod \"e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b\" (UID: \"64bde5b5-0bc5-42b8-9aa9-fc6495afca20\") " pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" Oct 30 18:19:07.492758 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.492738 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-credential-socket\") pod \"e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b\" (UID: \"64bde5b5-0bc5-42b8-9aa9-fc6495afca20\") " pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" Oct 30 18:19:07.492967 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.492911 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-istiod-ca-cert\") pod \"e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b\" (UID: \"64bde5b5-0bc5-42b8-9aa9-fc6495afca20\") " pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" Oct 30 18:19:07.495478 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.495458 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-istio-podinfo\") pod \"e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b\" (UID: \"64bde5b5-0bc5-42b8-9aa9-fc6495afca20\") " pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" Oct 30 18:19:07.495939 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.495881 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-istio-envoy\") pod \"e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b\" (UID: \"64bde5b5-0bc5-42b8-9aa9-fc6495afca20\") " pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" Oct 30 18:19:07.513360 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.513331 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vbjg\" (UniqueName: \"kubernetes.io/projected/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-kube-api-access-5vbjg\") pod \"e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b\" (UID: \"64bde5b5-0bc5-42b8-9aa9-fc6495afca20\") " pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" Oct 30 18:19:07.513485 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.513429 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-istio-token\") pod \"e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b\" (UID: \"64bde5b5-0bc5-42b8-9aa9-fc6495afca20\") " pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" Oct 30 18:19:07.663326 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:07.663244 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" Oct 30 18:19:08.078187 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:08.078113 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b"] Oct 30 18:19:08.078512 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:19:08.078114 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64bde5b5_0bc5_42b8_9aa9_fc6495afca20.slice/crio-2af47bb18f9225f46d40e3f294ea4a90136127cd35a1ffa9f4a416dff0920001 WatchSource:0}: Error finding container 2af47bb18f9225f46d40e3f294ea4a90136127cd35a1ffa9f4a416dff0920001: Status 404 returned error can't find the container with id 2af47bb18f9225f46d40e3f294ea4a90136127cd35a1ffa9f4a416dff0920001 Oct 30 18:19:08.080781 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:08.080736 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:19:08.080885 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:08.080816 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:19:08.080885 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:08.080845 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:19:08.388806 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:08.388777 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" event={"ID":"64bde5b5-0bc5-42b8-9aa9-fc6495afca20","Type":"ContainerStarted","Data":"267409ebd1bf2a05467b52052560a601115e9a93eb2d80d7effd332260f35161"} Oct 30 18:19:08.389026 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:08.388832 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" event={"ID":"64bde5b5-0bc5-42b8-9aa9-fc6495afca20","Type":"ContainerStarted","Data":"2af47bb18f9225f46d40e3f294ea4a90136127cd35a1ffa9f4a416dff0920001"} Oct 30 18:19:08.449120 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:08.449058 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" podStartSLOduration=1.449041238 podStartE2EDuration="1.449041238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:08.444242171 +0000 UTC m=+1245.534304510" watchObservedRunningTime="2026-10-30 18:19:08.449041238 +0000 UTC m=+1245.539103576" Oct 30 18:19:08.676274 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:08.672971 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" Oct 30 18:19:09.670397 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:09.670368 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" Oct 30 18:19:10.406337 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:10.406303 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" Oct 30 18:19:10.406561 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:10.406540 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-2csfr"] Oct 30 18:19:10.410946 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:10.410916 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" Oct 30 18:19:10.411082 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:10.410999 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-2csfr" Oct 30 18:19:10.429400 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:10.429378 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-0c6613ea\"" Oct 30 18:19:10.429586 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:10.429569 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-0c6613ea-dockercfg-mk7dp\"" Oct 30 18:19:10.450346 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:10.450318 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-2csfr"] Oct 30 18:19:10.529285 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:10.528321 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99a4ef6c-a90f-4b8b-a458-1f44618ce3c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-2csfr\" (UID: \"99a4ef6c-a90f-4b8b-a458-1f44618ce3c6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-2csfr" Oct 30 18:19:10.529285 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:10.528589 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbg9p\" (UniqueName: \"kubernetes.io/projected/99a4ef6c-a90f-4b8b-a458-1f44618ce3c6-kube-api-access-wbg9p\") pod \"payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-2csfr\" (UID: \"99a4ef6c-a90f-4b8b-a458-1f44618ce3c6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-2csfr" Oct 30 18:19:10.661178 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:10.659226 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99a4ef6c-a90f-4b8b-a458-1f44618ce3c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-2csfr\" (UID: \"99a4ef6c-a90f-4b8b-a458-1f44618ce3c6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-2csfr" Oct 30 18:19:10.662179 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:10.661438 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbg9p\" (UniqueName: \"kubernetes.io/projected/99a4ef6c-a90f-4b8b-a458-1f44618ce3c6-kube-api-access-wbg9p\") pod \"payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-2csfr\" (UID: \"99a4ef6c-a90f-4b8b-a458-1f44618ce3c6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-2csfr" Oct 30 18:19:10.663258 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:10.663235 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99a4ef6c-a90f-4b8b-a458-1f44618ce3c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-2csfr\" (UID: \"99a4ef6c-a90f-4b8b-a458-1f44618ce3c6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-2csfr" Oct 30 18:19:10.872164 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:10.836310 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-5bklf"] Oct 30 18:19:10.896725 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:10.896693 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbg9p\" (UniqueName: \"kubernetes.io/projected/99a4ef6c-a90f-4b8b-a458-1f44618ce3c6-kube-api-access-wbg9p\") pod \"payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-2csfr\" (UID: \"99a4ef6c-a90f-4b8b-a458-1f44618ce3c6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-2csfr" Oct 30 18:19:10.900853 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:10.900829 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-5bklf"] Oct 30 18:19:10.901335 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:10.900862 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-5bklf" Oct 30 18:19:11.028557 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:11.028527 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-2csfr" Oct 30 18:19:11.103147 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:11.103003 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/408d585f-9fe0-4631-bfc2-a35728e68706-plugins-config-volume\") pod \"payload-processing-e2e-mt-0c6613ea-86469f66f4-5bklf\" (UID: \"408d585f-9fe0-4631-bfc2-a35728e68706\") " pod="openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-5bklf" Oct 30 18:19:11.103494 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:11.103469 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6244l\" (UniqueName: \"kubernetes.io/projected/408d585f-9fe0-4631-bfc2-a35728e68706-kube-api-access-6244l\") pod \"payload-processing-e2e-mt-0c6613ea-86469f66f4-5bklf\" (UID: \"408d585f-9fe0-4631-bfc2-a35728e68706\") " pod="openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-5bklf" Oct 30 18:19:11.207782 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:11.207745 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/408d585f-9fe0-4631-bfc2-a35728e68706-plugins-config-volume\") pod \"payload-processing-e2e-mt-0c6613ea-86469f66f4-5bklf\" (UID: \"408d585f-9fe0-4631-bfc2-a35728e68706\") " pod="openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-5bklf" Oct 30 18:19:11.208049 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:11.208033 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6244l\" (UniqueName: \"kubernetes.io/projected/408d585f-9fe0-4631-bfc2-a35728e68706-kube-api-access-6244l\") pod \"payload-processing-e2e-mt-0c6613ea-86469f66f4-5bklf\" (UID: \"408d585f-9fe0-4631-bfc2-a35728e68706\") " pod="openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-5bklf" Oct 30 18:19:11.209225 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:11.209196 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/408d585f-9fe0-4631-bfc2-a35728e68706-plugins-config-volume\") pod \"payload-processing-e2e-mt-0c6613ea-86469f66f4-5bklf\" (UID: \"408d585f-9fe0-4631-bfc2-a35728e68706\") " pod="openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-5bklf" Oct 30 18:19:11.241956 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:11.241815 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6244l\" (UniqueName: \"kubernetes.io/projected/408d585f-9fe0-4631-bfc2-a35728e68706-kube-api-access-6244l\") pod \"payload-processing-e2e-mt-0c6613ea-86469f66f4-5bklf\" (UID: \"408d585f-9fe0-4631-bfc2-a35728e68706\") " pod="openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-5bklf" Oct 30 18:19:11.247468 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:11.247441 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-5bklf" Oct 30 18:19:11.469378 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:11.469221 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-2csfr"] Oct 30 18:19:11.477837 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:19:11.477774 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99a4ef6c_a90f_4b8b_a458_1f44618ce3c6.slice/crio-a04749f4da6d7a2cf9383ce9ecf581c8b93700b36fd93c6c0041e9a274c2203b WatchSource:0}: Error finding container a04749f4da6d7a2cf9383ce9ecf581c8b93700b36fd93c6c0041e9a274c2203b: Status 404 returned error can't find the container with id a04749f4da6d7a2cf9383ce9ecf581c8b93700b36fd93c6c0041e9a274c2203b Oct 30 18:19:11.798272 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:11.789886 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-5bklf"] Oct 30 18:19:12.550651 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:12.550610 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-2csfr" event={"ID":"99a4ef6c-a90f-4b8b-a458-1f44618ce3c6","Type":"ContainerStarted","Data":"9abf5d784cc2e649b25f12d490c03d502c900421704c46a179a95077ef74615f"} Oct 30 18:19:12.551117 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:12.550680 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-2csfr" Oct 30 18:19:12.551117 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:12.550706 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-2csfr" event={"ID":"99a4ef6c-a90f-4b8b-a458-1f44618ce3c6","Type":"ContainerStarted","Data":"a04749f4da6d7a2cf9383ce9ecf581c8b93700b36fd93c6c0041e9a274c2203b"} Oct 30 18:19:12.552468 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:12.552432 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-5bklf" event={"ID":"408d585f-9fe0-4631-bfc2-a35728e68706","Type":"ContainerStarted","Data":"e0b381ab1c7813e8d549400dc5ba8cf92a58be03415b9dd823be821568ada600"} Oct 30 18:19:12.552594 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:12.552492 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-5bklf" event={"ID":"408d585f-9fe0-4631-bfc2-a35728e68706","Type":"ContainerStarted","Data":"8de100585826a5e06cf4f2aaf20c6692a55553749418bef7442937c94e0f4f19"} Oct 30 18:19:12.552667 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:12.552620 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-5bklf" Oct 30 18:19:12.591890 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:12.590819 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-2csfr" podStartSLOduration=2.590802273 podStartE2EDuration="2.590802273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:12.590480089 +0000 UTC m=+1249.680542429" watchObservedRunningTime="2026-10-30 18:19:12.590802273 +0000 UTC m=+1249.680864613" Oct 30 18:19:12.630525 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:12.630448 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-5bklf" podStartSLOduration=2.630429257 podStartE2EDuration="2.630429257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:12.628904875 +0000 UTC m=+1249.718967214" watchObservedRunningTime="2026-10-30 18:19:12.630429257 +0000 UTC m=+1249.720491596" Oct 30 18:19:12.795260 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:12.795213 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74"] Oct 30 18:19:12.801128 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:12.801047 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" Oct 30 18:19:12.820632 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:12.820582 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-e9096a-openshift-default-dockercfg-msgn2\"" Oct 30 18:19:12.824716 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:12.824690 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74"] Oct 30 18:19:12.864902 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:12.864863 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl2xd\" (UniqueName: \"kubernetes.io/projected/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-kube-api-access-jl2xd\") pod \"e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74\" (UID: \"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247\") " pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" Oct 30 18:19:12.865099 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:12.864917 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-istiod-ca-cert\") pod \"e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74\" (UID: \"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247\") " pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" Oct 30 18:19:12.865099 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:12.865080 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-istio-data\") pod \"e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74\" (UID: \"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247\") " pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" Oct 30 18:19:12.865291 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:12.865264 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-workload-certs\") pod \"e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74\" (UID: \"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247\") " pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" Oct 30 18:19:12.865414 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:12.865389 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-workload-socket\") pod \"e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74\" (UID: \"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247\") " pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" Oct 30 18:19:12.865459 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:12.865439 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-credential-socket\") pod \"e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74\" (UID: \"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247\") " pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" Oct 30 18:19:12.865518 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:12.865464 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-istio-podinfo\") pod \"e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74\" (UID: \"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247\") " pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" Oct 30 18:19:12.865641 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:12.865622 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-istio-token\") pod \"e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74\" (UID: \"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247\") " pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" Oct 30 18:19:12.865758 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:12.865723 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-istio-envoy\") pod \"e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74\" (UID: \"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247\") " pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" Oct 30 18:19:12.966975 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:12.966933 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-istio-data\") pod \"e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74\" (UID: \"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247\") " pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" Oct 30 18:19:12.967197 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:12.967044 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-workload-certs\") pod \"e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74\" (UID: \"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247\") " pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" Oct 30 18:19:12.967197 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:12.967172 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-workload-socket\") pod \"e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74\" (UID: \"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247\") " pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" Oct 30 18:19:12.967347 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:12.967235 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-credential-socket\") pod \"e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74\" (UID: \"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247\") " pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" Oct 30 18:19:12.967347 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:12.967264 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-istio-podinfo\") pod \"e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74\" (UID: \"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247\") " pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" Oct 30 18:19:12.967451 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:12.967350 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-istio-token\") pod \"e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74\" (UID: \"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247\") " pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" Oct 30 18:19:12.967451 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:12.967400 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-istio-envoy\") pod \"e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74\" (UID: \"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247\") " pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" Oct 30 18:19:12.967451 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:12.967433 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl2xd\" (UniqueName: \"kubernetes.io/projected/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-kube-api-access-jl2xd\") pod \"e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74\" (UID: \"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247\") " pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" Oct 30 18:19:12.967592 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:12.967464 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-istiod-ca-cert\") pod \"e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74\" (UID: \"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247\") " pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" Oct 30 18:19:12.968392 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:12.968359 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-istiod-ca-cert\") pod \"e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74\" (UID: \"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247\") " pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" Oct 30 18:19:12.968720 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:12.968698 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-credential-socket\") pod \"e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74\" (UID: \"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247\") " pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" Oct 30 18:19:12.969377 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:12.969355 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-istio-data\") pod \"e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74\" (UID: \"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247\") " pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" Oct 30 18:19:12.969522 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:12.969370 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-workload-certs\") pod \"e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74\" (UID: \"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247\") " pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" Oct 30 18:19:12.969635 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:12.969607 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-workload-socket\") pod \"e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74\" (UID: \"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247\") " pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" Oct 30 18:19:12.971461 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:12.971435 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-istio-envoy\") pod \"e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74\" (UID: \"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247\") " pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" Oct 30 18:19:12.972927 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:12.972901 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-istio-podinfo\") pod \"e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74\" (UID: \"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247\") " pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" Oct 30 18:19:12.991525 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:12.991496 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-istio-token\") pod \"e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74\" (UID: \"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247\") " pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" Oct 30 18:19:12.991751 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:12.991728 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl2xd\" (UniqueName: \"kubernetes.io/projected/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-kube-api-access-jl2xd\") pod \"e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74\" (UID: \"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247\") " pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" Oct 30 18:19:13.122128 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:13.122099 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" Oct 30 18:19:13.253925 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:13.253892 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2ee874-69fddfb67-hscz8"] Oct 30 18:19:13.264439 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:13.263759 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ee874-69fddfb67-hscz8" Oct 30 18:19:13.278246 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:13.278205 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-2ee874\"" Oct 30 18:19:13.278430 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:13.278368 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-2ee874-dockercfg-xghq6\"" Oct 30 18:19:13.279350 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:13.279326 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2ee874-69fddfb67-hscz8"] Oct 30 18:19:13.351219 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:13.351183 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2ee874-5b975bd4-drq62"] Oct 30 18:19:13.355882 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:13.355862 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2ee874-5b975bd4-drq62" Oct 30 18:19:13.373715 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:13.373650 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm282\" (UniqueName: \"kubernetes.io/projected/e295ee2a-45e5-485d-9418-b046b37095d2-kube-api-access-mm282\") pod \"payload-pre-processing-e2e-api-a-2ee874-69fddfb67-hscz8\" (UID: \"e295ee2a-45e5-485d-9418-b046b37095d2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ee874-69fddfb67-hscz8" Oct 30 18:19:13.373837 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:13.373773 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f233a98-d41a-4a25-9901-418e48cebadd-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2ee874-5b975bd4-drq62\" (UID: \"6f233a98-d41a-4a25-9901-418e48cebadd\") " pod="openshift-ingress/payload-processing-e2e-api-a-2ee874-5b975bd4-drq62" Oct 30 18:19:13.373837 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:13.373807 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e295ee2a-45e5-485d-9418-b046b37095d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2ee874-69fddfb67-hscz8\" (UID: \"e295ee2a-45e5-485d-9418-b046b37095d2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ee874-69fddfb67-hscz8" Oct 30 18:19:13.373983 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:13.373963 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mln4d\" (UniqueName: \"kubernetes.io/projected/6f233a98-d41a-4a25-9901-418e48cebadd-kube-api-access-mln4d\") pod \"payload-processing-e2e-api-a-2ee874-5b975bd4-drq62\" (UID: \"6f233a98-d41a-4a25-9901-418e48cebadd\") " pod="openshift-ingress/payload-processing-e2e-api-a-2ee874-5b975bd4-drq62" Oct 30 18:19:13.685712 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:13.685680 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f233a98-d41a-4a25-9901-418e48cebadd-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2ee874-5b975bd4-drq62\" (UID: \"6f233a98-d41a-4a25-9901-418e48cebadd\") " pod="openshift-ingress/payload-processing-e2e-api-a-2ee874-5b975bd4-drq62" Oct 30 18:19:13.686333 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:13.686311 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e295ee2a-45e5-485d-9418-b046b37095d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2ee874-69fddfb67-hscz8\" (UID: \"e295ee2a-45e5-485d-9418-b046b37095d2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ee874-69fddfb67-hscz8" Oct 30 18:19:13.687385 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:13.687364 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mln4d\" (UniqueName: \"kubernetes.io/projected/6f233a98-d41a-4a25-9901-418e48cebadd-kube-api-access-mln4d\") pod \"payload-processing-e2e-api-a-2ee874-5b975bd4-drq62\" (UID: \"6f233a98-d41a-4a25-9901-418e48cebadd\") " pod="openshift-ingress/payload-processing-e2e-api-a-2ee874-5b975bd4-drq62" Oct 30 18:19:13.688033 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:13.688012 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm282\" (UniqueName: \"kubernetes.io/projected/e295ee2a-45e5-485d-9418-b046b37095d2-kube-api-access-mm282\") pod \"payload-pre-processing-e2e-api-a-2ee874-69fddfb67-hscz8\" (UID: \"e295ee2a-45e5-485d-9418-b046b37095d2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ee874-69fddfb67-hscz8" Oct 30 18:19:13.688425 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:13.687186 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f233a98-d41a-4a25-9901-418e48cebadd-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2ee874-5b975bd4-drq62\" (UID: \"6f233a98-d41a-4a25-9901-418e48cebadd\") " pod="openshift-ingress/payload-processing-e2e-api-a-2ee874-5b975bd4-drq62" Oct 30 18:19:13.689831 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:13.687199 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e295ee2a-45e5-485d-9418-b046b37095d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2ee874-69fddfb67-hscz8\" (UID: \"e295ee2a-45e5-485d-9418-b046b37095d2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ee874-69fddfb67-hscz8" Oct 30 18:19:13.689964 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:13.689103 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2ee874-5b975bd4-drq62"] Oct 30 18:19:13.690089 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:19:13.689546 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbf948c4_274b_46fd_bc3c_1c2f0d9c3247.slice/crio-018218b52a72839174f29782938f08719af2641d4180fa3775bc5d83784babae WatchSource:0}: Error finding container 018218b52a72839174f29782938f08719af2641d4180fa3775bc5d83784babae: Status 404 returned error can't find the container with id 018218b52a72839174f29782938f08719af2641d4180fa3775bc5d83784babae Oct 30 18:19:13.694671 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:13.694418 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:19:13.694671 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:13.694495 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:19:13.694671 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:13.694539 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:19:13.705926 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:13.705889 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74"] Oct 30 18:19:13.710802 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:13.710773 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mln4d\" (UniqueName: \"kubernetes.io/projected/6f233a98-d41a-4a25-9901-418e48cebadd-kube-api-access-mln4d\") pod \"payload-processing-e2e-api-a-2ee874-5b975bd4-drq62\" (UID: \"6f233a98-d41a-4a25-9901-418e48cebadd\") " pod="openshift-ingress/payload-processing-e2e-api-a-2ee874-5b975bd4-drq62" Oct 30 18:19:13.718943 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:13.718914 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm282\" (UniqueName: \"kubernetes.io/projected/e295ee2a-45e5-485d-9418-b046b37095d2-kube-api-access-mm282\") pod \"payload-pre-processing-e2e-api-a-2ee874-69fddfb67-hscz8\" (UID: \"e295ee2a-45e5-485d-9418-b046b37095d2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ee874-69fddfb67-hscz8" Oct 30 18:19:13.829862 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:13.829796 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh"] Oct 30 18:19:13.871777 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:13.871750 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh" Oct 30 18:19:13.874632 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:13.874603 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh"] Oct 30 18:19:13.890257 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:13.889408 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-0c6613ea-serving-cert\"" Oct 30 18:19:13.890632 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:13.890593 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ee874-69fddfb67-hscz8" Oct 30 18:19:13.897948 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:13.897920 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e328cf8-f89f-4f08-aafb-d481dd61ffd3-maas-api-tls\") pod \"maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh\" (UID: \"0e328cf8-f89f-4f08-aafb-d481dd61ffd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh" Oct 30 18:19:13.898151 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:13.898115 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e328cf8-f89f-4f08-aafb-d481dd61ffd3-metrics-certs\") pod \"maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh\" (UID: \"0e328cf8-f89f-4f08-aafb-d481dd61ffd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh" Oct 30 18:19:13.898334 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:13.898315 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56dr5\" (UniqueName: \"kubernetes.io/projected/0e328cf8-f89f-4f08-aafb-d481dd61ffd3-kube-api-access-56dr5\") pod \"maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh\" (UID: \"0e328cf8-f89f-4f08-aafb-d481dd61ffd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh" Oct 30 18:19:13.898570 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:13.898552 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e328cf8-f89f-4f08-aafb-d481dd61ffd3-openshift-service-ca\") pod \"maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh\" (UID: \"0e328cf8-f89f-4f08-aafb-d481dd61ffd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh" Oct 30 18:19:13.975694 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:13.974127 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2ee874-5b975bd4-drq62" Oct 30 18:19:14.001201 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:14.000511 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e328cf8-f89f-4f08-aafb-d481dd61ffd3-maas-api-tls\") pod \"maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh\" (UID: \"0e328cf8-f89f-4f08-aafb-d481dd61ffd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh" Oct 30 18:19:14.001201 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:14.000586 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e328cf8-f89f-4f08-aafb-d481dd61ffd3-metrics-certs\") pod \"maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh\" (UID: \"0e328cf8-f89f-4f08-aafb-d481dd61ffd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh" Oct 30 18:19:14.001201 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:14.000630 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56dr5\" (UniqueName: \"kubernetes.io/projected/0e328cf8-f89f-4f08-aafb-d481dd61ffd3-kube-api-access-56dr5\") pod \"maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh\" (UID: \"0e328cf8-f89f-4f08-aafb-d481dd61ffd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh" Oct 30 18:19:14.001201 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:14.000793 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e328cf8-f89f-4f08-aafb-d481dd61ffd3-openshift-service-ca\") pod \"maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh\" (UID: \"0e328cf8-f89f-4f08-aafb-d481dd61ffd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh" Oct 30 18:19:14.010803 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:14.001763 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e328cf8-f89f-4f08-aafb-d481dd61ffd3-openshift-service-ca\") pod \"maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh\" (UID: \"0e328cf8-f89f-4f08-aafb-d481dd61ffd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh" Oct 30 18:19:14.010803 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:19:14.001882 4442 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-serving-cert: secret "maas-api-e2e-mt-0c6613ea-serving-cert" not found Oct 30 18:19:14.010803 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:19:14.001942 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e328cf8-f89f-4f08-aafb-d481dd61ffd3-maas-api-tls podName:0e328cf8-f89f-4f08-aafb-d481dd61ffd3 nodeName:}" failed. No retries permitted until 2026-10-30 18:19:14.501921726 +0000 UTC m=+1251.591984056 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0e328cf8-f89f-4f08-aafb-d481dd61ffd3-maas-api-tls") pod "maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh" (UID: "0e328cf8-f89f-4f08-aafb-d481dd61ffd3") : secret "maas-api-e2e-mt-0c6613ea-serving-cert" not found Oct 30 18:19:14.090159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:14.089799 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e328cf8-f89f-4f08-aafb-d481dd61ffd3-metrics-certs\") pod \"maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh\" (UID: \"0e328cf8-f89f-4f08-aafb-d481dd61ffd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh" Oct 30 18:19:14.182303 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:14.182181 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56dr5\" (UniqueName: \"kubernetes.io/projected/0e328cf8-f89f-4f08-aafb-d481dd61ffd3-kube-api-access-56dr5\") pod \"maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh\" (UID: \"0e328cf8-f89f-4f08-aafb-d481dd61ffd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh" Oct 30 18:19:14.422601 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:14.422574 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2ee874-69fddfb67-hscz8"] Oct 30 18:19:14.424421 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:19:14.424390 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode295ee2a_45e5_485d_9418_b046b37095d2.slice/crio-c45f95ff72dce7eb3a6ec9380fc971d2c02abdb43c290e4c24e7d6b7ddd90acd WatchSource:0}: Error finding container c45f95ff72dce7eb3a6ec9380fc971d2c02abdb43c290e4c24e7d6b7ddd90acd: Status 404 returned error can't find the container with id c45f95ff72dce7eb3a6ec9380fc971d2c02abdb43c290e4c24e7d6b7ddd90acd Oct 30 18:19:14.511213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:14.511179 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e328cf8-f89f-4f08-aafb-d481dd61ffd3-maas-api-tls\") pod \"maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh\" (UID: \"0e328cf8-f89f-4f08-aafb-d481dd61ffd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh" Oct 30 18:19:14.514010 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:14.513989 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e328cf8-f89f-4f08-aafb-d481dd61ffd3-maas-api-tls\") pod \"maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh\" (UID: \"0e328cf8-f89f-4f08-aafb-d481dd61ffd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh" Oct 30 18:19:14.579803 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:14.579774 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2ee874-5b975bd4-drq62"] Oct 30 18:19:14.581207 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:19:14.581181 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f233a98_d41a_4a25_9901_418e48cebadd.slice/crio-7def7b0d39048a1ef21e513cda8bd58a86a2133d03faafd92ebb1ae0d77beee3 WatchSource:0}: Error finding container 7def7b0d39048a1ef21e513cda8bd58a86a2133d03faafd92ebb1ae0d77beee3: Status 404 returned error can't find the container with id 7def7b0d39048a1ef21e513cda8bd58a86a2133d03faafd92ebb1ae0d77beee3 Oct 30 18:19:14.685757 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:14.685725 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" event={"ID":"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247","Type":"ContainerStarted","Data":"318b9c893169d6da3c506b4f3cddfb1f088595d443be070791da37e9149af3ce"} Oct 30 18:19:14.686160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:14.685777 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" event={"ID":"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247","Type":"ContainerStarted","Data":"018218b52a72839174f29782938f08719af2641d4180fa3775bc5d83784babae"} Oct 30 18:19:14.687550 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:14.687528 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2ee874-5b975bd4-drq62" event={"ID":"6f233a98-d41a-4a25-9901-418e48cebadd","Type":"ContainerStarted","Data":"965942e71f9d587a54dcb01a2e5d7b9b841bc766d8534a317f6a99d27da6926e"} Oct 30 18:19:14.687635 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:14.687568 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2ee874-5b975bd4-drq62" event={"ID":"6f233a98-d41a-4a25-9901-418e48cebadd","Type":"ContainerStarted","Data":"7def7b0d39048a1ef21e513cda8bd58a86a2133d03faafd92ebb1ae0d77beee3"} Oct 30 18:19:14.687635 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:14.687591 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-2ee874-5b975bd4-drq62" Oct 30 18:19:14.689184 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:14.689128 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ee874-69fddfb67-hscz8" event={"ID":"e295ee2a-45e5-485d-9418-b046b37095d2","Type":"ContainerStarted","Data":"ac06a0977b9a226ebefb88de50e0e34679f723b72fd4f867967c4e8eb8266e6a"} Oct 30 18:19:14.689300 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:14.689205 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ee874-69fddfb67-hscz8" event={"ID":"e295ee2a-45e5-485d-9418-b046b37095d2","Type":"ContainerStarted","Data":"c45f95ff72dce7eb3a6ec9380fc971d2c02abdb43c290e4c24e7d6b7ddd90acd"} Oct 30 18:19:14.689300 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:14.689235 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ee874-69fddfb67-hscz8" Oct 30 18:19:14.735446 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:14.735322 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" podStartSLOduration=2.73530172 podStartE2EDuration="2.73530172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:14.731560448 +0000 UTC m=+1251.821622785" watchObservedRunningTime="2026-10-30 18:19:14.73530172 +0000 UTC m=+1251.825364060" Oct 30 18:19:14.805172 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:14.805094 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh" Oct 30 18:19:14.814335 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:14.814274 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ee874-69fddfb67-hscz8" podStartSLOduration=1.8142557830000001 podStartE2EDuration="1.814255783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:14.810390287 +0000 UTC m=+1251.900452627" watchObservedRunningTime="2026-10-30 18:19:14.814255783 +0000 UTC m=+1251.904318126" Oct 30 18:19:14.815625 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:14.815584 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-2ee874-5b975bd4-drq62" podStartSLOduration=1.815574755 podStartE2EDuration="1.815574755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:14.772332526 +0000 UTC m=+1251.862394867" watchObservedRunningTime="2026-10-30 18:19:14.815574755 +0000 UTC m=+1251.905637096" Oct 30 18:19:15.131058 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:15.124394 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" Oct 30 18:19:15.149170 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:15.143874 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" Oct 30 18:19:15.194596 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:15.188908 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh"] Oct 30 18:19:15.697685 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:15.697652 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh" event={"ID":"0e328cf8-f89f-4f08-aafb-d481dd61ffd3","Type":"ContainerStarted","Data":"d5167fa7e254d8649b0d16e48171ef202b62baeaee5fb4b70f9891e9d523da47"} Oct 30 18:19:15.698111 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:15.698055 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" Oct 30 18:19:15.699197 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:15.699170 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" Oct 30 18:19:16.710330 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:16.710295 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh" event={"ID":"0e328cf8-f89f-4f08-aafb-d481dd61ffd3","Type":"ContainerStarted","Data":"eb55ad414aa08dd5097f9744b4f7d9aa6db78bc4d68769c7639e83429203ea50"} Oct 30 18:19:16.710718 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:16.710379 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh" Oct 30 18:19:17.571550 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:19:17.571510 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:19:19.150469 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:19.150406 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh" podStartSLOduration=5.630482508 podStartE2EDuration="6.150386003s" totalImagesPullingTime="519.903495ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:16.764317982 +0000 UTC m=+1253.854380320" watchObservedRunningTime="2026-10-30 18:19:19.150386003 +0000 UTC m=+1256.240448343" Oct 30 18:19:19.153674 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:19.153645 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-5669779bc-7249d"] Oct 30 18:19:19.159835 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:19.159814 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-5669779bc-7249d" Oct 30 18:19:19.176880 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:19.176852 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-5669779bc-7249d"] Oct 30 18:19:19.186425 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:19.186342 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f42594-dockercfg-pkgrj\"" Oct 30 18:19:19.221446 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:19.221416 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f42594\"" Oct 30 18:19:19.448218 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:19.363879 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f42594-7c6f78d84-zkpr2"] Oct 30 18:19:19.453695 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:19.453657 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f42594-7c6f78d84-zkpr2"] Oct 30 18:19:19.453870 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:19.453786 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f42594-7c6f78d84-zkpr2" Oct 30 18:19:19.485663 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:19.485224 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c0da160-9bbb-4f72-9d56-b54a269a6496-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f42594-7c6f78d84-zkpr2\" (UID: \"0c0da160-9bbb-4f72-9d56-b54a269a6496\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f42594-7c6f78d84-zkpr2" Oct 30 18:19:19.485663 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:19.485322 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whtmk\" (UniqueName: \"kubernetes.io/projected/c1461e0a-7e0f-4942-87b4-b2b160dbc10f-kube-api-access-whtmk\") pod \"payload-pre-processing-e2e-ipp-a-f42594-5669779bc-7249d\" (UID: \"c1461e0a-7e0f-4942-87b4-b2b160dbc10f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-5669779bc-7249d" Oct 30 18:19:19.485663 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:19.485399 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1461e0a-7e0f-4942-87b4-b2b160dbc10f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f42594-5669779bc-7249d\" (UID: \"c1461e0a-7e0f-4942-87b4-b2b160dbc10f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-5669779bc-7249d" Oct 30 18:19:19.485663 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:19.485477 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qkbd\" (UniqueName: \"kubernetes.io/projected/0c0da160-9bbb-4f72-9d56-b54a269a6496-kube-api-access-9qkbd\") pod \"payload-processing-e2e-ipp-a-f42594-7c6f78d84-zkpr2\" (UID: \"0c0da160-9bbb-4f72-9d56-b54a269a6496\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f42594-7c6f78d84-zkpr2" Oct 30 18:19:19.642260 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:19.609807 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qkbd\" (UniqueName: \"kubernetes.io/projected/0c0da160-9bbb-4f72-9d56-b54a269a6496-kube-api-access-9qkbd\") pod \"payload-processing-e2e-ipp-a-f42594-7c6f78d84-zkpr2\" (UID: \"0c0da160-9bbb-4f72-9d56-b54a269a6496\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f42594-7c6f78d84-zkpr2" Oct 30 18:19:19.642435 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:19.636452 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qkbd\" (UniqueName: \"kubernetes.io/projected/0c0da160-9bbb-4f72-9d56-b54a269a6496-kube-api-access-9qkbd\") pod \"payload-processing-e2e-ipp-a-f42594-7c6f78d84-zkpr2\" (UID: \"0c0da160-9bbb-4f72-9d56-b54a269a6496\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f42594-7c6f78d84-zkpr2" Oct 30 18:19:19.643251 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:19.642750 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c0da160-9bbb-4f72-9d56-b54a269a6496-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f42594-7c6f78d84-zkpr2\" (UID: \"0c0da160-9bbb-4f72-9d56-b54a269a6496\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f42594-7c6f78d84-zkpr2" Oct 30 18:19:19.643251 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:19.642847 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whtmk\" (UniqueName: \"kubernetes.io/projected/c1461e0a-7e0f-4942-87b4-b2b160dbc10f-kube-api-access-whtmk\") pod \"payload-pre-processing-e2e-ipp-a-f42594-5669779bc-7249d\" (UID: \"c1461e0a-7e0f-4942-87b4-b2b160dbc10f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-5669779bc-7249d" Oct 30 18:19:19.643251 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:19.642954 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1461e0a-7e0f-4942-87b4-b2b160dbc10f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f42594-5669779bc-7249d\" (UID: \"c1461e0a-7e0f-4942-87b4-b2b160dbc10f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-5669779bc-7249d" Oct 30 18:19:19.643938 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:19.643909 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1461e0a-7e0f-4942-87b4-b2b160dbc10f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f42594-5669779bc-7249d\" (UID: \"c1461e0a-7e0f-4942-87b4-b2b160dbc10f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-5669779bc-7249d" Oct 30 18:19:19.644672 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:19.644649 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c0da160-9bbb-4f72-9d56-b54a269a6496-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f42594-7c6f78d84-zkpr2\" (UID: \"0c0da160-9bbb-4f72-9d56-b54a269a6496\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f42594-7c6f78d84-zkpr2" Oct 30 18:19:19.672891 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:19.672833 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whtmk\" (UniqueName: \"kubernetes.io/projected/c1461e0a-7e0f-4942-87b4-b2b160dbc10f-kube-api-access-whtmk\") pod \"payload-pre-processing-e2e-ipp-a-f42594-5669779bc-7249d\" (UID: \"c1461e0a-7e0f-4942-87b4-b2b160dbc10f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-5669779bc-7249d" Oct 30 18:19:19.787001 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:19.786914 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-5669779bc-7249d" Oct 30 18:19:19.849596 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:19.849150 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f42594-7c6f78d84-zkpr2" Oct 30 18:19:20.361756 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:20.361730 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-5669779bc-7249d"] Oct 30 18:19:20.426123 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:20.426099 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f42594-7c6f78d84-zkpr2"] Oct 30 18:19:20.433749 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:19:20.433724 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c0da160_9bbb_4f72_9d56_b54a269a6496.slice/crio-4b23eb623e5719c983ac8599cc101688ca694026c911519686cdcbca9ecc10cc WatchSource:0}: Error finding container 4b23eb623e5719c983ac8599cc101688ca694026c911519686cdcbca9ecc10cc: Status 404 returned error can't find the container with id 4b23eb623e5719c983ac8599cc101688ca694026c911519686cdcbca9ecc10cc Oct 30 18:19:20.806073 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:20.805997 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f42594-7c6f78d84-zkpr2" event={"ID":"0c0da160-9bbb-4f72-9d56-b54a269a6496","Type":"ContainerStarted","Data":"9fb27c601f129a6ef74635a1b464ba760b71e48aada717311ff2e02814db0eaf"} Oct 30 18:19:20.806349 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:20.806322 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f42594-7c6f78d84-zkpr2" event={"ID":"0c0da160-9bbb-4f72-9d56-b54a269a6496","Type":"ContainerStarted","Data":"4b23eb623e5719c983ac8599cc101688ca694026c911519686cdcbca9ecc10cc"} Oct 30 18:19:20.807268 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:20.807211 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f42594-7c6f78d84-zkpr2" Oct 30 18:19:20.824582 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:20.824547 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-5669779bc-7249d" event={"ID":"c1461e0a-7e0f-4942-87b4-b2b160dbc10f","Type":"ContainerStarted","Data":"5302010b7d675e4e1ead430fc997fa61ece8d4d400015f847c5c8c578815fe16"} Oct 30 18:19:20.824842 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:20.824822 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-5669779bc-7249d" event={"ID":"c1461e0a-7e0f-4942-87b4-b2b160dbc10f","Type":"ContainerStarted","Data":"2128d3e48ce1913e664310768c8d3f39d8e51c076278f88cbb61ad040961e841"} Oct 30 18:19:20.839319 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:20.839290 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-5669779bc-7249d" Oct 30 18:19:20.868006 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:20.867946 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f42594-7c6f78d84-zkpr2" podStartSLOduration=1.867927694 podStartE2EDuration="1.867927694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:20.85922935 +0000 UTC m=+1257.949291690" watchObservedRunningTime="2026-10-30 18:19:20.867927694 +0000 UTC m=+1257.957990035" Oct 30 18:19:20.872760 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:20.872727 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-84c84577c5-q85dn"] Oct 30 18:19:20.878526 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:20.878503 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-84c84577c5-q85dn" Oct 30 18:19:20.896278 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:20.893251 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e9096a\"" Oct 30 18:19:20.896278 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:20.893341 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-e9096a-dockercfg-fg488\"" Oct 30 18:19:20.898172 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:20.898149 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-84c84577c5-q85dn"] Oct 30 18:19:20.901906 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:20.901864 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-5669779bc-7249d" podStartSLOduration=1.901851935 podStartE2EDuration="1.901851935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:20.900043393 +0000 UTC m=+1257.990105731" watchObservedRunningTime="2026-10-30 18:19:20.901851935 +0000 UTC m=+1257.991914273" Oct 30 18:19:20.948858 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:20.948823 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e9096a-685c586796-k9fgv"] Oct 30 18:19:20.954242 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:20.954218 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9096a-685c586796-k9fgv" Oct 30 18:19:20.962965 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:20.962938 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6d9eb70-ee02-4a61-b625-83f906804c08-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e9096a-84c84577c5-q85dn\" (UID: \"b6d9eb70-ee02-4a61-b625-83f906804c08\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-84c84577c5-q85dn" Oct 30 18:19:20.963092 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:20.963027 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44j7c\" (UniqueName: \"kubernetes.io/projected/b6d9eb70-ee02-4a61-b625-83f906804c08-kube-api-access-44j7c\") pod \"payload-pre-processing-e2e-ipp-a-e9096a-84c84577c5-q85dn\" (UID: \"b6d9eb70-ee02-4a61-b625-83f906804c08\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-84c84577c5-q85dn" Oct 30 18:19:20.975128 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:20.975096 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e9096a-685c586796-k9fgv"] Oct 30 18:19:21.075208 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:21.074302 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6d9eb70-ee02-4a61-b625-83f906804c08-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e9096a-84c84577c5-q85dn\" (UID: \"b6d9eb70-ee02-4a61-b625-83f906804c08\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-84c84577c5-q85dn" Oct 30 18:19:21.075208 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:21.074424 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zgm5\" (UniqueName: \"kubernetes.io/projected/61a7e965-4538-4848-b50c-a5eb5744c8e3-kube-api-access-9zgm5\") pod \"payload-processing-e2e-ipp-a-e9096a-685c586796-k9fgv\" (UID: \"61a7e965-4538-4848-b50c-a5eb5744c8e3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e9096a-685c586796-k9fgv" Oct 30 18:19:21.075208 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:21.074481 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44j7c\" (UniqueName: \"kubernetes.io/projected/b6d9eb70-ee02-4a61-b625-83f906804c08-kube-api-access-44j7c\") pod \"payload-pre-processing-e2e-ipp-a-e9096a-84c84577c5-q85dn\" (UID: \"b6d9eb70-ee02-4a61-b625-83f906804c08\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-84c84577c5-q85dn" Oct 30 18:19:21.075208 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:21.074572 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61a7e965-4538-4848-b50c-a5eb5744c8e3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e9096a-685c586796-k9fgv\" (UID: \"61a7e965-4538-4848-b50c-a5eb5744c8e3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e9096a-685c586796-k9fgv" Oct 30 18:19:21.103485 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:21.076029 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6d9eb70-ee02-4a61-b625-83f906804c08-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e9096a-84c84577c5-q85dn\" (UID: \"b6d9eb70-ee02-4a61-b625-83f906804c08\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-84c84577c5-q85dn" Oct 30 18:19:21.177160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:21.176396 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zgm5\" (UniqueName: \"kubernetes.io/projected/61a7e965-4538-4848-b50c-a5eb5744c8e3-kube-api-access-9zgm5\") pod \"payload-processing-e2e-ipp-a-e9096a-685c586796-k9fgv\" (UID: \"61a7e965-4538-4848-b50c-a5eb5744c8e3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e9096a-685c586796-k9fgv" Oct 30 18:19:21.177160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:21.176524 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61a7e965-4538-4848-b50c-a5eb5744c8e3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e9096a-685c586796-k9fgv\" (UID: \"61a7e965-4538-4848-b50c-a5eb5744c8e3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e9096a-685c586796-k9fgv" Oct 30 18:19:21.181161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:21.177644 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61a7e965-4538-4848-b50c-a5eb5744c8e3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e9096a-685c586796-k9fgv\" (UID: \"61a7e965-4538-4848-b50c-a5eb5744c8e3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e9096a-685c586796-k9fgv" Oct 30 18:19:21.194151 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:21.194057 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44j7c\" (UniqueName: \"kubernetes.io/projected/b6d9eb70-ee02-4a61-b625-83f906804c08-kube-api-access-44j7c\") pod \"payload-pre-processing-e2e-ipp-a-e9096a-84c84577c5-q85dn\" (UID: \"b6d9eb70-ee02-4a61-b625-83f906804c08\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-84c84577c5-q85dn" Oct 30 18:19:21.211161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:21.203474 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-84c84577c5-q85dn" Oct 30 18:19:21.222248 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:21.215032 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zgm5\" (UniqueName: \"kubernetes.io/projected/61a7e965-4538-4848-b50c-a5eb5744c8e3-kube-api-access-9zgm5\") pod \"payload-processing-e2e-ipp-a-e9096a-685c586796-k9fgv\" (UID: \"61a7e965-4538-4848-b50c-a5eb5744c8e3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e9096a-685c586796-k9fgv" Oct 30 18:19:21.282173 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:21.280268 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9096a-685c586796-k9fgv" Oct 30 18:19:21.753948 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:21.753915 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-84c84577c5-q85dn"] Oct 30 18:19:21.755794 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:19:21.755765 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6d9eb70_ee02_4a61_b625_83f906804c08.slice/crio-26044cd77038c3dbf1948138ef05aa7a1fd81c7053f5d21037dbd4dba9dca116 WatchSource:0}: Error finding container 26044cd77038c3dbf1948138ef05aa7a1fd81c7053f5d21037dbd4dba9dca116: Status 404 returned error can't find the container with id 26044cd77038c3dbf1948138ef05aa7a1fd81c7053f5d21037dbd4dba9dca116 Oct 30 18:19:21.766045 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:21.766023 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e9096a-685c586796-k9fgv"] Oct 30 18:19:21.770920 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:19:21.770895 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61a7e965_4538_4848_b50c_a5eb5744c8e3.slice/crio-067c3f1870b161b2f350bfa4536600ec23f66c2b037bb55a810fb90b68c19a3f WatchSource:0}: Error finding container 067c3f1870b161b2f350bfa4536600ec23f66c2b037bb55a810fb90b68c19a3f: Status 404 returned error can't find the container with id 067c3f1870b161b2f350bfa4536600ec23f66c2b037bb55a810fb90b68c19a3f Oct 30 18:19:21.836457 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:21.836422 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9096a-685c586796-k9fgv" event={"ID":"61a7e965-4538-4848-b50c-a5eb5744c8e3","Type":"ContainerStarted","Data":"067c3f1870b161b2f350bfa4536600ec23f66c2b037bb55a810fb90b68c19a3f"} Oct 30 18:19:21.838187 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:21.838163 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-84c84577c5-q85dn" event={"ID":"b6d9eb70-ee02-4a61-b625-83f906804c08","Type":"ContainerStarted","Data":"26044cd77038c3dbf1948138ef05aa7a1fd81c7053f5d21037dbd4dba9dca116"} Oct 30 18:19:22.722652 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:22.722603 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh" podUID="0e328cf8-f89f-4f08-aafb-d481dd61ffd3" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.115:8443/health\": dial tcp 10.134.0.115:8443: connect: connection refused" Oct 30 18:19:22.850590 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:22.850548 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9096a-685c586796-k9fgv" event={"ID":"61a7e965-4538-4848-b50c-a5eb5744c8e3","Type":"ContainerStarted","Data":"520ca286297eb1c6fbc8f49af5ddcf4cd41f2b8a8943233082374e1f31d4e3dc"} Oct 30 18:19:22.851049 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:22.850789 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9096a-685c586796-k9fgv" Oct 30 18:19:22.852611 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:22.852583 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-84c84577c5-q85dn" event={"ID":"b6d9eb70-ee02-4a61-b625-83f906804c08","Type":"ContainerStarted","Data":"5bf8935316376cc4134d1f9e6cad4b4b8d8f8c0f130d88d01352198ca884e3c1"} Oct 30 18:19:22.852753 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:22.852726 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-84c84577c5-q85dn" Oct 30 18:19:22.888887 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:22.888837 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9096a-685c586796-k9fgv" podStartSLOduration=2.888821995 podStartE2EDuration="2.888821995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:22.88772774 +0000 UTC m=+1259.977790080" watchObservedRunningTime="2026-10-30 18:19:22.888821995 +0000 UTC m=+1259.978884335" Oct 30 18:19:22.933057 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:22.932994 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-84c84577c5-q85dn" podStartSLOduration=2.9329762539999997 podStartE2EDuration="2.932976254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:22.928426856 +0000 UTC m=+1260.018489196" watchObservedRunningTime="2026-10-30 18:19:22.932976254 +0000 UTC m=+1260.023038594" Oct 30 18:19:23.694573 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:23.694538 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-5bklf" Oct 30 18:19:23.695094 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:23.695070 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-2csfr" Oct 30 18:19:24.921027 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:24.920982 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2ee874-fcdcfc47c-ztm82"] Oct 30 18:19:24.927635 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:24.927601 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ee874-fcdcfc47c-ztm82" Oct 30 18:19:24.947546 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:24.947508 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2ee874-fcdcfc47c-ztm82"] Oct 30 18:19:24.950057 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:24.950028 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-2ee874-serving-cert\"" Oct 30 18:19:25.009629 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:25.009576 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93c780c7-a1ca-4b63-86bb-7b59dd49745a-openshift-service-ca\") pod \"maas-api-e2e-api-a-2ee874-fcdcfc47c-ztm82\" (UID: \"93c780c7-a1ca-4b63-86bb-7b59dd49745a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ee874-fcdcfc47c-ztm82" Oct 30 18:19:25.009809 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:25.009739 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93c780c7-a1ca-4b63-86bb-7b59dd49745a-metrics-certs\") pod \"maas-api-e2e-api-a-2ee874-fcdcfc47c-ztm82\" (UID: \"93c780c7-a1ca-4b63-86bb-7b59dd49745a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ee874-fcdcfc47c-ztm82" Oct 30 18:19:25.010066 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:25.010046 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mqwb\" (UniqueName: \"kubernetes.io/projected/93c780c7-a1ca-4b63-86bb-7b59dd49745a-kube-api-access-6mqwb\") pod \"maas-api-e2e-api-a-2ee874-fcdcfc47c-ztm82\" (UID: \"93c780c7-a1ca-4b63-86bb-7b59dd49745a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ee874-fcdcfc47c-ztm82" Oct 30 18:19:25.010166 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:25.010084 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93c780c7-a1ca-4b63-86bb-7b59dd49745a-maas-api-tls\") pod \"maas-api-e2e-api-a-2ee874-fcdcfc47c-ztm82\" (UID: \"93c780c7-a1ca-4b63-86bb-7b59dd49745a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ee874-fcdcfc47c-ztm82" Oct 30 18:19:25.133155 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:25.127699 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mqwb\" (UniqueName: \"kubernetes.io/projected/93c780c7-a1ca-4b63-86bb-7b59dd49745a-kube-api-access-6mqwb\") pod \"maas-api-e2e-api-a-2ee874-fcdcfc47c-ztm82\" (UID: \"93c780c7-a1ca-4b63-86bb-7b59dd49745a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ee874-fcdcfc47c-ztm82" Oct 30 18:19:25.133155 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:25.127816 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93c780c7-a1ca-4b63-86bb-7b59dd49745a-maas-api-tls\") pod \"maas-api-e2e-api-a-2ee874-fcdcfc47c-ztm82\" (UID: \"93c780c7-a1ca-4b63-86bb-7b59dd49745a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ee874-fcdcfc47c-ztm82" Oct 30 18:19:25.133155 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:25.127864 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93c780c7-a1ca-4b63-86bb-7b59dd49745a-openshift-service-ca\") pod \"maas-api-e2e-api-a-2ee874-fcdcfc47c-ztm82\" (UID: \"93c780c7-a1ca-4b63-86bb-7b59dd49745a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ee874-fcdcfc47c-ztm82" Oct 30 18:19:25.133155 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:25.127975 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93c780c7-a1ca-4b63-86bb-7b59dd49745a-metrics-certs\") pod \"maas-api-e2e-api-a-2ee874-fcdcfc47c-ztm82\" (UID: \"93c780c7-a1ca-4b63-86bb-7b59dd49745a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ee874-fcdcfc47c-ztm82" Oct 30 18:19:25.133155 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:19:25.129571 4442 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-2ee874-serving-cert: secret "maas-api-e2e-api-a-2ee874-serving-cert" not found Oct 30 18:19:25.133155 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:19:25.129638 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93c780c7-a1ca-4b63-86bb-7b59dd49745a-maas-api-tls podName:93c780c7-a1ca-4b63-86bb-7b59dd49745a nodeName:}" failed. No retries permitted until 2026-10-30 18:19:25.629616381 +0000 UTC m=+1262.719678700 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/93c780c7-a1ca-4b63-86bb-7b59dd49745a-maas-api-tls") pod "maas-api-e2e-api-a-2ee874-fcdcfc47c-ztm82" (UID: "93c780c7-a1ca-4b63-86bb-7b59dd49745a") : secret "maas-api-e2e-api-a-2ee874-serving-cert" not found Oct 30 18:19:25.141840 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:25.141802 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93c780c7-a1ca-4b63-86bb-7b59dd49745a-openshift-service-ca\") pod \"maas-api-e2e-api-a-2ee874-fcdcfc47c-ztm82\" (UID: \"93c780c7-a1ca-4b63-86bb-7b59dd49745a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ee874-fcdcfc47c-ztm82" Oct 30 18:19:25.183203 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:25.180053 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93c780c7-a1ca-4b63-86bb-7b59dd49745a-metrics-certs\") pod \"maas-api-e2e-api-a-2ee874-fcdcfc47c-ztm82\" (UID: \"93c780c7-a1ca-4b63-86bb-7b59dd49745a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ee874-fcdcfc47c-ztm82" Oct 30 18:19:25.206125 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:25.204573 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mqwb\" (UniqueName: \"kubernetes.io/projected/93c780c7-a1ca-4b63-86bb-7b59dd49745a-kube-api-access-6mqwb\") pod \"maas-api-e2e-api-a-2ee874-fcdcfc47c-ztm82\" (UID: \"93c780c7-a1ca-4b63-86bb-7b59dd49745a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ee874-fcdcfc47c-ztm82" Oct 30 18:19:25.636300 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:25.636264 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93c780c7-a1ca-4b63-86bb-7b59dd49745a-maas-api-tls\") pod \"maas-api-e2e-api-a-2ee874-fcdcfc47c-ztm82\" (UID: \"93c780c7-a1ca-4b63-86bb-7b59dd49745a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ee874-fcdcfc47c-ztm82" Oct 30 18:19:25.639080 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:25.639052 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93c780c7-a1ca-4b63-86bb-7b59dd49745a-maas-api-tls\") pod \"maas-api-e2e-api-a-2ee874-fcdcfc47c-ztm82\" (UID: \"93c780c7-a1ca-4b63-86bb-7b59dd49745a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ee874-fcdcfc47c-ztm82" Oct 30 18:19:25.700928 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:25.700900 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-2ee874-5b975bd4-drq62" Oct 30 18:19:25.701092 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:25.700965 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ee874-69fddfb67-hscz8" Oct 30 18:19:25.847147 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:25.847100 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ee874-fcdcfc47c-ztm82" Oct 30 18:19:26.435517 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:26.435172 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2ee874-fcdcfc47c-ztm82"] Oct 30 18:19:26.438223 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:19:26.438190 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93c780c7_a1ca_4b63_86bb_7b59dd49745a.slice/crio-cfa475aa72ddc9512b6bc4b5921a9ad46e88daf9672116b47de272c3ae776eb5 WatchSource:0}: Error finding container cfa475aa72ddc9512b6bc4b5921a9ad46e88daf9672116b47de272c3ae776eb5: Status 404 returned error can't find the container with id cfa475aa72ddc9512b6bc4b5921a9ad46e88daf9672116b47de272c3ae776eb5 Oct 30 18:19:26.888943 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:26.888818 4442 generic.go:356] "Generic (PLEG): container finished" podID="0e328cf8-f89f-4f08-aafb-d481dd61ffd3" containerID="eb55ad414aa08dd5097f9744b4f7d9aa6db78bc4d68769c7639e83429203ea50" exitCode=1 Oct 30 18:19:26.888943 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:26.888880 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh" event={"ID":"0e328cf8-f89f-4f08-aafb-d481dd61ffd3","Type":"ContainerDied","Data":"eb55ad414aa08dd5097f9744b4f7d9aa6db78bc4d68769c7639e83429203ea50"} Oct 30 18:19:26.889461 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:26.889443 4442 scope.go:122] "RemoveContainer" containerID="eb55ad414aa08dd5097f9744b4f7d9aa6db78bc4d68769c7639e83429203ea50" Oct 30 18:19:26.891114 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:26.891094 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ee874-fcdcfc47c-ztm82" event={"ID":"93c780c7-a1ca-4b63-86bb-7b59dd49745a","Type":"ContainerStarted","Data":"cfa475aa72ddc9512b6bc4b5921a9ad46e88daf9672116b47de272c3ae776eb5"} Oct 30 18:19:27.907489 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:27.907453 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh" event={"ID":"0e328cf8-f89f-4f08-aafb-d481dd61ffd3","Type":"ContainerStarted","Data":"cd5da74c7bd9c4b5075b5a61d6fa969c22e3866dda44d564f389f9eca9ffd66f"} Oct 30 18:19:27.907900 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:27.907684 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh" Oct 30 18:19:27.909154 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:27.909118 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ee874-fcdcfc47c-ztm82" event={"ID":"93c780c7-a1ca-4b63-86bb-7b59dd49745a","Type":"ContainerStarted","Data":"9a573d62f13e740c78fdbcbd9604f16ba7d9372de24290431104fffb16938cf4"} Oct 30 18:19:27.909254 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:27.909187 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ee874-fcdcfc47c-ztm82" Oct 30 18:19:27.990610 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:27.990558 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ee874-fcdcfc47c-ztm82" podStartSLOduration=3.628326067 podStartE2EDuration="3.990544162s" totalImagesPullingTime="362.218095ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:27.987198659 +0000 UTC m=+1265.077261000" watchObservedRunningTime="2026-10-30 18:19:27.990544162 +0000 UTC m=+1265.080606500" Oct 30 18:19:29.868026 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:29.867988 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8dc43c-884559b4-hgppv"] Oct 30 18:19:29.868631 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:29.868257 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8dc43c-884559b4-hgppv" podUID="d859a2eb-5149-4c88-a99b-85e08c7aaf2b" containerName="maas-api" containerID="cri-o://21dae01a25b325f96079a234df55bd1c05725d845a5b039e0244b22e0b4a8b75" gracePeriod=30 Oct 30 18:19:30.335177 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:30.331398 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8dc43c-884559b4-hgppv" Oct 30 18:19:30.429205 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:30.419522 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d859a2eb-5149-4c88-a99b-85e08c7aaf2b-kube-api-access-drxdw\" (UniqueName: \"kubernetes.io/projected/d859a2eb-5149-4c88-a99b-85e08c7aaf2b-kube-api-access-drxdw\") pod \"d859a2eb-5149-4c88-a99b-85e08c7aaf2b\" (UID: \"d859a2eb-5149-4c88-a99b-85e08c7aaf2b\") " Oct 30 18:19:30.429205 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:30.419763 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d859a2eb-5149-4c88-a99b-85e08c7aaf2b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d859a2eb-5149-4c88-a99b-85e08c7aaf2b-maas-api-tls\") pod \"d859a2eb-5149-4c88-a99b-85e08c7aaf2b\" (UID: \"d859a2eb-5149-4c88-a99b-85e08c7aaf2b\") " Oct 30 18:19:30.429205 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:30.419818 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d859a2eb-5149-4c88-a99b-85e08c7aaf2b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d859a2eb-5149-4c88-a99b-85e08c7aaf2b-metrics-certs\") pod \"d859a2eb-5149-4c88-a99b-85e08c7aaf2b\" (UID: \"d859a2eb-5149-4c88-a99b-85e08c7aaf2b\") " Oct 30 18:19:30.429205 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:30.419876 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d859a2eb-5149-4c88-a99b-85e08c7aaf2b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d859a2eb-5149-4c88-a99b-85e08c7aaf2b-openshift-service-ca\") pod \"d859a2eb-5149-4c88-a99b-85e08c7aaf2b\" (UID: \"d859a2eb-5149-4c88-a99b-85e08c7aaf2b\") " Oct 30 18:19:30.520211 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:30.505493 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d859a2eb-5149-4c88-a99b-85e08c7aaf2b-openshift-service-ca" pod "d859a2eb-5149-4c88-a99b-85e08c7aaf2b" (UID: "d859a2eb-5149-4c88-a99b-85e08c7aaf2b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:30.537380 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:30.535626 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d859a2eb-5149-4c88-a99b-85e08c7aaf2b-openshift-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:19:30.631166 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:30.627482 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d859a2eb-5149-4c88-a99b-85e08c7aaf2b-metrics-certs" pod "d859a2eb-5149-4c88-a99b-85e08c7aaf2b" (UID: "d859a2eb-5149-4c88-a99b-85e08c7aaf2b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:30.667162 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:30.666202 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d859a2eb-5149-4c88-a99b-85e08c7aaf2b-metrics-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:19:30.675160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:30.674332 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d859a2eb-5149-4c88-a99b-85e08c7aaf2b-maas-api-tls" pod "d859a2eb-5149-4c88-a99b-85e08c7aaf2b" (UID: "d859a2eb-5149-4c88-a99b-85e08c7aaf2b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:30.675160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:30.674907 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d859a2eb-5149-4c88-a99b-85e08c7aaf2b-kube-api-access-drxdw" pod "d859a2eb-5149-4c88-a99b-85e08c7aaf2b" (UID: "d859a2eb-5149-4c88-a99b-85e08c7aaf2b"). InnerVolumeSpecName "kube-api-access-drxdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:30.767299 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:30.767268 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d859a2eb-5149-4c88-a99b-85e08c7aaf2b-maas-api-tls\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:19:30.767299 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:30.767296 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drxdw\" (UniqueName: \"kubernetes.io/projected/d859a2eb-5149-4c88-a99b-85e08c7aaf2b-kube-api-access-drxdw\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:19:30.943437 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:30.943400 4442 generic.go:356] "Generic (PLEG): container finished" podID="d859a2eb-5149-4c88-a99b-85e08c7aaf2b" containerID="21dae01a25b325f96079a234df55bd1c05725d845a5b039e0244b22e0b4a8b75" exitCode=0 Oct 30 18:19:30.943889 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:30.943450 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8dc43c-884559b4-hgppv" Oct 30 18:19:30.943889 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:30.943488 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8dc43c-884559b4-hgppv" event={"ID":"d859a2eb-5149-4c88-a99b-85e08c7aaf2b","Type":"ContainerDied","Data":"21dae01a25b325f96079a234df55bd1c05725d845a5b039e0244b22e0b4a8b75"} Oct 30 18:19:30.943889 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:30.943549 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8dc43c-884559b4-hgppv" event={"ID":"d859a2eb-5149-4c88-a99b-85e08c7aaf2b","Type":"ContainerDied","Data":"e0f2cceffe8d477b716c78b880673e06100fa90fcc24943f6acf055f15bdc75e"} Oct 30 18:19:30.943889 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:30.943579 4442 scope.go:122] "RemoveContainer" containerID="21dae01a25b325f96079a234df55bd1c05725d845a5b039e0244b22e0b4a8b75" Oct 30 18:19:30.956446 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:30.956423 4442 scope.go:122] "RemoveContainer" containerID="21dae01a25b325f96079a234df55bd1c05725d845a5b039e0244b22e0b4a8b75" Oct 30 18:19:30.956770 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:19:30.956751 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21dae01a25b325f96079a234df55bd1c05725d845a5b039e0244b22e0b4a8b75\": container with ID starting with 21dae01a25b325f96079a234df55bd1c05725d845a5b039e0244b22e0b4a8b75 not found: ID does not exist" containerID="21dae01a25b325f96079a234df55bd1c05725d845a5b039e0244b22e0b4a8b75" Oct 30 18:19:30.956817 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:30.956781 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21dae01a25b325f96079a234df55bd1c05725d845a5b039e0244b22e0b4a8b75"} err="failed to get container status \"21dae01a25b325f96079a234df55bd1c05725d845a5b039e0244b22e0b4a8b75\": rpc error: code = NotFound desc = could not find container \"21dae01a25b325f96079a234df55bd1c05725d845a5b039e0244b22e0b4a8b75\": container with ID starting with 21dae01a25b325f96079a234df55bd1c05725d845a5b039e0244b22e0b4a8b75 not found: ID does not exist" Oct 30 18:19:31.088398 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:31.081978 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8dc43c-884559b4-hgppv"] Oct 30 18:19:31.101651 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:31.101621 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8dc43c-884559b4-hgppv"] Oct 30 18:19:31.841177 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:31.841128 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f42594-7c6f78d84-zkpr2" Oct 30 18:19:32.731512 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:32.731478 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f42594-666cc4c777-dm62x"] Oct 30 18:19:32.732261 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:32.732244 4442 cpu_manager.go:414] "containerMap: removing container" podUID="d859a2eb-5149-4c88-a99b-85e08c7aaf2b" containerName="maas-api" Oct 30 18:19:32.732316 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:32.732263 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d859a2eb-5149-4c88-a99b-85e08c7aaf2b" containerName="maas-api" Oct 30 18:19:32.732413 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:32.732404 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="d859a2eb-5149-4c88-a99b-85e08c7aaf2b" containerName="maas-api" Oct 30 18:19:32.737753 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:32.737734 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f42594-666cc4c777-dm62x" Oct 30 18:19:32.755567 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:32.755541 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f42594-666cc4c777-dm62x"] Oct 30 18:19:32.762409 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:32.762383 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-f42594-serving-cert\"" Oct 30 18:19:32.824477 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:32.824433 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f99c368a-c8a3-49be-afd7-c68980a7fa39-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f42594-666cc4c777-dm62x\" (UID: \"f99c368a-c8a3-49be-afd7-c68980a7fa39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f42594-666cc4c777-dm62x" Oct 30 18:19:32.824658 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:32.824533 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f99c368a-c8a3-49be-afd7-c68980a7fa39-metrics-certs\") pod \"maas-api-e2e-ipp-a-f42594-666cc4c777-dm62x\" (UID: \"f99c368a-c8a3-49be-afd7-c68980a7fa39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f42594-666cc4c777-dm62x" Oct 30 18:19:32.824658 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:32.824566 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f99c368a-c8a3-49be-afd7-c68980a7fa39-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f42594-666cc4c777-dm62x\" (UID: \"f99c368a-c8a3-49be-afd7-c68980a7fa39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f42594-666cc4c777-dm62x" Oct 30 18:19:32.824738 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:32.824663 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phtls\" (UniqueName: \"kubernetes.io/projected/f99c368a-c8a3-49be-afd7-c68980a7fa39-kube-api-access-phtls\") pod \"maas-api-e2e-ipp-a-f42594-666cc4c777-dm62x\" (UID: \"f99c368a-c8a3-49be-afd7-c68980a7fa39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f42594-666cc4c777-dm62x" Oct 30 18:19:32.856501 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:32.856470 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-5669779bc-7249d" Oct 30 18:19:32.926826 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:32.926778 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f99c368a-c8a3-49be-afd7-c68980a7fa39-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f42594-666cc4c777-dm62x\" (UID: \"f99c368a-c8a3-49be-afd7-c68980a7fa39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f42594-666cc4c777-dm62x" Oct 30 18:19:32.927027 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:32.926868 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f99c368a-c8a3-49be-afd7-c68980a7fa39-metrics-certs\") pod \"maas-api-e2e-ipp-a-f42594-666cc4c777-dm62x\" (UID: \"f99c368a-c8a3-49be-afd7-c68980a7fa39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f42594-666cc4c777-dm62x" Oct 30 18:19:32.927027 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:32.926924 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f99c368a-c8a3-49be-afd7-c68980a7fa39-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f42594-666cc4c777-dm62x\" (UID: \"f99c368a-c8a3-49be-afd7-c68980a7fa39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f42594-666cc4c777-dm62x" Oct 30 18:19:32.927027 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:32.927023 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phtls\" (UniqueName: \"kubernetes.io/projected/f99c368a-c8a3-49be-afd7-c68980a7fa39-kube-api-access-phtls\") pod \"maas-api-e2e-ipp-a-f42594-666cc4c777-dm62x\" (UID: \"f99c368a-c8a3-49be-afd7-c68980a7fa39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f42594-666cc4c777-dm62x" Oct 30 18:19:32.927653 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:32.927610 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f99c368a-c8a3-49be-afd7-c68980a7fa39-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f42594-666cc4c777-dm62x\" (UID: \"f99c368a-c8a3-49be-afd7-c68980a7fa39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f42594-666cc4c777-dm62x" Oct 30 18:19:32.931032 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:32.931005 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f99c368a-c8a3-49be-afd7-c68980a7fa39-metrics-certs\") pod \"maas-api-e2e-ipp-a-f42594-666cc4c777-dm62x\" (UID: \"f99c368a-c8a3-49be-afd7-c68980a7fa39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f42594-666cc4c777-dm62x" Oct 30 18:19:32.931212 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:32.931005 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f99c368a-c8a3-49be-afd7-c68980a7fa39-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f42594-666cc4c777-dm62x\" (UID: \"f99c368a-c8a3-49be-afd7-c68980a7fa39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f42594-666cc4c777-dm62x" Oct 30 18:19:32.948262 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:32.948230 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phtls\" (UniqueName: \"kubernetes.io/projected/f99c368a-c8a3-49be-afd7-c68980a7fa39-kube-api-access-phtls\") pod \"maas-api-e2e-ipp-a-f42594-666cc4c777-dm62x\" (UID: \"f99c368a-c8a3-49be-afd7-c68980a7fa39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f42594-666cc4c777-dm62x" Oct 30 18:19:33.055211 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:33.053732 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f42594-666cc4c777-dm62x" Oct 30 18:19:33.221284 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:33.221251 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d859a2eb-5149-4c88-a99b-85e08c7aaf2b" path="/var/lib/kubelet/pods/d859a2eb-5149-4c88-a99b-85e08c7aaf2b/volumes" Oct 30 18:19:33.550097 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:33.550069 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f42594-666cc4c777-dm62x"] Oct 30 18:19:33.551144 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:19:33.551101 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf99c368a_c8a3_49be_afd7_c68980a7fa39.slice/crio-527a8d52504eccbef3d4d421c2236c0607e18eb1a039bd45832c77e54ccfdb77 WatchSource:0}: Error finding container 527a8d52504eccbef3d4d421c2236c0607e18eb1a039bd45832c77e54ccfdb77: Status 404 returned error can't find the container with id 527a8d52504eccbef3d4d421c2236c0607e18eb1a039bd45832c77e54ccfdb77 Oct 30 18:19:33.867212 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:33.866414 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-84c84577c5-q85dn" Oct 30 18:19:33.952432 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:33.932499 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9096a-685c586796-k9fgv" Oct 30 18:19:33.952432 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:33.945318 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ee874-fcdcfc47c-ztm82" Oct 30 18:19:33.957498 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:33.955397 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh" Oct 30 18:19:34.162039 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:34.161977 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f42594-666cc4c777-dm62x" event={"ID":"f99c368a-c8a3-49be-afd7-c68980a7fa39","Type":"ContainerStarted","Data":"527a8d52504eccbef3d4d421c2236c0607e18eb1a039bd45832c77e54ccfdb77"} Oct 30 18:19:34.555572 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:34.555491 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9096a-7946dfc647-x8bk4"] Oct 30 18:19:34.560053 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:34.560031 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9096a-7946dfc647-x8bk4" Oct 30 18:19:34.575942 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:34.575911 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-e9096a-serving-cert\"" Oct 30 18:19:34.582562 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:34.582531 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9096a-7946dfc647-x8bk4"] Oct 30 18:19:34.658495 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:34.658457 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68fzg\" (UniqueName: \"kubernetes.io/projected/b9770481-db87-4fed-ad1e-3c261452983f-kube-api-access-68fzg\") pod \"maas-api-e2e-ipp-a-e9096a-7946dfc647-x8bk4\" (UID: \"b9770481-db87-4fed-ad1e-3c261452983f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9096a-7946dfc647-x8bk4" Oct 30 18:19:34.658726 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:34.658705 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9770481-db87-4fed-ad1e-3c261452983f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e9096a-7946dfc647-x8bk4\" (UID: \"b9770481-db87-4fed-ad1e-3c261452983f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9096a-7946dfc647-x8bk4" Oct 30 18:19:34.658855 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:34.658832 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9770481-db87-4fed-ad1e-3c261452983f-metrics-certs\") pod \"maas-api-e2e-ipp-a-e9096a-7946dfc647-x8bk4\" (UID: \"b9770481-db87-4fed-ad1e-3c261452983f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9096a-7946dfc647-x8bk4" Oct 30 18:19:34.659013 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:34.658994 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9770481-db87-4fed-ad1e-3c261452983f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e9096a-7946dfc647-x8bk4\" (UID: \"b9770481-db87-4fed-ad1e-3c261452983f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9096a-7946dfc647-x8bk4" Oct 30 18:19:34.760424 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:34.760383 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9770481-db87-4fed-ad1e-3c261452983f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e9096a-7946dfc647-x8bk4\" (UID: \"b9770481-db87-4fed-ad1e-3c261452983f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9096a-7946dfc647-x8bk4" Oct 30 18:19:34.760616 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:34.760489 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68fzg\" (UniqueName: \"kubernetes.io/projected/b9770481-db87-4fed-ad1e-3c261452983f-kube-api-access-68fzg\") pod \"maas-api-e2e-ipp-a-e9096a-7946dfc647-x8bk4\" (UID: \"b9770481-db87-4fed-ad1e-3c261452983f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9096a-7946dfc647-x8bk4" Oct 30 18:19:34.760683 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:34.760622 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9770481-db87-4fed-ad1e-3c261452983f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e9096a-7946dfc647-x8bk4\" (UID: \"b9770481-db87-4fed-ad1e-3c261452983f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9096a-7946dfc647-x8bk4" Oct 30 18:19:34.760741 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:34.760682 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9770481-db87-4fed-ad1e-3c261452983f-metrics-certs\") pod \"maas-api-e2e-ipp-a-e9096a-7946dfc647-x8bk4\" (UID: \"b9770481-db87-4fed-ad1e-3c261452983f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9096a-7946dfc647-x8bk4" Oct 30 18:19:34.761161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:34.761107 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9770481-db87-4fed-ad1e-3c261452983f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e9096a-7946dfc647-x8bk4\" (UID: \"b9770481-db87-4fed-ad1e-3c261452983f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9096a-7946dfc647-x8bk4" Oct 30 18:19:34.763519 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:34.763472 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9770481-db87-4fed-ad1e-3c261452983f-metrics-certs\") pod \"maas-api-e2e-ipp-a-e9096a-7946dfc647-x8bk4\" (UID: \"b9770481-db87-4fed-ad1e-3c261452983f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9096a-7946dfc647-x8bk4" Oct 30 18:19:34.763679 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:34.763627 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9770481-db87-4fed-ad1e-3c261452983f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e9096a-7946dfc647-x8bk4\" (UID: \"b9770481-db87-4fed-ad1e-3c261452983f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9096a-7946dfc647-x8bk4" Oct 30 18:19:34.826211 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:34.826041 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68fzg\" (UniqueName: \"kubernetes.io/projected/b9770481-db87-4fed-ad1e-3c261452983f-kube-api-access-68fzg\") pod \"maas-api-e2e-ipp-a-e9096a-7946dfc647-x8bk4\" (UID: \"b9770481-db87-4fed-ad1e-3c261452983f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9096a-7946dfc647-x8bk4" Oct 30 18:19:34.877853 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:34.877435 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9096a-7946dfc647-x8bk4" Oct 30 18:19:35.173224 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:35.173172 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f42594-666cc4c777-dm62x" event={"ID":"f99c368a-c8a3-49be-afd7-c68980a7fa39","Type":"ContainerStarted","Data":"2fc426065f9cb3e2637ddd913495ca5ea97494588545b1a9e82a8c576126afbd"} Oct 30 18:19:35.173507 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:35.173483 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f42594-666cc4c777-dm62x" Oct 30 18:19:35.212716 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:35.212657 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f42594-666cc4c777-dm62x" podStartSLOduration=2.747789377 podStartE2EDuration="3.212640543s" totalImagesPullingTime="464.851166ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:35.211407956 +0000 UTC m=+1272.301470310" watchObservedRunningTime="2026-10-30 18:19:35.212640543 +0000 UTC m=+1272.302702882" Oct 30 18:19:35.265191 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:35.265165 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9096a-7946dfc647-x8bk4"] Oct 30 18:19:35.267592 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:19:35.267559 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9770481_db87_4fed_ad1e_3c261452983f.slice/crio-aa71781e5f0a94da37bda9e1781641a1199ea457f7e5d044ff7df8a46326f7a8 WatchSource:0}: Error finding container aa71781e5f0a94da37bda9e1781641a1199ea457f7e5d044ff7df8a46326f7a8: Status 404 returned error can't find the container with id aa71781e5f0a94da37bda9e1781641a1199ea457f7e5d044ff7df8a46326f7a8 Oct 30 18:19:36.183853 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:36.183816 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9096a-7946dfc647-x8bk4" event={"ID":"b9770481-db87-4fed-ad1e-3c261452983f","Type":"ContainerStarted","Data":"be603036fa2f3e7eff39cbe679a08fa3a90b973b86726dde62f2965763c666fe"} Oct 30 18:19:36.184333 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:36.183878 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9096a-7946dfc647-x8bk4" event={"ID":"b9770481-db87-4fed-ad1e-3c261452983f","Type":"ContainerStarted","Data":"aa71781e5f0a94da37bda9e1781641a1199ea457f7e5d044ff7df8a46326f7a8"} Oct 30 18:19:36.184333 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:36.183907 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9096a-7946dfc647-x8bk4" Oct 30 18:19:36.224862 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:36.224797 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9096a-7946dfc647-x8bk4" podStartSLOduration=1.887783281 podStartE2EDuration="2.224775309s" totalImagesPullingTime="336.992028ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:36.222874871 +0000 UTC m=+1273.312937208" watchObservedRunningTime="2026-10-30 18:19:36.224775309 +0000 UTC m=+1273.314837655" Oct 30 18:19:41.189189 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:41.189156 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f42594-666cc4c777-dm62x" Oct 30 18:19:42.197511 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:42.197480 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9096a-7946dfc647-x8bk4" Oct 30 18:19:47.743156 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:19:47.742378 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:19:55.495117 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:55.494840 4442 patch_prober.go:29] interesting pod/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": read tcp 10.134.0.2:48112->10.134.0.109:15021: read: connection reset by peer" start-of-body= Oct 30 18:19:55.495117 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:55.494920 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" podUID="64bde5b5-0bc5-42b8-9aa9-fc6495afca20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": read tcp 10.134.0.2:48112->10.134.0.109:15021: read: connection reset by peer" Oct 30 18:19:56.485876 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:56.485844 4442 generic.go:356] "Generic (PLEG): container finished" podID="64bde5b5-0bc5-42b8-9aa9-fc6495afca20" containerID="267409ebd1bf2a05467b52052560a601115e9a93eb2d80d7effd332260f35161" exitCode=0 Oct 30 18:19:56.486102 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:56.485920 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" event={"ID":"64bde5b5-0bc5-42b8-9aa9-fc6495afca20","Type":"ContainerDied","Data":"267409ebd1bf2a05467b52052560a601115e9a93eb2d80d7effd332260f35161"} Oct 30 18:19:56.486437 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:56.486421 4442 scope.go:122] "RemoveContainer" containerID="267409ebd1bf2a05467b52052560a601115e9a93eb2d80d7effd332260f35161" Oct 30 18:19:56.488235 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:56.488210 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:19:56.488321 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:56.488278 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:19:56.488321 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:56.488312 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:19:57.499593 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:57.499561 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" event={"ID":"64bde5b5-0bc5-42b8-9aa9-fc6495afca20","Type":"ContainerStarted","Data":"163fe66746c6646fae1b65c5d4cfb6430eeb035a852193945dece5d1a36c1b29"} Oct 30 18:19:57.664276 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:57.664248 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" Oct 30 18:19:57.664460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:57.664429 4442 patch_prober.go:29] interesting pod/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 18:19:57.664504 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:57.664470 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" podUID="64bde5b5-0bc5-42b8-9aa9-fc6495afca20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 18:19:57.720182 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:57.720149 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f42594-7c6f78d84-zkpr2"] Oct 30 18:19:57.755816 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:57.755735 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-5669779bc-7249d"] Oct 30 18:19:58.691448 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:58.684398 4442 patch_prober.go:29] interesting pod/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 18:19:58.691448 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:58.684453 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" podUID="64bde5b5-0bc5-42b8-9aa9-fc6495afca20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 18:19:59.665103 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:59.665070 4442 patch_prober.go:29] interesting pod/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 18:19:59.665299 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:19:59.665118 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" podUID="64bde5b5-0bc5-42b8-9aa9-fc6495afca20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 18:20:00.664742 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:00.664707 4442 patch_prober.go:29] interesting pod/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 18:20:00.665307 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:00.664788 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" podUID="64bde5b5-0bc5-42b8-9aa9-fc6495afca20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 18:20:01.664670 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:01.664627 4442 patch_prober.go:29] interesting pod/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 18:20:01.664998 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:01.664678 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" podUID="64bde5b5-0bc5-42b8-9aa9-fc6495afca20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 18:20:02.708428 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:02.708369 4442 patch_prober.go:29] interesting pod/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 18:20:02.709038 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:02.708998 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" podUID="64bde5b5-0bc5-42b8-9aa9-fc6495afca20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 18:20:02.787191 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:02.779022 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-5669779bc-7249d" podUID="c1461e0a-7e0f-4942-87b4-b2b160dbc10f" containerName="payload-pre-processing" containerID="cri-o://5302010b7d675e4e1ead430fc997fa61ece8d4d400015f847c5c8c578815fe16" gracePeriod=25 Oct 30 18:20:02.909247 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:02.725902 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f42594-7c6f78d84-zkpr2" podUID="0c0da160-9bbb-4f72-9d56-b54a269a6496" containerName="payload-processing" containerID="cri-o://9fb27c601f129a6ef74635a1b464ba760b71e48aada717311ff2e02814db0eaf" gracePeriod=25 Oct 30 18:20:03.372849 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:03.372490 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f42594-7c6f78d84-zkpr2" Oct 30 18:20:03.568213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:03.405023 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c0da160-9bbb-4f72-9d56-b54a269a6496-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c0da160-9bbb-4f72-9d56-b54a269a6496-plugins-config-volume\") pod \"0c0da160-9bbb-4f72-9d56-b54a269a6496\" (UID: \"0c0da160-9bbb-4f72-9d56-b54a269a6496\") " Oct 30 18:20:03.568213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:03.405232 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c0da160-9bbb-4f72-9d56-b54a269a6496-kube-api-access-9qkbd\" (UniqueName: \"kubernetes.io/projected/0c0da160-9bbb-4f72-9d56-b54a269a6496-kube-api-access-9qkbd\") pod \"0c0da160-9bbb-4f72-9d56-b54a269a6496\" (UID: \"0c0da160-9bbb-4f72-9d56-b54a269a6496\") " Oct 30 18:20:03.568213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:03.548380 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c0da160-9bbb-4f72-9d56-b54a269a6496-plugins-config-volume" pod "0c0da160-9bbb-4f72-9d56-b54a269a6496" (UID: "0c0da160-9bbb-4f72-9d56-b54a269a6496"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:20:03.624181 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:03.568838 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c0da160-9bbb-4f72-9d56-b54a269a6496-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c0da160-9bbb-4f72-9d56-b54a269a6496-plugins-config-volume\") pod \"0c0da160-9bbb-4f72-9d56-b54a269a6496\" (UID: \"0c0da160-9bbb-4f72-9d56-b54a269a6496\") " Oct 30 18:20:03.624181 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:20:03.569362 4442 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/0c0da160-9bbb-4f72-9d56-b54a269a6496/volumes/kubernetes.io~configmap/plugins-config-volume Oct 30 18:20:03.624181 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:03.569385 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c0da160-9bbb-4f72-9d56-b54a269a6496-plugins-config-volume" pod "0c0da160-9bbb-4f72-9d56-b54a269a6496" (UID: "0c0da160-9bbb-4f72-9d56-b54a269a6496"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:20:03.624181 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:03.569652 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c0da160-9bbb-4f72-9d56-b54a269a6496-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:03.770646 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:03.770604 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c0da160-9bbb-4f72-9d56-b54a269a6496-kube-api-access-9qkbd" pod "0c0da160-9bbb-4f72-9d56-b54a269a6496" (UID: "0c0da160-9bbb-4f72-9d56-b54a269a6496"). InnerVolumeSpecName "kube-api-access-9qkbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:20:03.793155 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:03.793069 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qkbd\" (UniqueName: \"kubernetes.io/projected/0c0da160-9bbb-4f72-9d56-b54a269a6496-kube-api-access-9qkbd\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:03.795107 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:03.795082 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" Oct 30 18:20:03.797109 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:03.795963 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" Oct 30 18:20:03.797109 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:03.796903 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-5669779bc-7249d" Oct 30 18:20:03.797783 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:03.797765 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" Oct 30 18:20:03.894511 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:03.894481 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1461e0a-7e0f-4942-87b4-b2b160dbc10f-kube-api-access-whtmk\" (UniqueName: \"kubernetes.io/projected/c1461e0a-7e0f-4942-87b4-b2b160dbc10f-kube-api-access-whtmk\") pod \"c1461e0a-7e0f-4942-87b4-b2b160dbc10f\" (UID: \"c1461e0a-7e0f-4942-87b4-b2b160dbc10f\") " Oct 30 18:20:03.894704 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:03.894611 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1461e0a-7e0f-4942-87b4-b2b160dbc10f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1461e0a-7e0f-4942-87b4-b2b160dbc10f-plugins-config-volume\") pod \"c1461e0a-7e0f-4942-87b4-b2b160dbc10f\" (UID: \"c1461e0a-7e0f-4942-87b4-b2b160dbc10f\") " Oct 30 18:20:03.895019 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:03.894983 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1461e0a-7e0f-4942-87b4-b2b160dbc10f-plugins-config-volume" pod "c1461e0a-7e0f-4942-87b4-b2b160dbc10f" (UID: "c1461e0a-7e0f-4942-87b4-b2b160dbc10f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:20:03.895388 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:03.895361 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1461e0a-7e0f-4942-87b4-b2b160dbc10f-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:03.896854 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:03.896830 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1461e0a-7e0f-4942-87b4-b2b160dbc10f-kube-api-access-whtmk" pod "c1461e0a-7e0f-4942-87b4-b2b160dbc10f" (UID: "c1461e0a-7e0f-4942-87b4-b2b160dbc10f"). InnerVolumeSpecName "kube-api-access-whtmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:20:03.922724 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:03.922687 4442 generic.go:356] "Generic (PLEG): container finished" podID="0c0da160-9bbb-4f72-9d56-b54a269a6496" containerID="9fb27c601f129a6ef74635a1b464ba760b71e48aada717311ff2e02814db0eaf" exitCode=0 Oct 30 18:20:03.922910 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:03.922742 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f42594-7c6f78d84-zkpr2" event={"ID":"0c0da160-9bbb-4f72-9d56-b54a269a6496","Type":"ContainerDied","Data":"9fb27c601f129a6ef74635a1b464ba760b71e48aada717311ff2e02814db0eaf"} Oct 30 18:20:03.922910 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:03.922755 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f42594-7c6f78d84-zkpr2" Oct 30 18:20:03.922910 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:03.922821 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f42594-7c6f78d84-zkpr2" event={"ID":"0c0da160-9bbb-4f72-9d56-b54a269a6496","Type":"ContainerDied","Data":"4b23eb623e5719c983ac8599cc101688ca694026c911519686cdcbca9ecc10cc"} Oct 30 18:20:03.922910 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:03.922854 4442 scope.go:122] "RemoveContainer" containerID="9fb27c601f129a6ef74635a1b464ba760b71e48aada717311ff2e02814db0eaf" Oct 30 18:20:03.925988 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:03.925687 4442 generic.go:356] "Generic (PLEG): container finished" podID="c1461e0a-7e0f-4942-87b4-b2b160dbc10f" containerID="5302010b7d675e4e1ead430fc997fa61ece8d4d400015f847c5c8c578815fe16" exitCode=0 Oct 30 18:20:03.925988 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:03.925840 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-5669779bc-7249d" Oct 30 18:20:03.925988 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:03.925906 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-5669779bc-7249d" event={"ID":"c1461e0a-7e0f-4942-87b4-b2b160dbc10f","Type":"ContainerDied","Data":"5302010b7d675e4e1ead430fc997fa61ece8d4d400015f847c5c8c578815fe16"} Oct 30 18:20:03.925988 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:03.925947 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-5669779bc-7249d" event={"ID":"c1461e0a-7e0f-4942-87b4-b2b160dbc10f","Type":"ContainerDied","Data":"2128d3e48ce1913e664310768c8d3f39d8e51c076278f88cbb61ad040961e841"} Oct 30 18:20:03.951439 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:03.951417 4442 scope.go:122] "RemoveContainer" containerID="9fb27c601f129a6ef74635a1b464ba760b71e48aada717311ff2e02814db0eaf" Oct 30 18:20:03.951804 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:20:03.951777 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fb27c601f129a6ef74635a1b464ba760b71e48aada717311ff2e02814db0eaf\": container with ID starting with 9fb27c601f129a6ef74635a1b464ba760b71e48aada717311ff2e02814db0eaf not found: ID does not exist" containerID="9fb27c601f129a6ef74635a1b464ba760b71e48aada717311ff2e02814db0eaf" Oct 30 18:20:03.951932 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:03.951818 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fb27c601f129a6ef74635a1b464ba760b71e48aada717311ff2e02814db0eaf"} err="failed to get container status \"9fb27c601f129a6ef74635a1b464ba760b71e48aada717311ff2e02814db0eaf\": rpc error: code = NotFound desc = could not find container \"9fb27c601f129a6ef74635a1b464ba760b71e48aada717311ff2e02814db0eaf\": container with ID starting with 9fb27c601f129a6ef74635a1b464ba760b71e48aada717311ff2e02814db0eaf not found: ID does not exist" Oct 30 18:20:03.951932 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:03.951847 4442 scope.go:122] "RemoveContainer" containerID="5302010b7d675e4e1ead430fc997fa61ece8d4d400015f847c5c8c578815fe16" Oct 30 18:20:03.991176 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:03.980337 4442 scope.go:122] "RemoveContainer" containerID="5302010b7d675e4e1ead430fc997fa61ece8d4d400015f847c5c8c578815fe16" Oct 30 18:20:03.991176 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:20:03.980891 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5302010b7d675e4e1ead430fc997fa61ece8d4d400015f847c5c8c578815fe16\": container with ID starting with 5302010b7d675e4e1ead430fc997fa61ece8d4d400015f847c5c8c578815fe16 not found: ID does not exist" containerID="5302010b7d675e4e1ead430fc997fa61ece8d4d400015f847c5c8c578815fe16" Oct 30 18:20:03.991176 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:03.980944 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5302010b7d675e4e1ead430fc997fa61ece8d4d400015f847c5c8c578815fe16"} err="failed to get container status \"5302010b7d675e4e1ead430fc997fa61ece8d4d400015f847c5c8c578815fe16\": rpc error: code = NotFound desc = could not find container \"5302010b7d675e4e1ead430fc997fa61ece8d4d400015f847c5c8c578815fe16\": container with ID starting with 5302010b7d675e4e1ead430fc997fa61ece8d4d400015f847c5c8c578815fe16 not found: ID does not exist" Oct 30 18:20:04.008620 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:04.000244 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whtmk\" (UniqueName: \"kubernetes.io/projected/c1461e0a-7e0f-4942-87b4-b2b160dbc10f-kube-api-access-whtmk\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:04.055581 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:04.055380 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-5669779bc-7249d"] Oct 30 18:20:04.064252 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:04.064218 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-5669779bc-7249d"] Oct 30 18:20:04.086224 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:04.085275 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f42594-7c6f78d84-zkpr2"] Oct 30 18:20:04.091417 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:04.091390 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f42594-7c6f78d84-zkpr2"] Oct 30 18:20:05.069109 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:05.069033 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c0da160-9bbb-4f72-9d56-b54a269a6496" path="/var/lib/kubelet/pods/0c0da160-9bbb-4f72-9d56-b54a269a6496/volumes" Oct 30 18:20:05.069621 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:05.069605 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1461e0a-7e0f-4942-87b4-b2b160dbc10f" path="/var/lib/kubelet/pods/c1461e0a-7e0f-4942-87b4-b2b160dbc10f/volumes" Oct 30 18:20:05.102461 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:05.102432 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-7d8df6544c-mlnfd"] Oct 30 18:20:05.103177 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:05.103122 4442 cpu_manager.go:414] "containerMap: removing container" podUID="0c0da160-9bbb-4f72-9d56-b54a269a6496" containerName="payload-processing" Oct 30 18:20:05.103177 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:05.103152 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c0da160-9bbb-4f72-9d56-b54a269a6496" containerName="payload-processing" Oct 30 18:20:05.103177 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:05.103175 4442 cpu_manager.go:414] "containerMap: removing container" podUID="c1461e0a-7e0f-4942-87b4-b2b160dbc10f" containerName="payload-pre-processing" Oct 30 18:20:05.103310 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:05.103181 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1461e0a-7e0f-4942-87b4-b2b160dbc10f" containerName="payload-pre-processing" Oct 30 18:20:05.103352 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:05.103317 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c0da160-9bbb-4f72-9d56-b54a269a6496" containerName="payload-processing" Oct 30 18:20:05.103352 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:05.103330 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1461e0a-7e0f-4942-87b4-b2b160dbc10f" containerName="payload-pre-processing" Oct 30 18:20:05.108675 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:05.108654 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-7d8df6544c-mlnfd" Oct 30 18:20:05.125183 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:05.125153 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-7d8df6544c-mlnfd"] Oct 30 18:20:05.130867 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:05.130843 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-011d6f\"" Oct 30 18:20:05.131026 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:05.130898 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-011d6f-dockercfg-9wgsm\"" Oct 30 18:20:05.243157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:05.237095 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efec0626-054b-45be-a516-f0f8b4329b52-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-011d6f-7d8df6544c-mlnfd\" (UID: \"efec0626-054b-45be-a516-f0f8b4329b52\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-7d8df6544c-mlnfd" Oct 30 18:20:05.243157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:05.237330 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbmsv\" (UniqueName: \"kubernetes.io/projected/efec0626-054b-45be-a516-f0f8b4329b52-kube-api-access-jbmsv\") pod \"payload-pre-processing-e2e-ipp-a-011d6f-7d8df6544c-mlnfd\" (UID: \"efec0626-054b-45be-a516-f0f8b4329b52\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-7d8df6544c-mlnfd" Oct 30 18:20:05.326247 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:05.317339 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-011d6f-6747d9ddd4-bhn8d"] Oct 30 18:20:05.395272 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:05.371586 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbmsv\" (UniqueName: \"kubernetes.io/projected/efec0626-054b-45be-a516-f0f8b4329b52-kube-api-access-jbmsv\") pod \"payload-pre-processing-e2e-ipp-a-011d6f-7d8df6544c-mlnfd\" (UID: \"efec0626-054b-45be-a516-f0f8b4329b52\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-7d8df6544c-mlnfd" Oct 30 18:20:05.401150 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:05.395766 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efec0626-054b-45be-a516-f0f8b4329b52-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-011d6f-7d8df6544c-mlnfd\" (UID: \"efec0626-054b-45be-a516-f0f8b4329b52\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-7d8df6544c-mlnfd" Oct 30 18:20:05.402721 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:05.402314 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efec0626-054b-45be-a516-f0f8b4329b52-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-011d6f-7d8df6544c-mlnfd\" (UID: \"efec0626-054b-45be-a516-f0f8b4329b52\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-7d8df6544c-mlnfd" Oct 30 18:20:05.502157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:05.465803 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbmsv\" (UniqueName: \"kubernetes.io/projected/efec0626-054b-45be-a516-f0f8b4329b52-kube-api-access-jbmsv\") pod \"payload-pre-processing-e2e-ipp-a-011d6f-7d8df6544c-mlnfd\" (UID: \"efec0626-054b-45be-a516-f0f8b4329b52\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-7d8df6544c-mlnfd" Oct 30 18:20:05.607893 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:05.589456 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-011d6f-6747d9ddd4-bhn8d" Oct 30 18:20:05.607893 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:05.605558 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-011d6f-6747d9ddd4-bhn8d"] Oct 30 18:20:05.620811 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:05.620776 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3c3d410-8315-4ab0-a021-9e51cd8cf579-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-011d6f-6747d9ddd4-bhn8d\" (UID: \"e3c3d410-8315-4ab0-a021-9e51cd8cf579\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-011d6f-6747d9ddd4-bhn8d" Oct 30 18:20:05.621104 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:05.621081 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fbqt\" (UniqueName: \"kubernetes.io/projected/e3c3d410-8315-4ab0-a021-9e51cd8cf579-kube-api-access-4fbqt\") pod \"payload-processing-e2e-ipp-a-011d6f-6747d9ddd4-bhn8d\" (UID: \"e3c3d410-8315-4ab0-a021-9e51cd8cf579\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-011d6f-6747d9ddd4-bhn8d" Oct 30 18:20:05.722484 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:05.722447 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3c3d410-8315-4ab0-a021-9e51cd8cf579-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-011d6f-6747d9ddd4-bhn8d\" (UID: \"e3c3d410-8315-4ab0-a021-9e51cd8cf579\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-011d6f-6747d9ddd4-bhn8d" Oct 30 18:20:05.722655 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:05.722515 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fbqt\" (UniqueName: \"kubernetes.io/projected/e3c3d410-8315-4ab0-a021-9e51cd8cf579-kube-api-access-4fbqt\") pod \"payload-processing-e2e-ipp-a-011d6f-6747d9ddd4-bhn8d\" (UID: \"e3c3d410-8315-4ab0-a021-9e51cd8cf579\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-011d6f-6747d9ddd4-bhn8d" Oct 30 18:20:05.723230 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:05.723204 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3c3d410-8315-4ab0-a021-9e51cd8cf579-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-011d6f-6747d9ddd4-bhn8d\" (UID: \"e3c3d410-8315-4ab0-a021-9e51cd8cf579\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-011d6f-6747d9ddd4-bhn8d" Oct 30 18:20:05.724928 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:05.724899 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-7d8df6544c-mlnfd" Oct 30 18:20:05.741827 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:05.741798 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fbqt\" (UniqueName: \"kubernetes.io/projected/e3c3d410-8315-4ab0-a021-9e51cd8cf579-kube-api-access-4fbqt\") pod \"payload-processing-e2e-ipp-a-011d6f-6747d9ddd4-bhn8d\" (UID: \"e3c3d410-8315-4ab0-a021-9e51cd8cf579\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-011d6f-6747d9ddd4-bhn8d" Oct 30 18:20:05.940542 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:05.940272 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-011d6f-6747d9ddd4-bhn8d" Oct 30 18:20:06.056334 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:20:06.055762 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefec0626_054b_45be_a516_f0f8b4329b52.slice/crio-9a028005bf07ce84d76edf3fcea27eb710687e3e89d89329fee1090f72a6ea75 WatchSource:0}: Error finding container 9a028005bf07ce84d76edf3fcea27eb710687e3e89d89329fee1090f72a6ea75: Status 404 returned error can't find the container with id 9a028005bf07ce84d76edf3fcea27eb710687e3e89d89329fee1090f72a6ea75 Oct 30 18:20:06.062610 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:06.062537 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-7d8df6544c-mlnfd"] Oct 30 18:20:06.273640 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:20:06.273606 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3c3d410_8315_4ab0_a021_9e51cd8cf579.slice/crio-92d510dfd71bc157270316b9aef9a33afd4fdaca83e9c35335e694b93efa6377 WatchSource:0}: Error finding container 92d510dfd71bc157270316b9aef9a33afd4fdaca83e9c35335e694b93efa6377: Status 404 returned error can't find the container with id 92d510dfd71bc157270316b9aef9a33afd4fdaca83e9c35335e694b93efa6377 Oct 30 18:20:06.274075 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:06.273931 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-011d6f-6747d9ddd4-bhn8d"] Oct 30 18:20:06.961417 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:06.961381 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-011d6f-6747d9ddd4-bhn8d" event={"ID":"e3c3d410-8315-4ab0-a021-9e51cd8cf579","Type":"ContainerStarted","Data":"bf241951835b6885f961341296a8be3ada1a92e8940a3bd7fb566597434a0044"} Oct 30 18:20:06.961625 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:06.961446 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-011d6f-6747d9ddd4-bhn8d" event={"ID":"e3c3d410-8315-4ab0-a021-9e51cd8cf579","Type":"ContainerStarted","Data":"92d510dfd71bc157270316b9aef9a33afd4fdaca83e9c35335e694b93efa6377"} Oct 30 18:20:06.961625 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:06.961475 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-011d6f-6747d9ddd4-bhn8d" Oct 30 18:20:06.963983 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:06.963942 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-7d8df6544c-mlnfd" event={"ID":"efec0626-054b-45be-a516-f0f8b4329b52","Type":"ContainerStarted","Data":"75a87049b35ed90e6b4f9816ff095725eb0e94da27e1eb67ec98d3341aeb5de4"} Oct 30 18:20:06.964163 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:06.964013 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-7d8df6544c-mlnfd" Oct 30 18:20:06.964163 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:06.964036 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-7d8df6544c-mlnfd" event={"ID":"efec0626-054b-45be-a516-f0f8b4329b52","Type":"ContainerStarted","Data":"9a028005bf07ce84d76edf3fcea27eb710687e3e89d89329fee1090f72a6ea75"} Oct 30 18:20:07.001380 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:07.001329 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-011d6f-6747d9ddd4-bhn8d" podStartSLOduration=2.0013145 podStartE2EDuration="2.0013145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:07.000792933 +0000 UTC m=+1304.090855292" watchObservedRunningTime="2026-10-30 18:20:07.0013145 +0000 UTC m=+1304.091376856" Oct 30 18:20:07.045740 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:07.045680 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-7d8df6544c-mlnfd" podStartSLOduration=2.045666852 podStartE2EDuration="2.045666852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:07.041788494 +0000 UTC m=+1304.131850835" watchObservedRunningTime="2026-10-30 18:20:07.045666852 +0000 UTC m=+1304.135729192" Oct 30 18:20:07.732506 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:07.732471 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-de14f2-66c68cfd8-z94hs"] Oct 30 18:20:07.737429 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:07.737407 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-de14f2-66c68cfd8-z94hs" Oct 30 18:20:07.755572 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:07.755544 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-de14f2-66c68cfd8-z94hs"] Oct 30 18:20:07.761503 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:07.761479 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-de14f2\"" Oct 30 18:20:07.942161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:07.936444 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51d7f087-155a-47a3-b2d5-277a678d5ec0-plugins-config-volume\") pod \"payload-processing-e2e-api-b-de14f2-66c68cfd8-z94hs\" (UID: \"51d7f087-155a-47a3-b2d5-277a678d5ec0\") " pod="openshift-ingress/payload-processing-e2e-api-b-de14f2-66c68cfd8-z94hs" Oct 30 18:20:07.942161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:07.936567 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b52t\" (UniqueName: \"kubernetes.io/projected/51d7f087-155a-47a3-b2d5-277a678d5ec0-kube-api-access-9b52t\") pod \"payload-processing-e2e-api-b-de14f2-66c68cfd8-z94hs\" (UID: \"51d7f087-155a-47a3-b2d5-277a678d5ec0\") " pod="openshift-ingress/payload-processing-e2e-api-b-de14f2-66c68cfd8-z94hs" Oct 30 18:20:08.040430 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:08.040332 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51d7f087-155a-47a3-b2d5-277a678d5ec0-plugins-config-volume\") pod \"payload-processing-e2e-api-b-de14f2-66c68cfd8-z94hs\" (UID: \"51d7f087-155a-47a3-b2d5-277a678d5ec0\") " pod="openshift-ingress/payload-processing-e2e-api-b-de14f2-66c68cfd8-z94hs" Oct 30 18:20:08.040728 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:08.040708 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b52t\" (UniqueName: \"kubernetes.io/projected/51d7f087-155a-47a3-b2d5-277a678d5ec0-kube-api-access-9b52t\") pod \"payload-processing-e2e-api-b-de14f2-66c68cfd8-z94hs\" (UID: \"51d7f087-155a-47a3-b2d5-277a678d5ec0\") " pod="openshift-ingress/payload-processing-e2e-api-b-de14f2-66c68cfd8-z94hs" Oct 30 18:20:08.083904 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:08.069614 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e9096a-685c586796-k9fgv"] Oct 30 18:20:08.083904 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:08.070497 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51d7f087-155a-47a3-b2d5-277a678d5ec0-plugins-config-volume\") pod \"payload-processing-e2e-api-b-de14f2-66c68cfd8-z94hs\" (UID: \"51d7f087-155a-47a3-b2d5-277a678d5ec0\") " pod="openshift-ingress/payload-processing-e2e-api-b-de14f2-66c68cfd8-z94hs" Oct 30 18:20:08.088447 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:08.084845 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b52t\" (UniqueName: \"kubernetes.io/projected/51d7f087-155a-47a3-b2d5-277a678d5ec0-kube-api-access-9b52t\") pod \"payload-processing-e2e-api-b-de14f2-66c68cfd8-z94hs\" (UID: \"51d7f087-155a-47a3-b2d5-277a678d5ec0\") " pod="openshift-ingress/payload-processing-e2e-api-b-de14f2-66c68cfd8-z94hs" Oct 30 18:20:08.134161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:08.134088 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-84c84577c5-q85dn"] Oct 30 18:20:08.374225 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:08.368672 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-de14f2-66c68cfd8-z94hs" Oct 30 18:20:09.114426 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:20:09.112321 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51d7f087_155a_47a3_b2d5_277a678d5ec0.slice/crio-afaaaa0e6a4d95eca9ed6006606d319e810ca553b2a7ca0b4c541d10ca8c72a7 WatchSource:0}: Error finding container afaaaa0e6a4d95eca9ed6006606d319e810ca553b2a7ca0b4c541d10ca8c72a7: Status 404 returned error can't find the container with id afaaaa0e6a4d95eca9ed6006606d319e810ca553b2a7ca0b4c541d10ca8c72a7 Oct 30 18:20:09.157712 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:09.157668 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-de14f2-66c68cfd8-z94hs"] Oct 30 18:20:09.173314 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:09.173283 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-669b5044-wsbnn"] Oct 30 18:20:09.179167 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:09.179118 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-669b5044-wsbnn"] Oct 30 18:20:10.150991 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:10.150955 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-de14f2-66c68cfd8-z94hs" event={"ID":"51d7f087-155a-47a3-b2d5-277a678d5ec0","Type":"ContainerStarted","Data":"fd35c65f7676242bffc9e780e4b095969fad81276330ee59da5ed4da78419920"} Oct 30 18:20:10.151464 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:10.151018 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-de14f2-66c68cfd8-z94hs" event={"ID":"51d7f087-155a-47a3-b2d5-277a678d5ec0","Type":"ContainerStarted","Data":"afaaaa0e6a4d95eca9ed6006606d319e810ca553b2a7ca0b4c541d10ca8c72a7"} Oct 30 18:20:10.151464 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:10.151125 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-de14f2-66c68cfd8-z94hs" Oct 30 18:20:10.194055 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:10.194000 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-de14f2-66c68cfd8-z94hs" podStartSLOduration=3.193986635 podStartE2EDuration="3.193986635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:10.190331899 +0000 UTC m=+1307.280394249" watchObservedRunningTime="2026-10-30 18:20:10.193986635 +0000 UTC m=+1307.284048999" Oct 30 18:20:11.074158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:11.071442 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fed16271-5356-4f46-b6ba-98f5393aeef7" path="/var/lib/kubelet/pods/fed16271-5356-4f46-b6ba-98f5393aeef7/volumes" Oct 30 18:20:13.070839 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:13.070783 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9096a-685c586796-k9fgv" podUID="61a7e965-4538-4848-b50c-a5eb5744c8e3" containerName="payload-processing" containerID="cri-o://520ca286297eb1c6fbc8f49af5ddcf4cd41f2b8a8943233082374e1f31d4e3dc" gracePeriod=25 Oct 30 18:20:13.135568 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:13.135527 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-84c84577c5-q85dn" podUID="b6d9eb70-ee02-4a61-b625-83f906804c08" containerName="payload-pre-processing" containerID="cri-o://5bf8935316376cc4134d1f9e6cad4b4b8d8f8c0f130d88d01352198ca884e3c1" gracePeriod=25 Oct 30 18:20:13.515162 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:13.511575 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9096a-685c586796-k9fgv" Oct 30 18:20:13.565865 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:13.565837 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-84c84577c5-q85dn" Oct 30 18:20:13.620030 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:13.619937 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6d9eb70-ee02-4a61-b625-83f906804c08-kube-api-access-44j7c\" (UniqueName: \"kubernetes.io/projected/b6d9eb70-ee02-4a61-b625-83f906804c08-kube-api-access-44j7c\") pod \"b6d9eb70-ee02-4a61-b625-83f906804c08\" (UID: \"b6d9eb70-ee02-4a61-b625-83f906804c08\") " Oct 30 18:20:13.620220 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:13.620049 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61a7e965-4538-4848-b50c-a5eb5744c8e3-kube-api-access-9zgm5\" (UniqueName: \"kubernetes.io/projected/61a7e965-4538-4848-b50c-a5eb5744c8e3-kube-api-access-9zgm5\") pod \"61a7e965-4538-4848-b50c-a5eb5744c8e3\" (UID: \"61a7e965-4538-4848-b50c-a5eb5744c8e3\") " Oct 30 18:20:13.620287 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:13.620228 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61a7e965-4538-4848-b50c-a5eb5744c8e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61a7e965-4538-4848-b50c-a5eb5744c8e3-plugins-config-volume\") pod \"61a7e965-4538-4848-b50c-a5eb5744c8e3\" (UID: \"61a7e965-4538-4848-b50c-a5eb5744c8e3\") " Oct 30 18:20:13.620352 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:13.620337 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6d9eb70-ee02-4a61-b625-83f906804c08-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6d9eb70-ee02-4a61-b625-83f906804c08-plugins-config-volume\") pod \"b6d9eb70-ee02-4a61-b625-83f906804c08\" (UID: \"b6d9eb70-ee02-4a61-b625-83f906804c08\") " Oct 30 18:20:13.621372 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:13.621347 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6d9eb70-ee02-4a61-b625-83f906804c08-plugins-config-volume" pod "b6d9eb70-ee02-4a61-b625-83f906804c08" (UID: "b6d9eb70-ee02-4a61-b625-83f906804c08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:20:13.625915 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:13.625883 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61a7e965-4538-4848-b50c-a5eb5744c8e3-plugins-config-volume" pod "61a7e965-4538-4848-b50c-a5eb5744c8e3" (UID: "61a7e965-4538-4848-b50c-a5eb5744c8e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:20:13.628872 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:13.628843 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61a7e965-4538-4848-b50c-a5eb5744c8e3-kube-api-access-9zgm5" pod "61a7e965-4538-4848-b50c-a5eb5744c8e3" (UID: "61a7e965-4538-4848-b50c-a5eb5744c8e3"). InnerVolumeSpecName "kube-api-access-9zgm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:20:13.630786 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:13.630752 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6d9eb70-ee02-4a61-b625-83f906804c08-kube-api-access-44j7c" pod "b6d9eb70-ee02-4a61-b625-83f906804c08" (UID: "b6d9eb70-ee02-4a61-b625-83f906804c08"). InnerVolumeSpecName "kube-api-access-44j7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:20:13.722144 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:13.722107 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44j7c\" (UniqueName: \"kubernetes.io/projected/b6d9eb70-ee02-4a61-b625-83f906804c08-kube-api-access-44j7c\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:13.722315 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:13.722174 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zgm5\" (UniqueName: \"kubernetes.io/projected/61a7e965-4538-4848-b50c-a5eb5744c8e3-kube-api-access-9zgm5\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:13.722315 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:13.722183 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61a7e965-4538-4848-b50c-a5eb5744c8e3-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:13.722315 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:13.722192 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6d9eb70-ee02-4a61-b625-83f906804c08-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:14.188984 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:14.188946 4442 generic.go:356] "Generic (PLEG): container finished" podID="61a7e965-4538-4848-b50c-a5eb5744c8e3" containerID="520ca286297eb1c6fbc8f49af5ddcf4cd41f2b8a8943233082374e1f31d4e3dc" exitCode=0 Oct 30 18:20:14.189502 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:14.189039 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9096a-685c586796-k9fgv" Oct 30 18:20:14.189502 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:14.189083 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9096a-685c586796-k9fgv" event={"ID":"61a7e965-4538-4848-b50c-a5eb5744c8e3","Type":"ContainerDied","Data":"520ca286297eb1c6fbc8f49af5ddcf4cd41f2b8a8943233082374e1f31d4e3dc"} Oct 30 18:20:14.189502 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:14.189231 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9096a-685c586796-k9fgv" event={"ID":"61a7e965-4538-4848-b50c-a5eb5744c8e3","Type":"ContainerDied","Data":"067c3f1870b161b2f350bfa4536600ec23f66c2b037bb55a810fb90b68c19a3f"} Oct 30 18:20:14.189502 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:14.189267 4442 scope.go:122] "RemoveContainer" containerID="520ca286297eb1c6fbc8f49af5ddcf4cd41f2b8a8943233082374e1f31d4e3dc" Oct 30 18:20:14.191102 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:14.191074 4442 generic.go:356] "Generic (PLEG): container finished" podID="b6d9eb70-ee02-4a61-b625-83f906804c08" containerID="5bf8935316376cc4134d1f9e6cad4b4b8d8f8c0f130d88d01352198ca884e3c1" exitCode=0 Oct 30 18:20:14.191238 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:14.191124 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-84c84577c5-q85dn" event={"ID":"b6d9eb70-ee02-4a61-b625-83f906804c08","Type":"ContainerDied","Data":"5bf8935316376cc4134d1f9e6cad4b4b8d8f8c0f130d88d01352198ca884e3c1"} Oct 30 18:20:14.191238 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:14.191206 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-84c84577c5-q85dn" event={"ID":"b6d9eb70-ee02-4a61-b625-83f906804c08","Type":"ContainerDied","Data":"26044cd77038c3dbf1948138ef05aa7a1fd81c7053f5d21037dbd4dba9dca116"} Oct 30 18:20:14.191343 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:14.191309 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-84c84577c5-q85dn" Oct 30 18:20:14.204560 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:14.204537 4442 scope.go:122] "RemoveContainer" containerID="520ca286297eb1c6fbc8f49af5ddcf4cd41f2b8a8943233082374e1f31d4e3dc" Oct 30 18:20:14.210450 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:20:14.210422 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"520ca286297eb1c6fbc8f49af5ddcf4cd41f2b8a8943233082374e1f31d4e3dc\": container with ID starting with 520ca286297eb1c6fbc8f49af5ddcf4cd41f2b8a8943233082374e1f31d4e3dc not found: ID does not exist" containerID="520ca286297eb1c6fbc8f49af5ddcf4cd41f2b8a8943233082374e1f31d4e3dc" Oct 30 18:20:14.210598 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:14.210459 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"520ca286297eb1c6fbc8f49af5ddcf4cd41f2b8a8943233082374e1f31d4e3dc"} err="failed to get container status \"520ca286297eb1c6fbc8f49af5ddcf4cd41f2b8a8943233082374e1f31d4e3dc\": rpc error: code = NotFound desc = could not find container \"520ca286297eb1c6fbc8f49af5ddcf4cd41f2b8a8943233082374e1f31d4e3dc\": container with ID starting with 520ca286297eb1c6fbc8f49af5ddcf4cd41f2b8a8943233082374e1f31d4e3dc not found: ID does not exist" Oct 30 18:20:14.210598 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:14.210482 4442 scope.go:122] "RemoveContainer" containerID="5bf8935316376cc4134d1f9e6cad4b4b8d8f8c0f130d88d01352198ca884e3c1" Oct 30 18:20:14.221211 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:14.221190 4442 scope.go:122] "RemoveContainer" containerID="5bf8935316376cc4134d1f9e6cad4b4b8d8f8c0f130d88d01352198ca884e3c1" Oct 30 18:20:14.221514 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:20:14.221493 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bf8935316376cc4134d1f9e6cad4b4b8d8f8c0f130d88d01352198ca884e3c1\": container with ID starting with 5bf8935316376cc4134d1f9e6cad4b4b8d8f8c0f130d88d01352198ca884e3c1 not found: ID does not exist" containerID="5bf8935316376cc4134d1f9e6cad4b4b8d8f8c0f130d88d01352198ca884e3c1" Oct 30 18:20:14.221586 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:14.221523 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bf8935316376cc4134d1f9e6cad4b4b8d8f8c0f130d88d01352198ca884e3c1"} err="failed to get container status \"5bf8935316376cc4134d1f9e6cad4b4b8d8f8c0f130d88d01352198ca884e3c1\": rpc error: code = NotFound desc = could not find container \"5bf8935316376cc4134d1f9e6cad4b4b8d8f8c0f130d88d01352198ca884e3c1\": container with ID starting with 5bf8935316376cc4134d1f9e6cad4b4b8d8f8c0f130d88d01352198ca884e3c1 not found: ID does not exist" Oct 30 18:20:14.248369 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:14.248338 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-84c84577c5-q85dn"] Oct 30 18:20:14.259521 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:14.259493 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-84c84577c5-q85dn"] Oct 30 18:20:14.283567 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:14.283535 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e9096a-685c586796-k9fgv"] Oct 30 18:20:14.287970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:14.287939 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e9096a-685c586796-k9fgv"] Oct 30 18:20:14.956160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:14.954235 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-6976b597b7-2kmj8"] Oct 30 18:20:14.956160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:14.955454 4442 cpu_manager.go:414] "containerMap: removing container" podUID="61a7e965-4538-4848-b50c-a5eb5744c8e3" containerName="payload-processing" Oct 30 18:20:14.956160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:14.955475 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61a7e965-4538-4848-b50c-a5eb5744c8e3" containerName="payload-processing" Oct 30 18:20:14.956160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:14.955497 4442 cpu_manager.go:414] "containerMap: removing container" podUID="b6d9eb70-ee02-4a61-b625-83f906804c08" containerName="payload-pre-processing" Oct 30 18:20:14.956160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:14.955504 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6d9eb70-ee02-4a61-b625-83f906804c08" containerName="payload-pre-processing" Oct 30 18:20:14.956160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:14.955644 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="61a7e965-4538-4848-b50c-a5eb5744c8e3" containerName="payload-processing" Oct 30 18:20:14.956160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:14.955656 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6d9eb70-ee02-4a61-b625-83f906804c08" containerName="payload-pre-processing" Oct 30 18:20:14.964460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:14.964424 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-6976b597b7-2kmj8" Oct 30 18:20:14.971598 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:14.970360 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-6976b597b7-2kmj8"] Oct 30 18:20:14.980857 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:14.979631 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e9096a\"" Oct 30 18:20:14.980857 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:14.979771 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-e9096a-dockercfg-sfwnz\"" Oct 30 18:20:15.028821 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:15.025227 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72007b47-838f-4f2f-97d5-0d93d2d3e3f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e9096a-6976b597b7-2kmj8\" (UID: \"72007b47-838f-4f2f-97d5-0d93d2d3e3f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-6976b597b7-2kmj8" Oct 30 18:20:15.028821 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:15.025332 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vvhj\" (UniqueName: \"kubernetes.io/projected/72007b47-838f-4f2f-97d5-0d93d2d3e3f1-kube-api-access-4vvhj\") pod \"payload-pre-processing-e2e-ipp-a-e9096a-6976b597b7-2kmj8\" (UID: \"72007b47-838f-4f2f-97d5-0d93d2d3e3f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-6976b597b7-2kmj8" Oct 30 18:20:15.129159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:15.126913 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72007b47-838f-4f2f-97d5-0d93d2d3e3f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e9096a-6976b597b7-2kmj8\" (UID: \"72007b47-838f-4f2f-97d5-0d93d2d3e3f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-6976b597b7-2kmj8" Oct 30 18:20:15.129159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:15.126963 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vvhj\" (UniqueName: \"kubernetes.io/projected/72007b47-838f-4f2f-97d5-0d93d2d3e3f1-kube-api-access-4vvhj\") pod \"payload-pre-processing-e2e-ipp-a-e9096a-6976b597b7-2kmj8\" (UID: \"72007b47-838f-4f2f-97d5-0d93d2d3e3f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-6976b597b7-2kmj8" Oct 30 18:20:15.130358 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:15.130323 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72007b47-838f-4f2f-97d5-0d93d2d3e3f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e9096a-6976b597b7-2kmj8\" (UID: \"72007b47-838f-4f2f-97d5-0d93d2d3e3f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-6976b597b7-2kmj8" Oct 30 18:20:15.187228 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:15.173809 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vvhj\" (UniqueName: \"kubernetes.io/projected/72007b47-838f-4f2f-97d5-0d93d2d3e3f1-kube-api-access-4vvhj\") pod \"payload-pre-processing-e2e-ipp-a-e9096a-6976b597b7-2kmj8\" (UID: \"72007b47-838f-4f2f-97d5-0d93d2d3e3f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-6976b597b7-2kmj8" Oct 30 18:20:15.293799 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:15.293486 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-6976b597b7-2kmj8" Oct 30 18:20:15.338201 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:15.317892 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61a7e965-4538-4848-b50c-a5eb5744c8e3" path="/var/lib/kubelet/pods/61a7e965-4538-4848-b50c-a5eb5744c8e3/volumes" Oct 30 18:20:15.383157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:15.379388 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6d9eb70-ee02-4a61-b625-83f906804c08" path="/var/lib/kubelet/pods/b6d9eb70-ee02-4a61-b625-83f906804c08/volumes" Oct 30 18:20:15.383157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:15.379867 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e9096a-5dbbbd6d66-zfdxn"] Oct 30 18:20:15.418484 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:15.418452 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e9096a-5dbbbd6d66-zfdxn"] Oct 30 18:20:15.418659 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:15.418573 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9096a-5dbbbd6d66-zfdxn" Oct 30 18:20:15.629709 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:15.560714 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a94b8d26-825b-49b0-b1f5-0b5879807918-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e9096a-5dbbbd6d66-zfdxn\" (UID: \"a94b8d26-825b-49b0-b1f5-0b5879807918\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e9096a-5dbbbd6d66-zfdxn" Oct 30 18:20:15.629709 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:15.560795 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s98jg\" (UniqueName: \"kubernetes.io/projected/a94b8d26-825b-49b0-b1f5-0b5879807918-kube-api-access-s98jg\") pod \"payload-processing-e2e-ipp-a-e9096a-5dbbbd6d66-zfdxn\" (UID: \"a94b8d26-825b-49b0-b1f5-0b5879807918\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e9096a-5dbbbd6d66-zfdxn" Oct 30 18:20:15.700409 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:15.700366 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a94b8d26-825b-49b0-b1f5-0b5879807918-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e9096a-5dbbbd6d66-zfdxn\" (UID: \"a94b8d26-825b-49b0-b1f5-0b5879807918\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e9096a-5dbbbd6d66-zfdxn" Oct 30 18:20:15.700693 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:15.700674 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s98jg\" (UniqueName: \"kubernetes.io/projected/a94b8d26-825b-49b0-b1f5-0b5879807918-kube-api-access-s98jg\") pod \"payload-processing-e2e-ipp-a-e9096a-5dbbbd6d66-zfdxn\" (UID: \"a94b8d26-825b-49b0-b1f5-0b5879807918\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e9096a-5dbbbd6d66-zfdxn" Oct 30 18:20:15.703723 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:15.703695 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a94b8d26-825b-49b0-b1f5-0b5879807918-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e9096a-5dbbbd6d66-zfdxn\" (UID: \"a94b8d26-825b-49b0-b1f5-0b5879807918\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e9096a-5dbbbd6d66-zfdxn" Oct 30 18:20:15.744159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:15.722220 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s98jg\" (UniqueName: \"kubernetes.io/projected/a94b8d26-825b-49b0-b1f5-0b5879807918-kube-api-access-s98jg\") pod \"payload-processing-e2e-ipp-a-e9096a-5dbbbd6d66-zfdxn\" (UID: \"a94b8d26-825b-49b0-b1f5-0b5879807918\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e9096a-5dbbbd6d66-zfdxn" Oct 30 18:20:15.772472 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:15.772446 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9096a-5dbbbd6d66-zfdxn" Oct 30 18:20:16.003170 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:15.997334 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-6976b597b7-2kmj8"] Oct 30 18:20:16.012508 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:16.006520 4442 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:20:16.220743 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:16.220662 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e9096a-5dbbbd6d66-zfdxn"] Oct 30 18:20:16.441242 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:16.441200 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9096a-5dbbbd6d66-zfdxn" event={"ID":"a94b8d26-825b-49b0-b1f5-0b5879807918","Type":"ContainerStarted","Data":"4c44faa65eaf7fed55969e39e8a6ea6d4fa0b6640e17b07a4a8dfa1fd2981a58"} Oct 30 18:20:16.443608 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:16.443566 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-6976b597b7-2kmj8" event={"ID":"72007b47-838f-4f2f-97d5-0d93d2d3e3f1","Type":"ContainerStarted","Data":"6a11f926f8e7ed07d9aa01e53aa0e889725631448a33494b599009019332a304"} Oct 30 18:20:16.575154 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:16.575052 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-5bklf"] Oct 30 18:20:16.611384 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:16.611345 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-2csfr"] Oct 30 18:20:17.687088 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:20:17.687012 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:20:18.153201 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:18.146432 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-7d8df6544c-mlnfd" Oct 30 18:20:18.254157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:18.242420 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-011d6f-6747d9ddd4-bhn8d" Oct 30 18:20:21.177366 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:21.177334 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-de14f2-66c68cfd8-z94hs" Oct 30 18:20:21.538353 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:21.538255 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-6976b597b7-2kmj8" event={"ID":"72007b47-838f-4f2f-97d5-0d93d2d3e3f1","Type":"ContainerStarted","Data":"1cd4db0b0b5c0c9cb344ebae436c65f93f0680228ce0e2a550064dd1e1390faf"} Oct 30 18:20:21.539479 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:21.539438 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-6976b597b7-2kmj8" Oct 30 18:20:21.542799 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:21.542765 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9096a-5dbbbd6d66-zfdxn" event={"ID":"a94b8d26-825b-49b0-b1f5-0b5879807918","Type":"ContainerStarted","Data":"39a0af7aa0315475c5497bd64d9ef6a69f67c11cf189faa958faf5c7cd948030"} Oct 30 18:20:21.543639 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:21.543616 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9096a-5dbbbd6d66-zfdxn" Oct 30 18:20:21.577265 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:21.577215 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-5bklf" podUID="408d585f-9fe0-4631-bfc2-a35728e68706" containerName="payload-processing" containerID="cri-o://e0b381ab1c7813e8d549400dc5ba8cf92a58be03415b9dd823be821568ada600" gracePeriod=25 Oct 30 18:20:21.580483 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:21.580372 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-6976b597b7-2kmj8" podStartSLOduration=3.10520664 podStartE2EDuration="7.580353504s" totalImagesPullingTime="4.475146864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:21.577602127 +0000 UTC m=+1318.667664467" watchObservedRunningTime="2026-10-30 18:20:21.580353504 +0000 UTC m=+1318.670415845" Oct 30 18:20:21.611784 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:21.611751 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-2csfr" podUID="99a4ef6c-a90f-4b8b-a458-1f44618ce3c6" containerName="payload-pre-processing" containerID="cri-o://9abf5d784cc2e649b25f12d490c03d502c900421704c46a179a95077ef74615f" gracePeriod=25 Oct 30 18:20:21.618375 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:21.618326 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9096a-5dbbbd6d66-zfdxn" podStartSLOduration=2.30395311 podStartE2EDuration="6.618308973s" totalImagesPullingTime="4.314355863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:21.615121131 +0000 UTC m=+1318.705183471" watchObservedRunningTime="2026-10-30 18:20:21.618308973 +0000 UTC m=+1318.708371312" Oct 30 18:20:22.096619 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:22.096547 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-5bklf" Oct 30 18:20:22.197083 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:22.196223 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/408d585f-9fe0-4631-bfc2-a35728e68706-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/408d585f-9fe0-4631-bfc2-a35728e68706-plugins-config-volume\") pod \"408d585f-9fe0-4631-bfc2-a35728e68706\" (UID: \"408d585f-9fe0-4631-bfc2-a35728e68706\") " Oct 30 18:20:22.197083 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:22.196419 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/408d585f-9fe0-4631-bfc2-a35728e68706-kube-api-access-6244l\" (UniqueName: \"kubernetes.io/projected/408d585f-9fe0-4631-bfc2-a35728e68706-kube-api-access-6244l\") pod \"408d585f-9fe0-4631-bfc2-a35728e68706\" (UID: \"408d585f-9fe0-4631-bfc2-a35728e68706\") " Oct 30 18:20:22.199839 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:22.199506 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/408d585f-9fe0-4631-bfc2-a35728e68706-plugins-config-volume" pod "408d585f-9fe0-4631-bfc2-a35728e68706" (UID: "408d585f-9fe0-4631-bfc2-a35728e68706"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:20:22.200903 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:22.200862 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/408d585f-9fe0-4631-bfc2-a35728e68706-kube-api-access-6244l" pod "408d585f-9fe0-4631-bfc2-a35728e68706" (UID: "408d585f-9fe0-4631-bfc2-a35728e68706"). InnerVolumeSpecName "kube-api-access-6244l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:20:22.237036 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:22.237005 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-2csfr" Oct 30 18:20:22.299687 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:22.299653 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6244l\" (UniqueName: \"kubernetes.io/projected/408d585f-9fe0-4631-bfc2-a35728e68706-kube-api-access-6244l\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:22.299687 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:22.299691 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/408d585f-9fe0-4631-bfc2-a35728e68706-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:22.400953 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:22.400916 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99a4ef6c-a90f-4b8b-a458-1f44618ce3c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99a4ef6c-a90f-4b8b-a458-1f44618ce3c6-plugins-config-volume\") pod \"99a4ef6c-a90f-4b8b-a458-1f44618ce3c6\" (UID: \"99a4ef6c-a90f-4b8b-a458-1f44618ce3c6\") " Oct 30 18:20:22.401204 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:22.400979 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99a4ef6c-a90f-4b8b-a458-1f44618ce3c6-kube-api-access-wbg9p\" (UniqueName: \"kubernetes.io/projected/99a4ef6c-a90f-4b8b-a458-1f44618ce3c6-kube-api-access-wbg9p\") pod \"99a4ef6c-a90f-4b8b-a458-1f44618ce3c6\" (UID: \"99a4ef6c-a90f-4b8b-a458-1f44618ce3c6\") " Oct 30 18:20:22.401607 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:22.401573 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99a4ef6c-a90f-4b8b-a458-1f44618ce3c6-plugins-config-volume" pod "99a4ef6c-a90f-4b8b-a458-1f44618ce3c6" (UID: "99a4ef6c-a90f-4b8b-a458-1f44618ce3c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:20:22.403768 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:22.403739 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99a4ef6c-a90f-4b8b-a458-1f44618ce3c6-kube-api-access-wbg9p" pod "99a4ef6c-a90f-4b8b-a458-1f44618ce3c6" (UID: "99a4ef6c-a90f-4b8b-a458-1f44618ce3c6"). InnerVolumeSpecName "kube-api-access-wbg9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:20:22.503153 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:22.503098 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99a4ef6c-a90f-4b8b-a458-1f44618ce3c6-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:22.503153 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:22.503125 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbg9p\" (UniqueName: \"kubernetes.io/projected/99a4ef6c-a90f-4b8b-a458-1f44618ce3c6-kube-api-access-wbg9p\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:22.556707 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:22.556667 4442 generic.go:356] "Generic (PLEG): container finished" podID="99a4ef6c-a90f-4b8b-a458-1f44618ce3c6" containerID="9abf5d784cc2e649b25f12d490c03d502c900421704c46a179a95077ef74615f" exitCode=0 Oct 30 18:20:22.556911 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:22.556893 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-2csfr" event={"ID":"99a4ef6c-a90f-4b8b-a458-1f44618ce3c6","Type":"ContainerDied","Data":"9abf5d784cc2e649b25f12d490c03d502c900421704c46a179a95077ef74615f"} Oct 30 18:20:22.556976 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:22.556945 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-2csfr" event={"ID":"99a4ef6c-a90f-4b8b-a458-1f44618ce3c6","Type":"ContainerDied","Data":"a04749f4da6d7a2cf9383ce9ecf581c8b93700b36fd93c6c0041e9a274c2203b"} Oct 30 18:20:22.556976 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:22.556973 4442 scope.go:122] "RemoveContainer" containerID="9abf5d784cc2e649b25f12d490c03d502c900421704c46a179a95077ef74615f" Oct 30 18:20:22.557151 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:22.557117 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-2csfr" Oct 30 18:20:22.563777 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:22.563737 4442 generic.go:356] "Generic (PLEG): container finished" podID="408d585f-9fe0-4631-bfc2-a35728e68706" containerID="e0b381ab1c7813e8d549400dc5ba8cf92a58be03415b9dd823be821568ada600" exitCode=0 Oct 30 18:20:22.564833 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:22.564812 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-5bklf" Oct 30 18:20:22.567265 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:22.567206 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-5bklf" event={"ID":"408d585f-9fe0-4631-bfc2-a35728e68706","Type":"ContainerDied","Data":"e0b381ab1c7813e8d549400dc5ba8cf92a58be03415b9dd823be821568ada600"} Oct 30 18:20:22.567265 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:22.567256 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-5bklf" event={"ID":"408d585f-9fe0-4631-bfc2-a35728e68706","Type":"ContainerDied","Data":"8de100585826a5e06cf4f2aaf20c6692a55553749418bef7442937c94e0f4f19"} Oct 30 18:20:22.580462 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:22.580357 4442 scope.go:122] "RemoveContainer" containerID="9abf5d784cc2e649b25f12d490c03d502c900421704c46a179a95077ef74615f" Oct 30 18:20:22.580708 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:20:22.580686 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9abf5d784cc2e649b25f12d490c03d502c900421704c46a179a95077ef74615f\": container with ID starting with 9abf5d784cc2e649b25f12d490c03d502c900421704c46a179a95077ef74615f not found: ID does not exist" containerID="9abf5d784cc2e649b25f12d490c03d502c900421704c46a179a95077ef74615f" Oct 30 18:20:22.580750 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:22.580717 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9abf5d784cc2e649b25f12d490c03d502c900421704c46a179a95077ef74615f"} err="failed to get container status \"9abf5d784cc2e649b25f12d490c03d502c900421704c46a179a95077ef74615f\": rpc error: code = NotFound desc = could not find container \"9abf5d784cc2e649b25f12d490c03d502c900421704c46a179a95077ef74615f\": container with ID starting with 9abf5d784cc2e649b25f12d490c03d502c900421704c46a179a95077ef74615f not found: ID does not exist" Oct 30 18:20:22.580750 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:22.580735 4442 scope.go:122] "RemoveContainer" containerID="e0b381ab1c7813e8d549400dc5ba8cf92a58be03415b9dd823be821568ada600" Oct 30 18:20:22.607497 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:22.607438 4442 scope.go:122] "RemoveContainer" containerID="e0b381ab1c7813e8d549400dc5ba8cf92a58be03415b9dd823be821568ada600" Oct 30 18:20:22.608706 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:20:22.608596 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0b381ab1c7813e8d549400dc5ba8cf92a58be03415b9dd823be821568ada600\": container with ID starting with e0b381ab1c7813e8d549400dc5ba8cf92a58be03415b9dd823be821568ada600 not found: ID does not exist" containerID="e0b381ab1c7813e8d549400dc5ba8cf92a58be03415b9dd823be821568ada600" Oct 30 18:20:22.608706 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:22.608658 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0b381ab1c7813e8d549400dc5ba8cf92a58be03415b9dd823be821568ada600"} err="failed to get container status \"e0b381ab1c7813e8d549400dc5ba8cf92a58be03415b9dd823be821568ada600\": rpc error: code = NotFound desc = could not find container \"e0b381ab1c7813e8d549400dc5ba8cf92a58be03415b9dd823be821568ada600\": container with ID starting with e0b381ab1c7813e8d549400dc5ba8cf92a58be03415b9dd823be821568ada600 not found: ID does not exist" Oct 30 18:20:22.615138 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:22.615114 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-2csfr"] Oct 30 18:20:22.622014 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:22.621992 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0c6613ea-7978559fc9-2csfr"] Oct 30 18:20:22.675261 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:22.675179 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-5bklf"] Oct 30 18:20:22.680001 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:22.679963 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0c6613ea-86469f66f4-5bklf"] Oct 30 18:20:23.076154 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:23.075808 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="408d585f-9fe0-4631-bfc2-a35728e68706" path="/var/lib/kubelet/pods/408d585f-9fe0-4631-bfc2-a35728e68706/volumes" Oct 30 18:20:23.076564 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:23.076536 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99a4ef6c-a90f-4b8b-a458-1f44618ce3c6" path="/var/lib/kubelet/pods/99a4ef6c-a90f-4b8b-a458-1f44618ce3c6/volumes" Oct 30 18:20:23.407415 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:23.407378 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-011d6f-8458db466f-xvrj5"] Oct 30 18:20:23.408150 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:23.408124 4442 cpu_manager.go:414] "containerMap: removing container" podUID="408d585f-9fe0-4631-bfc2-a35728e68706" containerName="payload-processing" Oct 30 18:20:23.408198 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:23.408152 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="408d585f-9fe0-4631-bfc2-a35728e68706" containerName="payload-processing" Oct 30 18:20:23.408198 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:23.408187 4442 cpu_manager.go:414] "containerMap: removing container" podUID="99a4ef6c-a90f-4b8b-a458-1f44618ce3c6" containerName="payload-pre-processing" Oct 30 18:20:23.408198 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:23.408192 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99a4ef6c-a90f-4b8b-a458-1f44618ce3c6" containerName="payload-pre-processing" Oct 30 18:20:23.408302 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:23.408293 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="408d585f-9fe0-4631-bfc2-a35728e68706" containerName="payload-processing" Oct 30 18:20:23.408334 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:23.408309 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="99a4ef6c-a90f-4b8b-a458-1f44618ce3c6" containerName="payload-pre-processing" Oct 30 18:20:23.413849 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:23.413824 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-011d6f-8458db466f-xvrj5" Oct 30 18:20:23.437798 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:23.437183 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-011d6f-serving-cert\"" Oct 30 18:20:23.473945 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:23.473861 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-011d6f-8458db466f-xvrj5"] Oct 30 18:20:23.522493 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:23.522461 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d86cecf-4477-44f5-82b7-328fa9a81d50-maas-api-tls\") pod \"maas-api-e2e-ipp-a-011d6f-8458db466f-xvrj5\" (UID: \"5d86cecf-4477-44f5-82b7-328fa9a81d50\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-011d6f-8458db466f-xvrj5" Oct 30 18:20:23.522769 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:23.522731 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdcp5\" (UniqueName: \"kubernetes.io/projected/5d86cecf-4477-44f5-82b7-328fa9a81d50-kube-api-access-pdcp5\") pod \"maas-api-e2e-ipp-a-011d6f-8458db466f-xvrj5\" (UID: \"5d86cecf-4477-44f5-82b7-328fa9a81d50\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-011d6f-8458db466f-xvrj5" Oct 30 18:20:23.522867 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:23.522849 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d86cecf-4477-44f5-82b7-328fa9a81d50-metrics-certs\") pod \"maas-api-e2e-ipp-a-011d6f-8458db466f-xvrj5\" (UID: \"5d86cecf-4477-44f5-82b7-328fa9a81d50\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-011d6f-8458db466f-xvrj5" Oct 30 18:20:23.522928 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:23.522895 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d86cecf-4477-44f5-82b7-328fa9a81d50-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-011d6f-8458db466f-xvrj5\" (UID: \"5d86cecf-4477-44f5-82b7-328fa9a81d50\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-011d6f-8458db466f-xvrj5" Oct 30 18:20:23.631158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:23.624392 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdcp5\" (UniqueName: \"kubernetes.io/projected/5d86cecf-4477-44f5-82b7-328fa9a81d50-kube-api-access-pdcp5\") pod \"maas-api-e2e-ipp-a-011d6f-8458db466f-xvrj5\" (UID: \"5d86cecf-4477-44f5-82b7-328fa9a81d50\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-011d6f-8458db466f-xvrj5" Oct 30 18:20:23.631158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:23.624505 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d86cecf-4477-44f5-82b7-328fa9a81d50-metrics-certs\") pod \"maas-api-e2e-ipp-a-011d6f-8458db466f-xvrj5\" (UID: \"5d86cecf-4477-44f5-82b7-328fa9a81d50\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-011d6f-8458db466f-xvrj5" Oct 30 18:20:23.631158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:23.624560 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d86cecf-4477-44f5-82b7-328fa9a81d50-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-011d6f-8458db466f-xvrj5\" (UID: \"5d86cecf-4477-44f5-82b7-328fa9a81d50\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-011d6f-8458db466f-xvrj5" Oct 30 18:20:23.631158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:23.624636 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d86cecf-4477-44f5-82b7-328fa9a81d50-maas-api-tls\") pod \"maas-api-e2e-ipp-a-011d6f-8458db466f-xvrj5\" (UID: \"5d86cecf-4477-44f5-82b7-328fa9a81d50\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-011d6f-8458db466f-xvrj5" Oct 30 18:20:23.631158 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:20:23.624965 4442 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-011d6f-serving-cert: secret "maas-api-e2e-ipp-a-011d6f-serving-cert" not found Oct 30 18:20:23.631158 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:20:23.625031 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d86cecf-4477-44f5-82b7-328fa9a81d50-maas-api-tls podName:5d86cecf-4477-44f5-82b7-328fa9a81d50 nodeName:}" failed. No retries permitted until 2026-10-30 18:20:24.125008475 +0000 UTC m=+1321.215070799 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5d86cecf-4477-44f5-82b7-328fa9a81d50-maas-api-tls") pod "maas-api-e2e-ipp-a-011d6f-8458db466f-xvrj5" (UID: "5d86cecf-4477-44f5-82b7-328fa9a81d50") : secret "maas-api-e2e-ipp-a-011d6f-serving-cert" not found Oct 30 18:20:23.636158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:23.632526 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d86cecf-4477-44f5-82b7-328fa9a81d50-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-011d6f-8458db466f-xvrj5\" (UID: \"5d86cecf-4477-44f5-82b7-328fa9a81d50\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-011d6f-8458db466f-xvrj5" Oct 30 18:20:23.636158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:23.633280 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d86cecf-4477-44f5-82b7-328fa9a81d50-metrics-certs\") pod \"maas-api-e2e-ipp-a-011d6f-8458db466f-xvrj5\" (UID: \"5d86cecf-4477-44f5-82b7-328fa9a81d50\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-011d6f-8458db466f-xvrj5" Oct 30 18:20:23.701212 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:23.696940 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdcp5\" (UniqueName: \"kubernetes.io/projected/5d86cecf-4477-44f5-82b7-328fa9a81d50-kube-api-access-pdcp5\") pod \"maas-api-e2e-ipp-a-011d6f-8458db466f-xvrj5\" (UID: \"5d86cecf-4477-44f5-82b7-328fa9a81d50\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-011d6f-8458db466f-xvrj5" Oct 30 18:20:24.139778 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:24.139730 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d86cecf-4477-44f5-82b7-328fa9a81d50-maas-api-tls\") pod \"maas-api-e2e-ipp-a-011d6f-8458db466f-xvrj5\" (UID: \"5d86cecf-4477-44f5-82b7-328fa9a81d50\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-011d6f-8458db466f-xvrj5" Oct 30 18:20:24.144983 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:24.144949 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d86cecf-4477-44f5-82b7-328fa9a81d50-maas-api-tls\") pod \"maas-api-e2e-ipp-a-011d6f-8458db466f-xvrj5\" (UID: \"5d86cecf-4477-44f5-82b7-328fa9a81d50\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-011d6f-8458db466f-xvrj5" Oct 30 18:20:24.338074 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:24.338035 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-011d6f-8458db466f-xvrj5" Oct 30 18:20:24.765221 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:24.751077 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-011d6f-8458db466f-xvrj5"] Oct 30 18:20:24.782171 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:20:24.778571 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d86cecf_4477_44f5_82b7_328fa9a81d50.slice/crio-0bee74b7940ee88171160e0a7b6d3cac6803ff0d4a56b7c6c4f61b771c48dfa2 WatchSource:0}: Error finding container 0bee74b7940ee88171160e0a7b6d3cac6803ff0d4a56b7c6c4f61b771c48dfa2: Status 404 returned error can't find the container with id 0bee74b7940ee88171160e0a7b6d3cac6803ff0d4a56b7c6c4f61b771c48dfa2 Oct 30 18:20:25.669978 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:25.669944 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-011d6f-8458db466f-xvrj5" event={"ID":"5d86cecf-4477-44f5-82b7-328fa9a81d50","Type":"ContainerStarted","Data":"f9dcf823aa442002b866842159c8a460da783a08683dc074dffffe942ad42cce"} Oct 30 18:20:25.670180 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:25.670002 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-011d6f-8458db466f-xvrj5" event={"ID":"5d86cecf-4477-44f5-82b7-328fa9a81d50","Type":"ContainerStarted","Data":"0bee74b7940ee88171160e0a7b6d3cac6803ff0d4a56b7c6c4f61b771c48dfa2"} Oct 30 18:20:25.671030 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:25.670339 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-011d6f-8458db466f-xvrj5" Oct 30 18:20:25.713697 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:25.713635 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-011d6f-8458db466f-xvrj5" podStartSLOduration=2.290711976 podStartE2EDuration="2.713619067s" totalImagesPullingTime="422.907091ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:25.711753193 +0000 UTC m=+1322.801815532" watchObservedRunningTime="2026-10-30 18:20:25.713619067 +0000 UTC m=+1322.803681405" Oct 30 18:20:26.682988 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:26.682960 4442 generic.go:356] "Generic (PLEG): container finished" podID="cbf948c4-274b-46fd-bc3c-1c2f0d9c3247" containerID="318b9c893169d6da3c506b4f3cddfb1f088595d443be070791da37e9149af3ce" exitCode=0 Oct 30 18:20:26.683454 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:26.683040 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" event={"ID":"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247","Type":"ContainerDied","Data":"318b9c893169d6da3c506b4f3cddfb1f088595d443be070791da37e9149af3ce"} Oct 30 18:20:26.683861 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:26.683847 4442 scope.go:122] "RemoveContainer" containerID="318b9c893169d6da3c506b4f3cddfb1f088595d443be070791da37e9149af3ce" Oct 30 18:20:26.685361 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:26.685330 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:20:26.685467 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:26.685422 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:20:26.685532 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:26.685478 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:20:27.693692 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:27.693662 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" event={"ID":"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247","Type":"ContainerStarted","Data":"595fa4ed89e446d9f45d343940926444c7dfd92e0c8af8717839548b1907734b"} Oct 30 18:20:28.123083 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:28.123057 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" Oct 30 18:20:28.141164 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:28.137891 4442 patch_prober.go:29] interesting pod/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 18:20:28.141164 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:28.137980 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" podUID="cbf948c4-274b-46fd-bc3c-1c2f0d9c3247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 18:20:29.122933 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:29.122895 4442 patch_prober.go:29] interesting pod/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 18:20:29.123514 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:29.122959 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" podUID="cbf948c4-274b-46fd-bc3c-1c2f0d9c3247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 18:20:30.134161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:30.133467 4442 patch_prober.go:29] interesting pod/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 18:20:30.134820 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:30.134450 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" podUID="cbf948c4-274b-46fd-bc3c-1c2f0d9c3247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 18:20:30.703814 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:30.698257 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" Oct 30 18:20:30.788291 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:30.788245 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh"] Oct 30 18:20:30.788579 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:30.788551 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh" podUID="0e328cf8-f89f-4f08-aafb-d481dd61ffd3" containerName="maas-api" containerID="cri-o://cd5da74c7bd9c4b5075b5a61d6fa969c22e3866dda44d564f389f9eca9ffd66f" gracePeriod=30 Oct 30 18:20:31.297860 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:31.228803 4442 patch_prober.go:29] interesting pod/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 18:20:31.297860 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:31.228860 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" podUID="cbf948c4-274b-46fd-bc3c-1c2f0d9c3247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 18:20:31.711162 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:31.706076 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh" Oct 30 18:20:31.717400 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:31.717352 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-011d6f-8458db466f-xvrj5" Oct 30 18:20:31.786274 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:31.786241 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e328cf8-f89f-4f08-aafb-d481dd61ffd3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e328cf8-f89f-4f08-aafb-d481dd61ffd3-metrics-certs\") pod \"0e328cf8-f89f-4f08-aafb-d481dd61ffd3\" (UID: \"0e328cf8-f89f-4f08-aafb-d481dd61ffd3\") " Oct 30 18:20:31.787149 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:31.787107 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e328cf8-f89f-4f08-aafb-d481dd61ffd3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e328cf8-f89f-4f08-aafb-d481dd61ffd3-openshift-service-ca\") pod \"0e328cf8-f89f-4f08-aafb-d481dd61ffd3\" (UID: \"0e328cf8-f89f-4f08-aafb-d481dd61ffd3\") " Oct 30 18:20:31.791816 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:31.791513 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e328cf8-f89f-4f08-aafb-d481dd61ffd3-openshift-service-ca" pod "0e328cf8-f89f-4f08-aafb-d481dd61ffd3" (UID: "0e328cf8-f89f-4f08-aafb-d481dd61ffd3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:20:31.793328 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:31.793309 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e328cf8-f89f-4f08-aafb-d481dd61ffd3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e328cf8-f89f-4f08-aafb-d481dd61ffd3-maas-api-tls\") pod \"0e328cf8-f89f-4f08-aafb-d481dd61ffd3\" (UID: \"0e328cf8-f89f-4f08-aafb-d481dd61ffd3\") " Oct 30 18:20:31.793878 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:31.793860 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e328cf8-f89f-4f08-aafb-d481dd61ffd3-kube-api-access-56dr5\" (UniqueName: \"kubernetes.io/projected/0e328cf8-f89f-4f08-aafb-d481dd61ffd3-kube-api-access-56dr5\") pod \"0e328cf8-f89f-4f08-aafb-d481dd61ffd3\" (UID: \"0e328cf8-f89f-4f08-aafb-d481dd61ffd3\") " Oct 30 18:20:31.819213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:31.803506 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e328cf8-f89f-4f08-aafb-d481dd61ffd3-openshift-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:31.892037 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:31.891979 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e328cf8-f89f-4f08-aafb-d481dd61ffd3-metrics-certs" pod "0e328cf8-f89f-4f08-aafb-d481dd61ffd3" (UID: "0e328cf8-f89f-4f08-aafb-d481dd61ffd3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:20:31.897348 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:31.897063 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e328cf8-f89f-4f08-aafb-d481dd61ffd3-kube-api-access-56dr5" pod "0e328cf8-f89f-4f08-aafb-d481dd61ffd3" (UID: "0e328cf8-f89f-4f08-aafb-d481dd61ffd3"). InnerVolumeSpecName "kube-api-access-56dr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:20:31.897348 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:31.897281 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e328cf8-f89f-4f08-aafb-d481dd61ffd3-maas-api-tls" pod "0e328cf8-f89f-4f08-aafb-d481dd61ffd3" (UID: "0e328cf8-f89f-4f08-aafb-d481dd61ffd3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:20:31.897590 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:31.897449 4442 generic.go:356] "Generic (PLEG): container finished" podID="0e328cf8-f89f-4f08-aafb-d481dd61ffd3" containerID="cd5da74c7bd9c4b5075b5a61d6fa969c22e3866dda44d564f389f9eca9ffd66f" exitCode=0 Oct 30 18:20:31.897691 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:31.897643 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh" event={"ID":"0e328cf8-f89f-4f08-aafb-d481dd61ffd3","Type":"ContainerDied","Data":"cd5da74c7bd9c4b5075b5a61d6fa969c22e3866dda44d564f389f9eca9ffd66f"} Oct 30 18:20:31.897691 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:31.897693 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh" event={"ID":"0e328cf8-f89f-4f08-aafb-d481dd61ffd3","Type":"ContainerDied","Data":"d5167fa7e254d8649b0d16e48171ef202b62baeaee5fb4b70f9891e9d523da47"} Oct 30 18:20:31.897881 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:31.897719 4442 scope.go:122] "RemoveContainer" containerID="cd5da74c7bd9c4b5075b5a61d6fa969c22e3866dda44d564f389f9eca9ffd66f" Oct 30 18:20:31.897968 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:31.897897 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh" Oct 30 18:20:31.952157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:31.926453 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e328cf8-f89f-4f08-aafb-d481dd61ffd3-maas-api-tls\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:31.952428 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:31.952404 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56dr5\" (UniqueName: \"kubernetes.io/projected/0e328cf8-f89f-4f08-aafb-d481dd61ffd3-kube-api-access-56dr5\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:31.952525 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:31.952511 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e328cf8-f89f-4f08-aafb-d481dd61ffd3-metrics-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:31.972179 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:31.970036 4442 scope.go:122] "RemoveContainer" containerID="eb55ad414aa08dd5097f9744b4f7d9aa6db78bc4d68769c7639e83429203ea50" Oct 30 18:20:32.044566 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:32.044496 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh"] Oct 30 18:20:32.052533 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:32.051860 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0c6613ea-75cb5c7f6d-c7fjh"] Oct 30 18:20:32.109900 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:32.108908 4442 scope.go:122] "RemoveContainer" containerID="cd5da74c7bd9c4b5075b5a61d6fa969c22e3866dda44d564f389f9eca9ffd66f" Oct 30 18:20:32.109900 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:20:32.109409 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd5da74c7bd9c4b5075b5a61d6fa969c22e3866dda44d564f389f9eca9ffd66f\": container with ID starting with cd5da74c7bd9c4b5075b5a61d6fa969c22e3866dda44d564f389f9eca9ffd66f not found: ID does not exist" containerID="cd5da74c7bd9c4b5075b5a61d6fa969c22e3866dda44d564f389f9eca9ffd66f" Oct 30 18:20:32.109900 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:32.109445 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd5da74c7bd9c4b5075b5a61d6fa969c22e3866dda44d564f389f9eca9ffd66f"} err="failed to get container status \"cd5da74c7bd9c4b5075b5a61d6fa969c22e3866dda44d564f389f9eca9ffd66f\": rpc error: code = NotFound desc = could not find container \"cd5da74c7bd9c4b5075b5a61d6fa969c22e3866dda44d564f389f9eca9ffd66f\": container with ID starting with cd5da74c7bd9c4b5075b5a61d6fa969c22e3866dda44d564f389f9eca9ffd66f not found: ID does not exist" Oct 30 18:20:32.109900 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:32.109473 4442 scope.go:122] "RemoveContainer" containerID="eb55ad414aa08dd5097f9744b4f7d9aa6db78bc4d68769c7639e83429203ea50" Oct 30 18:20:32.109900 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:20:32.109758 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb55ad414aa08dd5097f9744b4f7d9aa6db78bc4d68769c7639e83429203ea50\": container with ID starting with eb55ad414aa08dd5097f9744b4f7d9aa6db78bc4d68769c7639e83429203ea50 not found: ID does not exist" containerID="eb55ad414aa08dd5097f9744b4f7d9aa6db78bc4d68769c7639e83429203ea50" Oct 30 18:20:32.109900 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:32.109784 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb55ad414aa08dd5097f9744b4f7d9aa6db78bc4d68769c7639e83429203ea50"} err="failed to get container status \"eb55ad414aa08dd5097f9744b4f7d9aa6db78bc4d68769c7639e83429203ea50\": rpc error: code = NotFound desc = could not find container \"eb55ad414aa08dd5097f9744b4f7d9aa6db78bc4d68769c7639e83429203ea50\": container with ID starting with eb55ad414aa08dd5097f9744b4f7d9aa6db78bc4d68769c7639e83429203ea50 not found: ID does not exist" Oct 30 18:20:32.209378 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:32.209348 4442 patch_prober.go:29] interesting pod/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 18:20:32.209754 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:32.209726 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" podUID="cbf948c4-274b-46fd-bc3c-1c2f0d9c3247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 18:20:33.071955 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:33.071898 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e328cf8-f89f-4f08-aafb-d481dd61ffd3" path="/var/lib/kubelet/pods/0e328cf8-f89f-4f08-aafb-d481dd61ffd3/volumes" Oct 30 18:20:33.122852 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:33.122815 4442 patch_prober.go:29] interesting pod/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 18:20:33.123027 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:33.122881 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" podUID="cbf948c4-274b-46fd-bc3c-1c2f0d9c3247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 18:20:33.586117 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:33.586085 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9096a-5dbbbd6d66-zfdxn" Oct 30 18:20:33.586530 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:33.586502 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-6976b597b7-2kmj8" Oct 30 18:20:34.127649 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:34.127621 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" Oct 30 18:20:34.129053 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:34.129032 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" Oct 30 18:20:35.428498 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:35.428458 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-695db48787-488nc"] Oct 30 18:20:35.429213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:35.429197 4442 cpu_manager.go:414] "containerMap: removing container" podUID="0e328cf8-f89f-4f08-aafb-d481dd61ffd3" containerName="maas-api" Oct 30 18:20:35.429213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:35.429211 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e328cf8-f89f-4f08-aafb-d481dd61ffd3" containerName="maas-api" Oct 30 18:20:35.429378 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:35.429337 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e328cf8-f89f-4f08-aafb-d481dd61ffd3" containerName="maas-api" Oct 30 18:20:35.429378 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:35.429348 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e328cf8-f89f-4f08-aafb-d481dd61ffd3" containerName="maas-api" Oct 30 18:20:35.435907 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:35.435886 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-695db48787-488nc" Oct 30 18:20:35.453351 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:35.453319 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-695db48787-488nc"] Oct 30 18:20:35.458603 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:35.458571 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f826dd-dockercfg-w2lz5\"" Oct 30 18:20:35.458603 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:35.458571 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f826dd\"" Oct 30 18:20:35.490725 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:35.490676 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e586a1a8-85e4-47df-b62a-72c35b4c5f30-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f826dd-695db48787-488nc\" (UID: \"e586a1a8-85e4-47df-b62a-72c35b4c5f30\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-695db48787-488nc" Oct 30 18:20:35.491071 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:35.491040 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr6rx\" (UniqueName: \"kubernetes.io/projected/e586a1a8-85e4-47df-b62a-72c35b4c5f30-kube-api-access-xr6rx\") pod \"payload-pre-processing-e2e-ipp-b-f826dd-695db48787-488nc\" (UID: \"e586a1a8-85e4-47df-b62a-72c35b4c5f30\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-695db48787-488nc" Oct 30 18:20:35.696825 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:35.693645 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f826dd-b9f9b8586-jjsqm"] Oct 30 18:20:35.696825 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:35.695345 4442 cpu_manager.go:414] "containerMap: removing container" podUID="0e328cf8-f89f-4f08-aafb-d481dd61ffd3" containerName="maas-api" Oct 30 18:20:35.696825 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:35.695367 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e328cf8-f89f-4f08-aafb-d481dd61ffd3" containerName="maas-api" Oct 30 18:20:35.849310 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:35.760058 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e586a1a8-85e4-47df-b62a-72c35b4c5f30-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f826dd-695db48787-488nc\" (UID: \"e586a1a8-85e4-47df-b62a-72c35b4c5f30\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-695db48787-488nc" Oct 30 18:20:35.849310 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:35.760997 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f826dd-b9f9b8586-jjsqm" Oct 30 18:20:35.849310 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:35.774980 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr6rx\" (UniqueName: \"kubernetes.io/projected/e586a1a8-85e4-47df-b62a-72c35b4c5f30-kube-api-access-xr6rx\") pod \"payload-pre-processing-e2e-ipp-b-f826dd-695db48787-488nc\" (UID: \"e586a1a8-85e4-47df-b62a-72c35b4c5f30\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-695db48787-488nc" Oct 30 18:20:35.849310 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:35.776376 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e586a1a8-85e4-47df-b62a-72c35b4c5f30-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f826dd-695db48787-488nc\" (UID: \"e586a1a8-85e4-47df-b62a-72c35b4c5f30\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-695db48787-488nc" Oct 30 18:20:35.849310 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:35.835469 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr6rx\" (UniqueName: \"kubernetes.io/projected/e586a1a8-85e4-47df-b62a-72c35b4c5f30-kube-api-access-xr6rx\") pod \"payload-pre-processing-e2e-ipp-b-f826dd-695db48787-488nc\" (UID: \"e586a1a8-85e4-47df-b62a-72c35b4c5f30\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-695db48787-488nc" Oct 30 18:20:35.875304 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:35.874022 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f826dd-b9f9b8586-jjsqm"] Oct 30 18:20:35.878317 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:35.878270 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dndwr\" (UniqueName: \"kubernetes.io/projected/7cd95b94-2685-45a6-a939-0b93af7457d1-kube-api-access-dndwr\") pod \"payload-processing-e2e-ipp-b-f826dd-b9f9b8586-jjsqm\" (UID: \"7cd95b94-2685-45a6-a939-0b93af7457d1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f826dd-b9f9b8586-jjsqm" Oct 30 18:20:35.878566 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:35.878549 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cd95b94-2685-45a6-a939-0b93af7457d1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f826dd-b9f9b8586-jjsqm\" (UID: \"7cd95b94-2685-45a6-a939-0b93af7457d1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f826dd-b9f9b8586-jjsqm" Oct 30 18:20:35.980265 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:35.980124 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dndwr\" (UniqueName: \"kubernetes.io/projected/7cd95b94-2685-45a6-a939-0b93af7457d1-kube-api-access-dndwr\") pod \"payload-processing-e2e-ipp-b-f826dd-b9f9b8586-jjsqm\" (UID: \"7cd95b94-2685-45a6-a939-0b93af7457d1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f826dd-b9f9b8586-jjsqm" Oct 30 18:20:35.980265 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:35.980205 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cd95b94-2685-45a6-a939-0b93af7457d1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f826dd-b9f9b8586-jjsqm\" (UID: \"7cd95b94-2685-45a6-a939-0b93af7457d1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f826dd-b9f9b8586-jjsqm" Oct 30 18:20:35.980950 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:35.980921 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cd95b94-2685-45a6-a939-0b93af7457d1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f826dd-b9f9b8586-jjsqm\" (UID: \"7cd95b94-2685-45a6-a939-0b93af7457d1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f826dd-b9f9b8586-jjsqm" Oct 30 18:20:35.999753 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:35.999717 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dndwr\" (UniqueName: \"kubernetes.io/projected/7cd95b94-2685-45a6-a939-0b93af7457d1-kube-api-access-dndwr\") pod \"payload-processing-e2e-ipp-b-f826dd-b9f9b8586-jjsqm\" (UID: \"7cd95b94-2685-45a6-a939-0b93af7457d1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f826dd-b9f9b8586-jjsqm" Oct 30 18:20:36.053687 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:36.053660 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-695db48787-488nc" Oct 30 18:20:36.220817 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:36.220775 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f826dd-b9f9b8586-jjsqm" Oct 30 18:20:36.373636 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:20:36.373607 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode586a1a8_85e4_47df_b62a_72c35b4c5f30.slice/crio-a2104f48eaf4c6cbe6043c47b6ef9e42cc06d81c26379cedc120a58f5c556b45 WatchSource:0}: Error finding container a2104f48eaf4c6cbe6043c47b6ef9e42cc06d81c26379cedc120a58f5c556b45: Status 404 returned error can't find the container with id a2104f48eaf4c6cbe6043c47b6ef9e42cc06d81c26379cedc120a58f5c556b45 Oct 30 18:20:36.375275 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:36.375248 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-695db48787-488nc"] Oct 30 18:20:36.534672 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:20:36.534637 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cd95b94_2685_45a6_a939_0b93af7457d1.slice/crio-dfbaef48748521db352bebef0a998ce578c73b2b5b06b0ac4424466367b51b3c WatchSource:0}: Error finding container dfbaef48748521db352bebef0a998ce578c73b2b5b06b0ac4424466367b51b3c: Status 404 returned error can't find the container with id dfbaef48748521db352bebef0a998ce578c73b2b5b06b0ac4424466367b51b3c Oct 30 18:20:36.535651 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:36.535617 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f826dd-b9f9b8586-jjsqm"] Oct 30 18:20:36.944026 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:36.943995 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-695db48787-488nc" event={"ID":"e586a1a8-85e4-47df-b62a-72c35b4c5f30","Type":"ContainerStarted","Data":"f6d6fad0054451f82d7dcd04c392a277fa2f347fa27e654c46facdc99a59a0dc"} Oct 30 18:20:36.944252 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:36.944057 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-695db48787-488nc" event={"ID":"e586a1a8-85e4-47df-b62a-72c35b4c5f30","Type":"ContainerStarted","Data":"a2104f48eaf4c6cbe6043c47b6ef9e42cc06d81c26379cedc120a58f5c556b45"} Oct 30 18:20:36.944252 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:36.944157 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-695db48787-488nc" Oct 30 18:20:36.946104 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:36.946077 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f826dd-b9f9b8586-jjsqm" event={"ID":"7cd95b94-2685-45a6-a939-0b93af7457d1","Type":"ContainerStarted","Data":"9bce653a92b49baf5437e2fabe6def341779721ab51220fd5ecadf1dbad45e3b"} Oct 30 18:20:36.946261 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:36.946127 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f826dd-b9f9b8586-jjsqm" event={"ID":"7cd95b94-2685-45a6-a939-0b93af7457d1","Type":"ContainerStarted","Data":"dfbaef48748521db352bebef0a998ce578c73b2b5b06b0ac4424466367b51b3c"} Oct 30 18:20:36.946261 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:36.946183 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f826dd-b9f9b8586-jjsqm" Oct 30 18:20:36.987077 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:36.987016 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-695db48787-488nc" podStartSLOduration=1.9869975119999999 podStartE2EDuration="1.986997512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:36.982961276 +0000 UTC m=+1334.073023620" watchObservedRunningTime="2026-10-30 18:20:36.986997512 +0000 UTC m=+1334.077059851" Oct 30 18:20:37.023010 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:37.022949 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f826dd-b9f9b8586-jjsqm" podStartSLOduration=2.022930175 podStartE2EDuration="2.022930175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:37.02043924 +0000 UTC m=+1334.110501582" watchObservedRunningTime="2026-10-30 18:20:37.022930175 +0000 UTC m=+1334.112992515" Oct 30 18:20:41.421997 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:41.421964 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff"] Oct 30 18:20:41.422508 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:41.422153 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" podUID="ec2a970c-4563-416f-9357-379fbec2f852" containerName="istio-proxy" containerID="cri-o://81b63abe77385480a2f2d4e0a273e4d4d98eb61dd19f8ca7e7ef46b21e33dffc" gracePeriod=30 Oct 30 18:20:41.438778 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:41.436569 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj"] Oct 30 18:20:41.451607 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:41.443118 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp"] Oct 30 18:20:41.451607 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:41.451245 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz"] Oct 30 18:20:41.451607 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:41.451439 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" podUID="5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b" containerName="istio-proxy" containerID="cri-o://c107c0144fd8662e10c3814030d867147b8aa0a211d6b1a9cc9ecf9081340b67" gracePeriod=30 Oct 30 18:20:41.451607 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:41.449408 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" podUID="07931e20-5652-4e4c-a9d4-6cec3a6c7163" containerName="istio-proxy" containerID="cri-o://fa87941b15d713233cd7be7cb46cfc6416cc195c2dcf625ae02c31e329384b97" gracePeriod=30 Oct 30 18:20:41.453170 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:41.451747 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" podUID="5fae7458-0960-4008-b3ae-79d4518ea89e" containerName="istio-proxy" containerID="cri-o://c861c8220d8ad843ba519bc95760067264d3f944a9e4b1493ba8e3eb54fc56fa" gracePeriod=30 Oct 30 18:20:45.184388 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:45.184346 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt"] Oct 30 18:20:45.190633 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:45.190610 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" Oct 30 18:20:45.219496 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:45.219464 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-76135e-openshift-default-dockercfg-5hkcg\"" Oct 30 18:20:45.225171 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:45.225126 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt"] Oct 30 18:20:45.353200 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:45.353159 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22d32885-e0c5-475e-b497-d10d09015d22-istio-podinfo\") pod \"e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt\" (UID: \"22d32885-e0c5-475e-b497-d10d09015d22\") " pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" Oct 30 18:20:45.353548 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:45.353528 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22d32885-e0c5-475e-b497-d10d09015d22-istiod-ca-cert\") pod \"e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt\" (UID: \"22d32885-e0c5-475e-b497-d10d09015d22\") " pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" Oct 30 18:20:45.353735 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:45.353717 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv5f4\" (UniqueName: \"kubernetes.io/projected/22d32885-e0c5-475e-b497-d10d09015d22-kube-api-access-gv5f4\") pod \"e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt\" (UID: \"22d32885-e0c5-475e-b497-d10d09015d22\") " pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" Oct 30 18:20:45.353876 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:45.353860 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22d32885-e0c5-475e-b497-d10d09015d22-istio-data\") pod \"e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt\" (UID: \"22d32885-e0c5-475e-b497-d10d09015d22\") " pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" Oct 30 18:20:45.354045 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:45.354031 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22d32885-e0c5-475e-b497-d10d09015d22-workload-socket\") pod \"e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt\" (UID: \"22d32885-e0c5-475e-b497-d10d09015d22\") " pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" Oct 30 18:20:45.354189 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:45.354173 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22d32885-e0c5-475e-b497-d10d09015d22-credential-socket\") pod \"e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt\" (UID: \"22d32885-e0c5-475e-b497-d10d09015d22\") " pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" Oct 30 18:20:45.354363 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:45.354346 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22d32885-e0c5-475e-b497-d10d09015d22-workload-certs\") pod \"e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt\" (UID: \"22d32885-e0c5-475e-b497-d10d09015d22\") " pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" Oct 30 18:20:45.354491 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:45.354476 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22d32885-e0c5-475e-b497-d10d09015d22-istio-envoy\") pod \"e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt\" (UID: \"22d32885-e0c5-475e-b497-d10d09015d22\") " pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" Oct 30 18:20:45.354678 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:45.354655 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22d32885-e0c5-475e-b497-d10d09015d22-istio-token\") pod \"e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt\" (UID: \"22d32885-e0c5-475e-b497-d10d09015d22\") " pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" Oct 30 18:20:45.468207 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:45.462291 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22d32885-e0c5-475e-b497-d10d09015d22-istio-podinfo\") pod \"e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt\" (UID: \"22d32885-e0c5-475e-b497-d10d09015d22\") " pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" Oct 30 18:20:45.468207 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:45.462369 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22d32885-e0c5-475e-b497-d10d09015d22-istiod-ca-cert\") pod \"e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt\" (UID: \"22d32885-e0c5-475e-b497-d10d09015d22\") " pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" Oct 30 18:20:45.468207 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:45.462441 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv5f4\" (UniqueName: \"kubernetes.io/projected/22d32885-e0c5-475e-b497-d10d09015d22-kube-api-access-gv5f4\") pod \"e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt\" (UID: \"22d32885-e0c5-475e-b497-d10d09015d22\") " pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" Oct 30 18:20:45.468207 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:45.462472 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22d32885-e0c5-475e-b497-d10d09015d22-istio-data\") pod \"e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt\" (UID: \"22d32885-e0c5-475e-b497-d10d09015d22\") " pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" Oct 30 18:20:45.468207 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:45.462565 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22d32885-e0c5-475e-b497-d10d09015d22-workload-socket\") pod \"e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt\" (UID: \"22d32885-e0c5-475e-b497-d10d09015d22\") " pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" Oct 30 18:20:45.468207 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:45.462598 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22d32885-e0c5-475e-b497-d10d09015d22-credential-socket\") pod \"e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt\" (UID: \"22d32885-e0c5-475e-b497-d10d09015d22\") " pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" Oct 30 18:20:45.468207 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:45.462638 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22d32885-e0c5-475e-b497-d10d09015d22-workload-certs\") pod \"e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt\" (UID: \"22d32885-e0c5-475e-b497-d10d09015d22\") " pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" Oct 30 18:20:45.468207 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:45.462676 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22d32885-e0c5-475e-b497-d10d09015d22-istio-envoy\") pod \"e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt\" (UID: \"22d32885-e0c5-475e-b497-d10d09015d22\") " pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" Oct 30 18:20:45.468207 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:45.462760 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22d32885-e0c5-475e-b497-d10d09015d22-istio-token\") pod \"e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt\" (UID: \"22d32885-e0c5-475e-b497-d10d09015d22\") " pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" Oct 30 18:20:45.468207 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:45.464242 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22d32885-e0c5-475e-b497-d10d09015d22-workload-socket\") pod \"e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt\" (UID: \"22d32885-e0c5-475e-b497-d10d09015d22\") " pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" Oct 30 18:20:45.468207 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:45.464917 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22d32885-e0c5-475e-b497-d10d09015d22-istiod-ca-cert\") pod \"e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt\" (UID: \"22d32885-e0c5-475e-b497-d10d09015d22\") " pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" Oct 30 18:20:45.496160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:45.473609 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22d32885-e0c5-475e-b497-d10d09015d22-credential-socket\") pod \"e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt\" (UID: \"22d32885-e0c5-475e-b497-d10d09015d22\") " pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" Oct 30 18:20:45.496462 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:45.482179 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22d32885-e0c5-475e-b497-d10d09015d22-istio-data\") pod \"e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt\" (UID: \"22d32885-e0c5-475e-b497-d10d09015d22\") " pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" Oct 30 18:20:45.496637 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:45.482789 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22d32885-e0c5-475e-b497-d10d09015d22-istio-podinfo\") pod \"e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt\" (UID: \"22d32885-e0c5-475e-b497-d10d09015d22\") " pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" Oct 30 18:20:45.496754 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:45.483041 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22d32885-e0c5-475e-b497-d10d09015d22-workload-certs\") pod \"e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt\" (UID: \"22d32885-e0c5-475e-b497-d10d09015d22\") " pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" Oct 30 18:20:45.499554 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:45.499528 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22d32885-e0c5-475e-b497-d10d09015d22-istio-token\") pod \"e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt\" (UID: \"22d32885-e0c5-475e-b497-d10d09015d22\") " pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" Oct 30 18:20:45.516493 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:45.516459 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22d32885-e0c5-475e-b497-d10d09015d22-istio-envoy\") pod \"e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt\" (UID: \"22d32885-e0c5-475e-b497-d10d09015d22\") " pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" Oct 30 18:20:45.517368 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:45.517341 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv5f4\" (UniqueName: \"kubernetes.io/projected/22d32885-e0c5-475e-b497-d10d09015d22-kube-api-access-gv5f4\") pod \"e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt\" (UID: \"22d32885-e0c5-475e-b497-d10d09015d22\") " pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" Oct 30 18:20:45.524442 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:45.524417 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" Oct 30 18:20:46.041801 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.041770 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt"] Oct 30 18:20:46.042352 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:20:46.042322 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22d32885_e0c5_475e_b497_d10d09015d22.slice/crio-990fdd4289c572c7cddf1d24338e5bb599e99fd5aea8abc5bd95997f33ea194c WatchSource:0}: Error finding container 990fdd4289c572c7cddf1d24338e5bb599e99fd5aea8abc5bd95997f33ea194c: Status 404 returned error can't find the container with id 990fdd4289c572c7cddf1d24338e5bb599e99fd5aea8abc5bd95997f33ea194c Oct 30 18:20:46.045200 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.045164 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:20:46.045305 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.045246 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:20:46.045305 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.045292 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:20:46.125770 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.125740 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" event={"ID":"22d32885-e0c5-475e-b497-d10d09015d22","Type":"ContainerStarted","Data":"990fdd4289c572c7cddf1d24338e5bb599e99fd5aea8abc5bd95997f33ea194c"} Oct 30 18:20:46.809196 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.809164 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" Oct 30 18:20:46.857848 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.857518 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" Oct 30 18:20:46.905687 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.905627 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec2a970c-4563-416f-9357-379fbec2f852-istio-token\" (UniqueName: \"kubernetes.io/projected/ec2a970c-4563-416f-9357-379fbec2f852-istio-token\") pod \"ec2a970c-4563-416f-9357-379fbec2f852\" (UID: \"ec2a970c-4563-416f-9357-379fbec2f852\") " Oct 30 18:20:46.905687 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.905685 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fae7458-0960-4008-b3ae-79d4518ea89e-istio-token\" (UniqueName: \"kubernetes.io/projected/5fae7458-0960-4008-b3ae-79d4518ea89e-istio-token\") pod \"5fae7458-0960-4008-b3ae-79d4518ea89e\" (UID: \"5fae7458-0960-4008-b3ae-79d4518ea89e\") " Oct 30 18:20:46.905970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.905765 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec2a970c-4563-416f-9357-379fbec2f852-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2a970c-4563-416f-9357-379fbec2f852-workload-socket\") pod \"ec2a970c-4563-416f-9357-379fbec2f852\" (UID: \"ec2a970c-4563-416f-9357-379fbec2f852\") " Oct 30 18:20:46.905970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.905850 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fae7458-0960-4008-b3ae-79d4518ea89e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fae7458-0960-4008-b3ae-79d4518ea89e-workload-socket\") pod \"5fae7458-0960-4008-b3ae-79d4518ea89e\" (UID: \"5fae7458-0960-4008-b3ae-79d4518ea89e\") " Oct 30 18:20:46.905970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.905884 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec2a970c-4563-416f-9357-379fbec2f852-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec2a970c-4563-416f-9357-379fbec2f852-workload-certs\") pod \"ec2a970c-4563-416f-9357-379fbec2f852\" (UID: \"ec2a970c-4563-416f-9357-379fbec2f852\") " Oct 30 18:20:46.905970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.905948 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fae7458-0960-4008-b3ae-79d4518ea89e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fae7458-0960-4008-b3ae-79d4518ea89e-credential-socket\") pod \"5fae7458-0960-4008-b3ae-79d4518ea89e\" (UID: \"5fae7458-0960-4008-b3ae-79d4518ea89e\") " Oct 30 18:20:46.906179 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.905993 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fae7458-0960-4008-b3ae-79d4518ea89e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fae7458-0960-4008-b3ae-79d4518ea89e-istio-envoy\") pod \"5fae7458-0960-4008-b3ae-79d4518ea89e\" (UID: \"5fae7458-0960-4008-b3ae-79d4518ea89e\") " Oct 30 18:20:46.906179 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.906119 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec2a970c-4563-416f-9357-379fbec2f852-kube-api-access-mkf2l\" (UniqueName: \"kubernetes.io/projected/ec2a970c-4563-416f-9357-379fbec2f852-kube-api-access-mkf2l\") pod \"ec2a970c-4563-416f-9357-379fbec2f852\" (UID: \"ec2a970c-4563-416f-9357-379fbec2f852\") " Oct 30 18:20:46.906179 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.906162 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec2a970c-4563-416f-9357-379fbec2f852-workload-socket" pod "ec2a970c-4563-416f-9357-379fbec2f852" (UID: "ec2a970c-4563-416f-9357-379fbec2f852"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:20:46.906329 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.906192 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fae7458-0960-4008-b3ae-79d4518ea89e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fae7458-0960-4008-b3ae-79d4518ea89e-istio-data\") pod \"5fae7458-0960-4008-b3ae-79d4518ea89e\" (UID: \"5fae7458-0960-4008-b3ae-79d4518ea89e\") " Oct 30 18:20:46.906329 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.906233 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec2a970c-4563-416f-9357-379fbec2f852-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec2a970c-4563-416f-9357-379fbec2f852-istiod-ca-cert\") pod \"ec2a970c-4563-416f-9357-379fbec2f852\" (UID: \"ec2a970c-4563-416f-9357-379fbec2f852\") " Oct 30 18:20:46.906329 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.906262 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fae7458-0960-4008-b3ae-79d4518ea89e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fae7458-0960-4008-b3ae-79d4518ea89e-workload-certs\") pod \"5fae7458-0960-4008-b3ae-79d4518ea89e\" (UID: \"5fae7458-0960-4008-b3ae-79d4518ea89e\") " Oct 30 18:20:46.906565 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.906353 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fae7458-0960-4008-b3ae-79d4518ea89e-credential-socket" pod "5fae7458-0960-4008-b3ae-79d4518ea89e" (UID: "5fae7458-0960-4008-b3ae-79d4518ea89e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:20:46.906565 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.906539 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec2a970c-4563-416f-9357-379fbec2f852-workload-certs" pod "ec2a970c-4563-416f-9357-379fbec2f852" (UID: "ec2a970c-4563-416f-9357-379fbec2f852"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:20:46.906699 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.906685 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fae7458-0960-4008-b3ae-79d4518ea89e-istio-data" pod "5fae7458-0960-4008-b3ae-79d4518ea89e" (UID: "5fae7458-0960-4008-b3ae-79d4518ea89e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:20:46.910823 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.910556 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fae7458-0960-4008-b3ae-79d4518ea89e-istio-envoy" pod "5fae7458-0960-4008-b3ae-79d4518ea89e" (UID: "5fae7458-0960-4008-b3ae-79d4518ea89e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:20:46.910961 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.910940 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fae7458-0960-4008-b3ae-79d4518ea89e-workload-certs" pod "5fae7458-0960-4008-b3ae-79d4518ea89e" (UID: "5fae7458-0960-4008-b3ae-79d4518ea89e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:20:46.911497 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.911471 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec2a970c-4563-416f-9357-379fbec2f852-istio-data" pod "ec2a970c-4563-416f-9357-379fbec2f852" (UID: "ec2a970c-4563-416f-9357-379fbec2f852"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:20:46.912403 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.912360 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec2a970c-4563-416f-9357-379fbec2f852-istiod-ca-cert" pod "ec2a970c-4563-416f-9357-379fbec2f852" (UID: "ec2a970c-4563-416f-9357-379fbec2f852"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:20:46.915195 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.906191 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fae7458-0960-4008-b3ae-79d4518ea89e-workload-socket" pod "5fae7458-0960-4008-b3ae-79d4518ea89e" (UID: "5fae7458-0960-4008-b3ae-79d4518ea89e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:20:46.915195 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.906336 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec2a970c-4563-416f-9357-379fbec2f852-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec2a970c-4563-416f-9357-379fbec2f852-istio-data\") pod \"ec2a970c-4563-416f-9357-379fbec2f852\" (UID: \"ec2a970c-4563-416f-9357-379fbec2f852\") " Oct 30 18:20:46.915195 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.912638 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fae7458-0960-4008-b3ae-79d4518ea89e-kube-api-access-r59zc\" (UniqueName: \"kubernetes.io/projected/5fae7458-0960-4008-b3ae-79d4518ea89e-kube-api-access-r59zc\") pod \"5fae7458-0960-4008-b3ae-79d4518ea89e\" (UID: \"5fae7458-0960-4008-b3ae-79d4518ea89e\") " Oct 30 18:20:46.915195 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.912678 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec2a970c-4563-416f-9357-379fbec2f852-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec2a970c-4563-416f-9357-379fbec2f852-istio-envoy\") pod \"ec2a970c-4563-416f-9357-379fbec2f852\" (UID: \"ec2a970c-4563-416f-9357-379fbec2f852\") " Oct 30 18:20:46.915195 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.912726 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec2a970c-4563-416f-9357-379fbec2f852-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec2a970c-4563-416f-9357-379fbec2f852-istio-podinfo\") pod \"ec2a970c-4563-416f-9357-379fbec2f852\" (UID: \"ec2a970c-4563-416f-9357-379fbec2f852\") " Oct 30 18:20:46.915195 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.912778 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5fae7458-0960-4008-b3ae-79d4518ea89e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fae7458-0960-4008-b3ae-79d4518ea89e-istio-podinfo\") pod \"5fae7458-0960-4008-b3ae-79d4518ea89e\" (UID: \"5fae7458-0960-4008-b3ae-79d4518ea89e\") " Oct 30 18:20:46.915195 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.912808 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec2a970c-4563-416f-9357-379fbec2f852-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2a970c-4563-416f-9357-379fbec2f852-credential-socket\") pod \"ec2a970c-4563-416f-9357-379fbec2f852\" (UID: \"ec2a970c-4563-416f-9357-379fbec2f852\") " Oct 30 18:20:46.915195 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.912859 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fae7458-0960-4008-b3ae-79d4518ea89e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fae7458-0960-4008-b3ae-79d4518ea89e-istiod-ca-cert\") pod \"5fae7458-0960-4008-b3ae-79d4518ea89e\" (UID: \"5fae7458-0960-4008-b3ae-79d4518ea89e\") " Oct 30 18:20:46.915195 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.914460 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fae7458-0960-4008-b3ae-79d4518ea89e-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:46.915195 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.914481 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fae7458-0960-4008-b3ae-79d4518ea89e-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:46.915195 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.914498 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fae7458-0960-4008-b3ae-79d4518ea89e-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:46.915195 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.914511 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec2a970c-4563-416f-9357-379fbec2f852-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:46.915195 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.914524 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fae7458-0960-4008-b3ae-79d4518ea89e-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:46.915195 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.914538 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec2a970c-4563-416f-9357-379fbec2f852-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:46.915195 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.914550 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2a970c-4563-416f-9357-379fbec2f852-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:46.915195 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.914563 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fae7458-0960-4008-b3ae-79d4518ea89e-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:46.915195 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.914577 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec2a970c-4563-416f-9357-379fbec2f852-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:46.915195 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.914925 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fae7458-0960-4008-b3ae-79d4518ea89e-istiod-ca-cert" pod "5fae7458-0960-4008-b3ae-79d4518ea89e" (UID: "5fae7458-0960-4008-b3ae-79d4518ea89e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:20:46.916330 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.915619 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec2a970c-4563-416f-9357-379fbec2f852-istio-token" pod "ec2a970c-4563-416f-9357-379fbec2f852" (UID: "ec2a970c-4563-416f-9357-379fbec2f852"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:20:46.916330 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.915868 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec2a970c-4563-416f-9357-379fbec2f852-credential-socket" pod "ec2a970c-4563-416f-9357-379fbec2f852" (UID: "ec2a970c-4563-416f-9357-379fbec2f852"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:20:46.920634 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.920608 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fae7458-0960-4008-b3ae-79d4518ea89e-istio-token" pod "5fae7458-0960-4008-b3ae-79d4518ea89e" (UID: "5fae7458-0960-4008-b3ae-79d4518ea89e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:20:46.921729 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.921704 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec2a970c-4563-416f-9357-379fbec2f852-kube-api-access-mkf2l" pod "ec2a970c-4563-416f-9357-379fbec2f852" (UID: "ec2a970c-4563-416f-9357-379fbec2f852"). InnerVolumeSpecName "kube-api-access-mkf2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:20:46.942915 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.942305 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec2a970c-4563-416f-9357-379fbec2f852-istio-envoy" pod "ec2a970c-4563-416f-9357-379fbec2f852" (UID: "ec2a970c-4563-416f-9357-379fbec2f852"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:20:46.942915 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.942313 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5fae7458-0960-4008-b3ae-79d4518ea89e-istio-podinfo" pod "5fae7458-0960-4008-b3ae-79d4518ea89e" (UID: "5fae7458-0960-4008-b3ae-79d4518ea89e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:20:46.945484 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.945459 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec2a970c-4563-416f-9357-379fbec2f852-istio-podinfo" pod "ec2a970c-4563-416f-9357-379fbec2f852" (UID: "ec2a970c-4563-416f-9357-379fbec2f852"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:20:46.947077 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.947051 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fae7458-0960-4008-b3ae-79d4518ea89e-kube-api-access-r59zc" pod "5fae7458-0960-4008-b3ae-79d4518ea89e" (UID: "5fae7458-0960-4008-b3ae-79d4518ea89e"). InnerVolumeSpecName "kube-api-access-r59zc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:20:46.966188 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.965084 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" Oct 30 18:20:46.970939 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:46.970912 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" Oct 30 18:20:47.015630 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.015595 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07931e20-5652-4e4c-a9d4-6cec3a6c7163-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07931e20-5652-4e4c-a9d4-6cec3a6c7163-istio-envoy\") pod \"07931e20-5652-4e4c-a9d4-6cec3a6c7163\" (UID: \"07931e20-5652-4e4c-a9d4-6cec3a6c7163\") " Oct 30 18:20:47.015794 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.015655 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-istio-podinfo\") pod \"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b\" (UID: \"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b\") " Oct 30 18:20:47.015794 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.015707 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07931e20-5652-4e4c-a9d4-6cec3a6c7163-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07931e20-5652-4e4c-a9d4-6cec3a6c7163-istio-podinfo\") pod \"07931e20-5652-4e4c-a9d4-6cec3a6c7163\" (UID: \"07931e20-5652-4e4c-a9d4-6cec3a6c7163\") " Oct 30 18:20:47.015794 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.015764 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-credential-socket\") pod \"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b\" (UID: \"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b\") " Oct 30 18:20:47.015920 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.015824 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07931e20-5652-4e4c-a9d4-6cec3a6c7163-kube-api-access-9mgqt\" (UniqueName: \"kubernetes.io/projected/07931e20-5652-4e4c-a9d4-6cec3a6c7163-kube-api-access-9mgqt\") pod \"07931e20-5652-4e4c-a9d4-6cec3a6c7163\" (UID: \"07931e20-5652-4e4c-a9d4-6cec3a6c7163\") " Oct 30 18:20:47.015920 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.015868 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-workload-certs\") pod \"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b\" (UID: \"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b\") " Oct 30 18:20:47.015920 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.015908 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-istio-token\" (UniqueName: \"kubernetes.io/projected/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-istio-token\") pod \"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b\" (UID: \"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b\") " Oct 30 18:20:47.016025 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.015943 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07931e20-5652-4e4c-a9d4-6cec3a6c7163-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07931e20-5652-4e4c-a9d4-6cec3a6c7163-credential-socket\") pod \"07931e20-5652-4e4c-a9d4-6cec3a6c7163\" (UID: \"07931e20-5652-4e4c-a9d4-6cec3a6c7163\") " Oct 30 18:20:47.016025 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.015978 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-kube-api-access-b26ph\" (UniqueName: \"kubernetes.io/projected/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-kube-api-access-b26ph\") pod \"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b\" (UID: \"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b\") " Oct 30 18:20:47.016098 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.016090 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07931e20-5652-4e4c-a9d4-6cec3a6c7163-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07931e20-5652-4e4c-a9d4-6cec3a6c7163-istiod-ca-cert\") pod \"07931e20-5652-4e4c-a9d4-6cec3a6c7163\" (UID: \"07931e20-5652-4e4c-a9d4-6cec3a6c7163\") " Oct 30 18:20:47.017668 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.017630 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07931e20-5652-4e4c-a9d4-6cec3a6c7163-credential-socket" pod "07931e20-5652-4e4c-a9d4-6cec3a6c7163" (UID: "07931e20-5652-4e4c-a9d4-6cec3a6c7163"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:20:47.018011 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.017984 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-workload-certs" pod "5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b" (UID: "5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:20:47.018369 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.016155 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07931e20-5652-4e4c-a9d4-6cec3a6c7163-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07931e20-5652-4e4c-a9d4-6cec3a6c7163-workload-certs\") pod \"07931e20-5652-4e4c-a9d4-6cec3a6c7163\" (UID: \"07931e20-5652-4e4c-a9d4-6cec3a6c7163\") " Oct 30 18:20:47.018475 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.018429 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07931e20-5652-4e4c-a9d4-6cec3a6c7163-istio-token\" (UniqueName: \"kubernetes.io/projected/07931e20-5652-4e4c-a9d4-6cec3a6c7163-istio-token\") pod \"07931e20-5652-4e4c-a9d4-6cec3a6c7163\" (UID: \"07931e20-5652-4e4c-a9d4-6cec3a6c7163\") " Oct 30 18:20:47.018559 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.018498 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-istio-envoy\") pod \"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b\" (UID: \"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b\") " Oct 30 18:20:47.018603 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.018561 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-istio-data\") pod \"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b\" (UID: \"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b\") " Oct 30 18:20:47.018654 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.018623 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07931e20-5652-4e4c-a9d4-6cec3a6c7163-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07931e20-5652-4e4c-a9d4-6cec3a6c7163-workload-socket\") pod \"07931e20-5652-4e4c-a9d4-6cec3a6c7163\" (UID: \"07931e20-5652-4e4c-a9d4-6cec3a6c7163\") " Oct 30 18:20:47.018913 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.018888 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07931e20-5652-4e4c-a9d4-6cec3a6c7163-workload-socket" pod "07931e20-5652-4e4c-a9d4-6cec3a6c7163" (UID: "07931e20-5652-4e4c-a9d4-6cec3a6c7163"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:20:47.022437 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.022395 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-credential-socket" pod "5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b" (UID: "5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:20:47.022663 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.022639 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-workload-socket\") pod \"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b\" (UID: \"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b\") " Oct 30 18:20:47.022943 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.022925 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07931e20-5652-4e4c-a9d4-6cec3a6c7163-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07931e20-5652-4e4c-a9d4-6cec3a6c7163-istio-data\") pod \"07931e20-5652-4e4c-a9d4-6cec3a6c7163\" (UID: \"07931e20-5652-4e4c-a9d4-6cec3a6c7163\") " Oct 30 18:20:47.028225 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.023802 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-istio-token" pod "5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b" (UID: "5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:20:47.028225 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.024030 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-workload-socket" pod "5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b" (UID: "5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:20:47.028225 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.025325 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-istio-data" pod "5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b" (UID: "5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:20:47.028225 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.025636 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07931e20-5652-4e4c-a9d4-6cec3a6c7163-istiod-ca-cert" pod "07931e20-5652-4e4c-a9d4-6cec3a6c7163" (UID: "07931e20-5652-4e4c-a9d4-6cec3a6c7163"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:20:47.028225 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.025885 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07931e20-5652-4e4c-a9d4-6cec3a6c7163-workload-certs" pod "07931e20-5652-4e4c-a9d4-6cec3a6c7163" (UID: "07931e20-5652-4e4c-a9d4-6cec3a6c7163"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:20:47.028225 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.026173 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07931e20-5652-4e4c-a9d4-6cec3a6c7163-istio-data" pod "07931e20-5652-4e4c-a9d4-6cec3a6c7163" (UID: "07931e20-5652-4e4c-a9d4-6cec3a6c7163"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:20:47.028225 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.027384 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07931e20-5652-4e4c-a9d4-6cec3a6c7163-istio-envoy" pod "07931e20-5652-4e4c-a9d4-6cec3a6c7163" (UID: "07931e20-5652-4e4c-a9d4-6cec3a6c7163"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:20:47.029341 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.029314 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-istiod-ca-cert\") pod \"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b\" (UID: \"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b\") " Oct 30 18:20:47.029960 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.029833 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-istiod-ca-cert" pod "5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b" (UID: "5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:20:47.030246 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.030220 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-istio-envoy" pod "5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b" (UID: "5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:20:47.030784 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.030763 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07931e20-5652-4e4c-a9d4-6cec3a6c7163-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:47.030867 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.030787 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:47.030867 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.030803 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07931e20-5652-4e4c-a9d4-6cec3a6c7163-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:47.030867 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.030815 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:47.030867 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.030829 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:47.030867 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.030845 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkf2l\" (UniqueName: \"kubernetes.io/projected/ec2a970c-4563-416f-9357-379fbec2f852-kube-api-access-mkf2l\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:47.030867 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.030858 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:47.031352 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.030872 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07931e20-5652-4e4c-a9d4-6cec3a6c7163-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:47.031352 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.030904 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r59zc\" (UniqueName: \"kubernetes.io/projected/5fae7458-0960-4008-b3ae-79d4518ea89e-kube-api-access-r59zc\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:47.031352 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.030920 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec2a970c-4563-416f-9357-379fbec2f852-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:47.031352 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.030934 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07931e20-5652-4e4c-a9d4-6cec3a6c7163-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:47.031352 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.030948 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec2a970c-4563-416f-9357-379fbec2f852-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:47.031352 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.030963 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07931e20-5652-4e4c-a9d4-6cec3a6c7163-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:47.031352 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.030977 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fae7458-0960-4008-b3ae-79d4518ea89e-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:47.031352 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.030993 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2a970c-4563-416f-9357-379fbec2f852-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:47.031352 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.031009 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fae7458-0960-4008-b3ae-79d4518ea89e-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:47.031352 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.031022 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:47.031352 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.031034 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:47.031352 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.031048 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec2a970c-4563-416f-9357-379fbec2f852-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:47.031352 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.031061 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fae7458-0960-4008-b3ae-79d4518ea89e-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:47.031352 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.031075 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07931e20-5652-4e4c-a9d4-6cec3a6c7163-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:47.031352 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.031089 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:47.044694 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.044608 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-kube-api-access-b26ph" pod "5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b" (UID: "5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b"). InnerVolumeSpecName "kube-api-access-b26ph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:20:47.044694 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.044659 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07931e20-5652-4e4c-a9d4-6cec3a6c7163-istio-token" pod "07931e20-5652-4e4c-a9d4-6cec3a6c7163" (UID: "07931e20-5652-4e4c-a9d4-6cec3a6c7163"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:20:47.044915 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.044791 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07931e20-5652-4e4c-a9d4-6cec3a6c7163-kube-api-access-9mgqt" pod "07931e20-5652-4e4c-a9d4-6cec3a6c7163" (UID: "07931e20-5652-4e4c-a9d4-6cec3a6c7163"). InnerVolumeSpecName "kube-api-access-9mgqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:20:47.044915 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.044876 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-istio-podinfo" pod "5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b" (UID: "5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:20:47.046809 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.046776 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07931e20-5652-4e4c-a9d4-6cec3a6c7163-istio-podinfo" pod "07931e20-5652-4e4c-a9d4-6cec3a6c7163" (UID: "07931e20-5652-4e4c-a9d4-6cec3a6c7163"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:20:47.132367 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.132335 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07931e20-5652-4e4c-a9d4-6cec3a6c7163-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:47.132367 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.132369 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:47.132626 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.132381 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07931e20-5652-4e4c-a9d4-6cec3a6c7163-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:47.132626 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.132395 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mgqt\" (UniqueName: \"kubernetes.io/projected/07931e20-5652-4e4c-a9d4-6cec3a6c7163-kube-api-access-9mgqt\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:47.132626 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.132408 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b26ph\" (UniqueName: \"kubernetes.io/projected/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b-kube-api-access-b26ph\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:47.137942 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.137907 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" event={"ID":"22d32885-e0c5-475e-b497-d10d09015d22","Type":"ContainerStarted","Data":"77ad92f64f90785547d2304549fc0b52597b278094c6e0174453e4fab2ed6f21"} Oct 30 18:20:47.139619 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.139592 4442 generic.go:356] "Generic (PLEG): container finished" podID="ec2a970c-4563-416f-9357-379fbec2f852" containerID="81b63abe77385480a2f2d4e0a273e4d4d98eb61dd19f8ca7e7ef46b21e33dffc" exitCode=0 Oct 30 18:20:47.139741 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.139632 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" Oct 30 18:20:47.139741 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.139676 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" event={"ID":"ec2a970c-4563-416f-9357-379fbec2f852","Type":"ContainerDied","Data":"81b63abe77385480a2f2d4e0a273e4d4d98eb61dd19f8ca7e7ef46b21e33dffc"} Oct 30 18:20:47.139741 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.139738 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff" event={"ID":"ec2a970c-4563-416f-9357-379fbec2f852","Type":"ContainerDied","Data":"a334c72f59a941b5c4aef154f49bc1328b7ae345f21459d9fe734797d6fa6b99"} Oct 30 18:20:47.139913 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.139769 4442 scope.go:122] "RemoveContainer" containerID="81b63abe77385480a2f2d4e0a273e4d4d98eb61dd19f8ca7e7ef46b21e33dffc" Oct 30 18:20:47.142665 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.142642 4442 generic.go:356] "Generic (PLEG): container finished" podID="07931e20-5652-4e4c-a9d4-6cec3a6c7163" containerID="fa87941b15d713233cd7be7cb46cfc6416cc195c2dcf625ae02c31e329384b97" exitCode=0 Oct 30 18:20:47.142789 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.142680 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" Oct 30 18:20:47.142789 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.142772 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" event={"ID":"07931e20-5652-4e4c-a9d4-6cec3a6c7163","Type":"ContainerDied","Data":"fa87941b15d713233cd7be7cb46cfc6416cc195c2dcf625ae02c31e329384b97"} Oct 30 18:20:47.142905 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.142815 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj" event={"ID":"07931e20-5652-4e4c-a9d4-6cec3a6c7163","Type":"ContainerDied","Data":"d22bec5580cfc782a71804f39395e14733f4b0b2847f9c7592827bafb0a4bbf9"} Oct 30 18:20:47.145068 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.145012 4442 generic.go:356] "Generic (PLEG): container finished" podID="5fae7458-0960-4008-b3ae-79d4518ea89e" containerID="c861c8220d8ad843ba519bc95760067264d3f944a9e4b1493ba8e3eb54fc56fa" exitCode=0 Oct 30 18:20:47.145068 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.145053 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" Oct 30 18:20:47.145383 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.145312 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" event={"ID":"5fae7458-0960-4008-b3ae-79d4518ea89e","Type":"ContainerDied","Data":"c861c8220d8ad843ba519bc95760067264d3f944a9e4b1493ba8e3eb54fc56fa"} Oct 30 18:20:47.145502 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.145410 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp" event={"ID":"5fae7458-0960-4008-b3ae-79d4518ea89e","Type":"ContainerDied","Data":"5d60b5e6329dac95d3e408cbf38b062a648f96c0e6c15a2710cf1d3215218c1b"} Oct 30 18:20:47.147273 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.147253 4442 generic.go:356] "Generic (PLEG): container finished" podID="5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b" containerID="c107c0144fd8662e10c3814030d867147b8aa0a211d6b1a9cc9ecf9081340b67" exitCode=0 Oct 30 18:20:47.147387 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.147296 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" event={"ID":"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b","Type":"ContainerDied","Data":"c107c0144fd8662e10c3814030d867147b8aa0a211d6b1a9cc9ecf9081340b67"} Oct 30 18:20:47.147387 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.147307 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" Oct 30 18:20:47.147387 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.147332 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz" event={"ID":"5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b","Type":"ContainerDied","Data":"64fe6c3c2875a2e9c645c06998623494a54dad168ecfdb73b63325abbbf497d4"} Oct 30 18:20:47.154769 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.154750 4442 scope.go:122] "RemoveContainer" containerID="81b63abe77385480a2f2d4e0a273e4d4d98eb61dd19f8ca7e7ef46b21e33dffc" Oct 30 18:20:47.155077 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:20:47.155056 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81b63abe77385480a2f2d4e0a273e4d4d98eb61dd19f8ca7e7ef46b21e33dffc\": container with ID starting with 81b63abe77385480a2f2d4e0a273e4d4d98eb61dd19f8ca7e7ef46b21e33dffc not found: ID does not exist" containerID="81b63abe77385480a2f2d4e0a273e4d4d98eb61dd19f8ca7e7ef46b21e33dffc" Oct 30 18:20:47.155172 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.155084 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81b63abe77385480a2f2d4e0a273e4d4d98eb61dd19f8ca7e7ef46b21e33dffc"} err="failed to get container status \"81b63abe77385480a2f2d4e0a273e4d4d98eb61dd19f8ca7e7ef46b21e33dffc\": rpc error: code = NotFound desc = could not find container \"81b63abe77385480a2f2d4e0a273e4d4d98eb61dd19f8ca7e7ef46b21e33dffc\": container with ID starting with 81b63abe77385480a2f2d4e0a273e4d4d98eb61dd19f8ca7e7ef46b21e33dffc not found: ID does not exist" Oct 30 18:20:47.155172 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.155101 4442 scope.go:122] "RemoveContainer" containerID="fa87941b15d713233cd7be7cb46cfc6416cc195c2dcf625ae02c31e329384b97" Oct 30 18:20:47.183468 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.183442 4442 scope.go:122] "RemoveContainer" containerID="fa87941b15d713233cd7be7cb46cfc6416cc195c2dcf625ae02c31e329384b97" Oct 30 18:20:47.183941 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:20:47.183909 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa87941b15d713233cd7be7cb46cfc6416cc195c2dcf625ae02c31e329384b97\": container with ID starting with fa87941b15d713233cd7be7cb46cfc6416cc195c2dcf625ae02c31e329384b97 not found: ID does not exist" containerID="fa87941b15d713233cd7be7cb46cfc6416cc195c2dcf625ae02c31e329384b97" Oct 30 18:20:47.184097 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.183946 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa87941b15d713233cd7be7cb46cfc6416cc195c2dcf625ae02c31e329384b97"} err="failed to get container status \"fa87941b15d713233cd7be7cb46cfc6416cc195c2dcf625ae02c31e329384b97\": rpc error: code = NotFound desc = could not find container \"fa87941b15d713233cd7be7cb46cfc6416cc195c2dcf625ae02c31e329384b97\": container with ID starting with fa87941b15d713233cd7be7cb46cfc6416cc195c2dcf625ae02c31e329384b97 not found: ID does not exist" Oct 30 18:20:47.184097 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.183974 4442 scope.go:122] "RemoveContainer" containerID="c861c8220d8ad843ba519bc95760067264d3f944a9e4b1493ba8e3eb54fc56fa" Oct 30 18:20:47.188145 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.185284 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" podStartSLOduration=2.185271279 podStartE2EDuration="2.185271279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:47.18205353 +0000 UTC m=+1344.272115869" watchObservedRunningTime="2026-10-30 18:20:47.185271279 +0000 UTC m=+1344.275333617" Oct 30 18:20:47.196346 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.196309 4442 scope.go:122] "RemoveContainer" containerID="c861c8220d8ad843ba519bc95760067264d3f944a9e4b1493ba8e3eb54fc56fa" Oct 30 18:20:47.196686 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:20:47.196665 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c861c8220d8ad843ba519bc95760067264d3f944a9e4b1493ba8e3eb54fc56fa\": container with ID starting with c861c8220d8ad843ba519bc95760067264d3f944a9e4b1493ba8e3eb54fc56fa not found: ID does not exist" containerID="c861c8220d8ad843ba519bc95760067264d3f944a9e4b1493ba8e3eb54fc56fa" Oct 30 18:20:47.196773 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.196699 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c861c8220d8ad843ba519bc95760067264d3f944a9e4b1493ba8e3eb54fc56fa"} err="failed to get container status \"c861c8220d8ad843ba519bc95760067264d3f944a9e4b1493ba8e3eb54fc56fa\": rpc error: code = NotFound desc = could not find container \"c861c8220d8ad843ba519bc95760067264d3f944a9e4b1493ba8e3eb54fc56fa\": container with ID starting with c861c8220d8ad843ba519bc95760067264d3f944a9e4b1493ba8e3eb54fc56fa not found: ID does not exist" Oct 30 18:20:47.196773 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.196724 4442 scope.go:122] "RemoveContainer" containerID="c107c0144fd8662e10c3814030d867147b8aa0a211d6b1a9cc9ecf9081340b67" Oct 30 18:20:47.207109 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.207084 4442 scope.go:122] "RemoveContainer" containerID="c107c0144fd8662e10c3814030d867147b8aa0a211d6b1a9cc9ecf9081340b67" Oct 30 18:20:47.207456 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:20:47.207434 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c107c0144fd8662e10c3814030d867147b8aa0a211d6b1a9cc9ecf9081340b67\": container with ID starting with c107c0144fd8662e10c3814030d867147b8aa0a211d6b1a9cc9ecf9081340b67 not found: ID does not exist" containerID="c107c0144fd8662e10c3814030d867147b8aa0a211d6b1a9cc9ecf9081340b67" Oct 30 18:20:47.207536 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.207468 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c107c0144fd8662e10c3814030d867147b8aa0a211d6b1a9cc9ecf9081340b67"} err="failed to get container status \"c107c0144fd8662e10c3814030d867147b8aa0a211d6b1a9cc9ecf9081340b67\": rpc error: code = NotFound desc = could not find container \"c107c0144fd8662e10c3814030d867147b8aa0a211d6b1a9cc9ecf9081340b67\": container with ID starting with c107c0144fd8662e10c3814030d867147b8aa0a211d6b1a9cc9ecf9081340b67 not found: ID does not exist" Oct 30 18:20:47.221487 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.221466 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff"] Oct 30 18:20:47.226944 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.226918 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-h4qff"] Oct 30 18:20:47.259748 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.259713 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj"] Oct 30 18:20:47.264190 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.264092 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-vznqj"] Oct 30 18:20:47.379579 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.375203 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz"] Oct 30 18:20:47.386251 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.386180 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-nlwkz"] Oct 30 18:20:47.400296 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.400202 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp"] Oct 30 18:20:47.430734 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.430667 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-xfmwp"] Oct 30 18:20:47.525358 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.525326 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" Oct 30 18:20:47.533203 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:47.533174 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" Oct 30 18:20:47.567797 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:20:47.567741 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:20:48.008157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:48.004365 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f826dd-b9f9b8586-jjsqm" Oct 30 18:20:48.009486 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:48.009431 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-695db48787-488nc" Oct 30 18:20:48.417167 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:48.415597 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" Oct 30 18:20:48.417167 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:48.417040 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" Oct 30 18:20:48.708815 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:48.708733 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-011d6f-6747d9ddd4-bhn8d"] Oct 30 18:20:48.745379 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:48.745344 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-7d8df6544c-mlnfd"] Oct 30 18:20:49.098521 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:49.096910 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07931e20-5652-4e4c-a9d4-6cec3a6c7163" path="/var/lib/kubelet/pods/07931e20-5652-4e4c-a9d4-6cec3a6c7163/volumes" Oct 30 18:20:49.098521 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:49.097821 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b" path="/var/lib/kubelet/pods/5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b/volumes" Oct 30 18:20:49.099051 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:49.098621 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fae7458-0960-4008-b3ae-79d4518ea89e" path="/var/lib/kubelet/pods/5fae7458-0960-4008-b3ae-79d4518ea89e/volumes" Oct 30 18:20:49.101434 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:49.100153 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec2a970c-4563-416f-9357-379fbec2f852" path="/var/lib/kubelet/pods/ec2a970c-4563-416f-9357-379fbec2f852/volumes" Oct 30 18:20:51.665818 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:51.665778 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-6b67694855-jkbgw"] Oct 30 18:20:51.666586 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:51.666566 4442 cpu_manager.go:414] "containerMap: removing container" podUID="07931e20-5652-4e4c-a9d4-6cec3a6c7163" containerName="istio-proxy" Oct 30 18:20:51.666586 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:51.666588 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07931e20-5652-4e4c-a9d4-6cec3a6c7163" containerName="istio-proxy" Oct 30 18:20:51.666710 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:51.666610 4442 cpu_manager.go:414] "containerMap: removing container" podUID="5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b" containerName="istio-proxy" Oct 30 18:20:51.666710 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:51.666615 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b" containerName="istio-proxy" Oct 30 18:20:51.666710 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:51.666622 4442 cpu_manager.go:414] "containerMap: removing container" podUID="5fae7458-0960-4008-b3ae-79d4518ea89e" containerName="istio-proxy" Oct 30 18:20:51.666710 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:51.666627 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fae7458-0960-4008-b3ae-79d4518ea89e" containerName="istio-proxy" Oct 30 18:20:51.666710 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:51.666636 4442 cpu_manager.go:414] "containerMap: removing container" podUID="ec2a970c-4563-416f-9357-379fbec2f852" containerName="istio-proxy" Oct 30 18:20:51.666710 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:51.666641 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec2a970c-4563-416f-9357-379fbec2f852" containerName="istio-proxy" Oct 30 18:20:51.667011 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:51.666773 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="07931e20-5652-4e4c-a9d4-6cec3a6c7163" containerName="istio-proxy" Oct 30 18:20:51.667011 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:51.666784 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a570dcf-cd81-4a07-8fe9-f56e8e3fc86b" containerName="istio-proxy" Oct 30 18:20:51.667011 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:51.666792 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fae7458-0960-4008-b3ae-79d4518ea89e" containerName="istio-proxy" Oct 30 18:20:51.667011 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:51.666799 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec2a970c-4563-416f-9357-379fbec2f852" containerName="istio-proxy" Oct 30 18:20:51.672354 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:51.672331 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-6b67694855-jkbgw" Oct 30 18:20:51.690832 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:51.690800 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-6b67694855-jkbgw"] Oct 30 18:20:51.699476 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:51.699445 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-76135e\"" Oct 30 18:20:51.699666 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:51.699561 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-76135e-dockercfg-t5wj2\"" Oct 30 18:20:51.916211 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:51.915336 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4115d55-b178-4ab3-89d2-f2bc728b8183-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-76135e-6b67694855-jkbgw\" (UID: \"b4115d55-b178-4ab3-89d2-f2bc728b8183\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-6b67694855-jkbgw" Oct 30 18:20:51.916211 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:51.915393 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6g48\" (UniqueName: \"kubernetes.io/projected/b4115d55-b178-4ab3-89d2-f2bc728b8183-kube-api-access-b6g48\") pod \"payload-pre-processing-e2e-ipp-b-76135e-6b67694855-jkbgw\" (UID: \"b4115d55-b178-4ab3-89d2-f2bc728b8183\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-6b67694855-jkbgw" Oct 30 18:20:51.960195 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:51.959193 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-76135e-759fd4f54d-jcj6v"] Oct 30 18:20:52.031346 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:52.028418 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4115d55-b178-4ab3-89d2-f2bc728b8183-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-76135e-6b67694855-jkbgw\" (UID: \"b4115d55-b178-4ab3-89d2-f2bc728b8183\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-6b67694855-jkbgw" Oct 30 18:20:52.031346 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:52.028470 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6g48\" (UniqueName: \"kubernetes.io/projected/b4115d55-b178-4ab3-89d2-f2bc728b8183-kube-api-access-b6g48\") pod \"payload-pre-processing-e2e-ipp-b-76135e-6b67694855-jkbgw\" (UID: \"b4115d55-b178-4ab3-89d2-f2bc728b8183\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-6b67694855-jkbgw" Oct 30 18:20:52.031346 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:52.030185 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4115d55-b178-4ab3-89d2-f2bc728b8183-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-76135e-6b67694855-jkbgw\" (UID: \"b4115d55-b178-4ab3-89d2-f2bc728b8183\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-6b67694855-jkbgw" Oct 30 18:20:52.125156 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:52.124116 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6g48\" (UniqueName: \"kubernetes.io/projected/b4115d55-b178-4ab3-89d2-f2bc728b8183-kube-api-access-b6g48\") pod \"payload-pre-processing-e2e-ipp-b-76135e-6b67694855-jkbgw\" (UID: \"b4115d55-b178-4ab3-89d2-f2bc728b8183\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-6b67694855-jkbgw" Oct 30 18:20:52.138530 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:52.136162 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-76135e-759fd4f54d-jcj6v"] Oct 30 18:20:52.138804 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:52.137294 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-76135e-759fd4f54d-jcj6v" Oct 30 18:20:52.238704 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:52.238608 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vmt2\" (UniqueName: \"kubernetes.io/projected/528c3010-3850-4dad-9695-9e05bacd97b8-kube-api-access-7vmt2\") pod \"payload-processing-e2e-ipp-b-76135e-759fd4f54d-jcj6v\" (UID: \"528c3010-3850-4dad-9695-9e05bacd97b8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-76135e-759fd4f54d-jcj6v" Oct 30 18:20:52.239010 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:52.238978 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/528c3010-3850-4dad-9695-9e05bacd97b8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-76135e-759fd4f54d-jcj6v\" (UID: \"528c3010-3850-4dad-9695-9e05bacd97b8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-76135e-759fd4f54d-jcj6v" Oct 30 18:20:52.287531 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:52.287491 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-6b67694855-jkbgw" Oct 30 18:20:52.367159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:52.355152 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vmt2\" (UniqueName: \"kubernetes.io/projected/528c3010-3850-4dad-9695-9e05bacd97b8-kube-api-access-7vmt2\") pod \"payload-processing-e2e-ipp-b-76135e-759fd4f54d-jcj6v\" (UID: \"528c3010-3850-4dad-9695-9e05bacd97b8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-76135e-759fd4f54d-jcj6v" Oct 30 18:20:52.367159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:52.355431 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/528c3010-3850-4dad-9695-9e05bacd97b8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-76135e-759fd4f54d-jcj6v\" (UID: \"528c3010-3850-4dad-9695-9e05bacd97b8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-76135e-759fd4f54d-jcj6v" Oct 30 18:20:52.367159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:52.356506 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/528c3010-3850-4dad-9695-9e05bacd97b8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-76135e-759fd4f54d-jcj6v\" (UID: \"528c3010-3850-4dad-9695-9e05bacd97b8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-76135e-759fd4f54d-jcj6v" Oct 30 18:20:52.403159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:52.402907 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vmt2\" (UniqueName: \"kubernetes.io/projected/528c3010-3850-4dad-9695-9e05bacd97b8-kube-api-access-7vmt2\") pod \"payload-processing-e2e-ipp-b-76135e-759fd4f54d-jcj6v\" (UID: \"528c3010-3850-4dad-9695-9e05bacd97b8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-76135e-759fd4f54d-jcj6v" Oct 30 18:20:52.497718 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:52.477862 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-76135e-759fd4f54d-jcj6v" Oct 30 18:20:52.970185 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:52.969002 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-6b67694855-jkbgw"] Oct 30 18:20:53.040958 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:53.040933 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-76135e-759fd4f54d-jcj6v"] Oct 30 18:20:53.041608 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:20:53.041580 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod528c3010_3850_4dad_9695_9e05bacd97b8.slice/crio-f125e6a7bd63d9bb091dfab35bb5f830f30df30cd1fbcfea647be7588af08f6e WatchSource:0}: Error finding container f125e6a7bd63d9bb091dfab35bb5f830f30df30cd1fbcfea647be7588af08f6e: Status 404 returned error can't find the container with id f125e6a7bd63d9bb091dfab35bb5f830f30df30cd1fbcfea647be7588af08f6e Oct 30 18:20:53.705436 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:53.705398 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-76135e-759fd4f54d-jcj6v" event={"ID":"528c3010-3850-4dad-9695-9e05bacd97b8","Type":"ContainerStarted","Data":"93168e7e5566feb6d2fce5a6ae1bbdc31e8a9c9faab8f0564bf4900a96c29919"} Oct 30 18:20:53.705653 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:53.705454 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-76135e-759fd4f54d-jcj6v" event={"ID":"528c3010-3850-4dad-9695-9e05bacd97b8","Type":"ContainerStarted","Data":"f125e6a7bd63d9bb091dfab35bb5f830f30df30cd1fbcfea647be7588af08f6e"} Oct 30 18:20:53.705653 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:53.705493 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-76135e-759fd4f54d-jcj6v" Oct 30 18:20:53.707213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:53.707182 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-6b67694855-jkbgw" event={"ID":"b4115d55-b178-4ab3-89d2-f2bc728b8183","Type":"ContainerStarted","Data":"247f87d4a309f2b2ff251f1edaf22bfe795b1eaa5bf96653b8b2748f460a55cc"} Oct 30 18:20:53.707323 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:53.707253 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-6b67694855-jkbgw" event={"ID":"b4115d55-b178-4ab3-89d2-f2bc728b8183","Type":"ContainerStarted","Data":"94c1299e8be12ba613adc22dd0551aa636123cc5885a8fb7f675bfabeff228a4"} Oct 30 18:20:53.707323 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:53.707275 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-6b67694855-jkbgw" Oct 30 18:20:53.709570 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:53.709538 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-011d6f-6747d9ddd4-bhn8d" podUID="e3c3d410-8315-4ab0-a021-9e51cd8cf579" containerName="payload-processing" containerID="cri-o://bf241951835b6885f961341296a8be3ada1a92e8940a3bd7fb566597434a0044" gracePeriod=25 Oct 30 18:20:53.746554 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:53.746518 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-7d8df6544c-mlnfd" podUID="efec0626-054b-45be-a516-f0f8b4329b52" containerName="payload-pre-processing" containerID="cri-o://75a87049b35ed90e6b4f9816ff095725eb0e94da27e1eb67ec98d3341aeb5de4" gracePeriod=25 Oct 30 18:20:53.753079 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:53.753034 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-76135e-759fd4f54d-jcj6v" podStartSLOduration=2.753018279 podStartE2EDuration="2.753018279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:53.749188325 +0000 UTC m=+1350.839250665" watchObservedRunningTime="2026-10-30 18:20:53.753018279 +0000 UTC m=+1350.843080617" Oct 30 18:20:53.793324 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:53.793251 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-6b67694855-jkbgw" podStartSLOduration=2.793232466 podStartE2EDuration="2.793232466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:53.788241883 +0000 UTC m=+1350.878304246" watchObservedRunningTime="2026-10-30 18:20:53.793232466 +0000 UTC m=+1350.883294807" Oct 30 18:20:54.186240 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.186212 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-011d6f-6747d9ddd4-bhn8d" Oct 30 18:20:54.267213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.267164 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-7d8df6544c-mlnfd" Oct 30 18:20:54.269324 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.269300 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3c3d410-8315-4ab0-a021-9e51cd8cf579-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3c3d410-8315-4ab0-a021-9e51cd8cf579-plugins-config-volume\") pod \"e3c3d410-8315-4ab0-a021-9e51cd8cf579\" (UID: \"e3c3d410-8315-4ab0-a021-9e51cd8cf579\") " Oct 30 18:20:54.269459 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.269376 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3c3d410-8315-4ab0-a021-9e51cd8cf579-kube-api-access-4fbqt\" (UniqueName: \"kubernetes.io/projected/e3c3d410-8315-4ab0-a021-9e51cd8cf579-kube-api-access-4fbqt\") pod \"e3c3d410-8315-4ab0-a021-9e51cd8cf579\" (UID: \"e3c3d410-8315-4ab0-a021-9e51cd8cf579\") " Oct 30 18:20:54.269764 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.269731 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3c3d410-8315-4ab0-a021-9e51cd8cf579-plugins-config-volume" pod "e3c3d410-8315-4ab0-a021-9e51cd8cf579" (UID: "e3c3d410-8315-4ab0-a021-9e51cd8cf579"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:20:54.270059 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.270040 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3c3d410-8315-4ab0-a021-9e51cd8cf579-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:54.280171 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.272223 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3c3d410-8315-4ab0-a021-9e51cd8cf579-kube-api-access-4fbqt" pod "e3c3d410-8315-4ab0-a021-9e51cd8cf579" (UID: "e3c3d410-8315-4ab0-a021-9e51cd8cf579"). InnerVolumeSpecName "kube-api-access-4fbqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:20:54.371210 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.371157 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efec0626-054b-45be-a516-f0f8b4329b52-kube-api-access-jbmsv\" (UniqueName: \"kubernetes.io/projected/efec0626-054b-45be-a516-f0f8b4329b52-kube-api-access-jbmsv\") pod \"efec0626-054b-45be-a516-f0f8b4329b52\" (UID: \"efec0626-054b-45be-a516-f0f8b4329b52\") " Oct 30 18:20:54.371385 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.371278 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efec0626-054b-45be-a516-f0f8b4329b52-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efec0626-054b-45be-a516-f0f8b4329b52-plugins-config-volume\") pod \"efec0626-054b-45be-a516-f0f8b4329b52\" (UID: \"efec0626-054b-45be-a516-f0f8b4329b52\") " Oct 30 18:20:54.371676 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.371650 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efec0626-054b-45be-a516-f0f8b4329b52-plugins-config-volume" pod "efec0626-054b-45be-a516-f0f8b4329b52" (UID: "efec0626-054b-45be-a516-f0f8b4329b52"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:20:54.372079 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.372052 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fbqt\" (UniqueName: \"kubernetes.io/projected/e3c3d410-8315-4ab0-a021-9e51cd8cf579-kube-api-access-4fbqt\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:54.372216 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.372088 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efec0626-054b-45be-a516-f0f8b4329b52-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:54.374382 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.374348 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efec0626-054b-45be-a516-f0f8b4329b52-kube-api-access-jbmsv" pod "efec0626-054b-45be-a516-f0f8b4329b52" (UID: "efec0626-054b-45be-a516-f0f8b4329b52"). InnerVolumeSpecName "kube-api-access-jbmsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:20:54.473294 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.473223 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbmsv\" (UniqueName: \"kubernetes.io/projected/efec0626-054b-45be-a516-f0f8b4329b52-kube-api-access-jbmsv\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:20:54.526979 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.526941 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f826dd-7cf496bf8b-qwhl9"] Oct 30 18:20:54.527735 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.527713 4442 cpu_manager.go:414] "containerMap: removing container" podUID="efec0626-054b-45be-a516-f0f8b4329b52" containerName="payload-pre-processing" Oct 30 18:20:54.527735 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.527734 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efec0626-054b-45be-a516-f0f8b4329b52" containerName="payload-pre-processing" Oct 30 18:20:54.527893 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.527771 4442 cpu_manager.go:414] "containerMap: removing container" podUID="e3c3d410-8315-4ab0-a021-9e51cd8cf579" containerName="payload-processing" Oct 30 18:20:54.527893 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.527776 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3c3d410-8315-4ab0-a021-9e51cd8cf579" containerName="payload-processing" Oct 30 18:20:54.527893 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.527871 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3c3d410-8315-4ab0-a021-9e51cd8cf579" containerName="payload-processing" Oct 30 18:20:54.527893 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.527883 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="efec0626-054b-45be-a516-f0f8b4329b52" containerName="payload-pre-processing" Oct 30 18:20:54.538714 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.536636 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f826dd-7cf496bf8b-qwhl9" Oct 30 18:20:54.553198 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.553168 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f826dd-7cf496bf8b-qwhl9"] Oct 30 18:20:54.554360 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.554337 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-f826dd-serving-cert\"" Oct 30 18:20:54.675659 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.675624 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34802091-fbcd-4b48-90da-96ee88ec3aae-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f826dd-7cf496bf8b-qwhl9\" (UID: \"34802091-fbcd-4b48-90da-96ee88ec3aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f826dd-7cf496bf8b-qwhl9" Oct 30 18:20:54.675840 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.675740 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34802091-fbcd-4b48-90da-96ee88ec3aae-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f826dd-7cf496bf8b-qwhl9\" (UID: \"34802091-fbcd-4b48-90da-96ee88ec3aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f826dd-7cf496bf8b-qwhl9" Oct 30 18:20:54.675939 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.675916 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxjsj\" (UniqueName: \"kubernetes.io/projected/34802091-fbcd-4b48-90da-96ee88ec3aae-kube-api-access-nxjsj\") pod \"maas-api-e2e-ipp-b-f826dd-7cf496bf8b-qwhl9\" (UID: \"34802091-fbcd-4b48-90da-96ee88ec3aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f826dd-7cf496bf8b-qwhl9" Oct 30 18:20:54.676048 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.676032 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34802091-fbcd-4b48-90da-96ee88ec3aae-metrics-certs\") pod \"maas-api-e2e-ipp-b-f826dd-7cf496bf8b-qwhl9\" (UID: \"34802091-fbcd-4b48-90da-96ee88ec3aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f826dd-7cf496bf8b-qwhl9" Oct 30 18:20:54.731476 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.723553 4442 generic.go:356] "Generic (PLEG): container finished" podID="e3c3d410-8315-4ab0-a021-9e51cd8cf579" containerID="bf241951835b6885f961341296a8be3ada1a92e8940a3bd7fb566597434a0044" exitCode=0 Oct 30 18:20:54.731476 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.723743 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-011d6f-6747d9ddd4-bhn8d" event={"ID":"e3c3d410-8315-4ab0-a021-9e51cd8cf579","Type":"ContainerDied","Data":"bf241951835b6885f961341296a8be3ada1a92e8940a3bd7fb566597434a0044"} Oct 30 18:20:54.731476 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.723798 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-011d6f-6747d9ddd4-bhn8d" event={"ID":"e3c3d410-8315-4ab0-a021-9e51cd8cf579","Type":"ContainerDied","Data":"92d510dfd71bc157270316b9aef9a33afd4fdaca83e9c35335e694b93efa6377"} Oct 30 18:20:54.731476 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.723832 4442 scope.go:122] "RemoveContainer" containerID="bf241951835b6885f961341296a8be3ada1a92e8940a3bd7fb566597434a0044" Oct 30 18:20:54.731476 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.724006 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-011d6f-6747d9ddd4-bhn8d" Oct 30 18:20:54.804817 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.788255 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34802091-fbcd-4b48-90da-96ee88ec3aae-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f826dd-7cf496bf8b-qwhl9\" (UID: \"34802091-fbcd-4b48-90da-96ee88ec3aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f826dd-7cf496bf8b-qwhl9" Oct 30 18:20:54.804817 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.788425 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxjsj\" (UniqueName: \"kubernetes.io/projected/34802091-fbcd-4b48-90da-96ee88ec3aae-kube-api-access-nxjsj\") pod \"maas-api-e2e-ipp-b-f826dd-7cf496bf8b-qwhl9\" (UID: \"34802091-fbcd-4b48-90da-96ee88ec3aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f826dd-7cf496bf8b-qwhl9" Oct 30 18:20:54.804817 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.788515 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34802091-fbcd-4b48-90da-96ee88ec3aae-metrics-certs\") pod \"maas-api-e2e-ipp-b-f826dd-7cf496bf8b-qwhl9\" (UID: \"34802091-fbcd-4b48-90da-96ee88ec3aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f826dd-7cf496bf8b-qwhl9" Oct 30 18:20:54.804817 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.788656 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34802091-fbcd-4b48-90da-96ee88ec3aae-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f826dd-7cf496bf8b-qwhl9\" (UID: \"34802091-fbcd-4b48-90da-96ee88ec3aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f826dd-7cf496bf8b-qwhl9" Oct 30 18:20:54.804817 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.789657 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34802091-fbcd-4b48-90da-96ee88ec3aae-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f826dd-7cf496bf8b-qwhl9\" (UID: \"34802091-fbcd-4b48-90da-96ee88ec3aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f826dd-7cf496bf8b-qwhl9" Oct 30 18:20:54.804817 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:20:54.789780 4442 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-f826dd-serving-cert: secret "maas-api-e2e-ipp-b-f826dd-serving-cert" not found Oct 30 18:20:54.804817 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:20:54.789842 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34802091-fbcd-4b48-90da-96ee88ec3aae-maas-api-tls podName:34802091-fbcd-4b48-90da-96ee88ec3aae nodeName:}" failed. No retries permitted until 2026-10-30 18:20:55.289821046 +0000 UTC m=+1352.379883363 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/34802091-fbcd-4b48-90da-96ee88ec3aae-maas-api-tls") pod "maas-api-e2e-ipp-b-f826dd-7cf496bf8b-qwhl9" (UID: "34802091-fbcd-4b48-90da-96ee88ec3aae") : secret "maas-api-e2e-ipp-b-f826dd-serving-cert" not found Oct 30 18:20:54.831371 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.830937 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34802091-fbcd-4b48-90da-96ee88ec3aae-metrics-certs\") pod \"maas-api-e2e-ipp-b-f826dd-7cf496bf8b-qwhl9\" (UID: \"34802091-fbcd-4b48-90da-96ee88ec3aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f826dd-7cf496bf8b-qwhl9" Oct 30 18:20:54.842410 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.842333 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxjsj\" (UniqueName: \"kubernetes.io/projected/34802091-fbcd-4b48-90da-96ee88ec3aae-kube-api-access-nxjsj\") pod \"maas-api-e2e-ipp-b-f826dd-7cf496bf8b-qwhl9\" (UID: \"34802091-fbcd-4b48-90da-96ee88ec3aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f826dd-7cf496bf8b-qwhl9" Oct 30 18:20:54.857668 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.849667 4442 generic.go:356] "Generic (PLEG): container finished" podID="efec0626-054b-45be-a516-f0f8b4329b52" containerID="75a87049b35ed90e6b4f9816ff095725eb0e94da27e1eb67ec98d3341aeb5de4" exitCode=0 Oct 30 18:20:54.857668 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.852301 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-7d8df6544c-mlnfd" Oct 30 18:20:54.866455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.865115 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-7d8df6544c-mlnfd" event={"ID":"efec0626-054b-45be-a516-f0f8b4329b52","Type":"ContainerDied","Data":"75a87049b35ed90e6b4f9816ff095725eb0e94da27e1eb67ec98d3341aeb5de4"} Oct 30 18:20:54.866455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.865203 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-7d8df6544c-mlnfd" event={"ID":"efec0626-054b-45be-a516-f0f8b4329b52","Type":"ContainerDied","Data":"9a028005bf07ce84d76edf3fcea27eb710687e3e89d89329fee1090f72a6ea75"} Oct 30 18:20:54.911987 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.911687 4442 scope.go:122] "RemoveContainer" containerID="bf241951835b6885f961341296a8be3ada1a92e8940a3bd7fb566597434a0044" Oct 30 18:20:54.912122 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:20:54.912040 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf241951835b6885f961341296a8be3ada1a92e8940a3bd7fb566597434a0044\": container with ID starting with bf241951835b6885f961341296a8be3ada1a92e8940a3bd7fb566597434a0044 not found: ID does not exist" containerID="bf241951835b6885f961341296a8be3ada1a92e8940a3bd7fb566597434a0044" Oct 30 18:20:54.912122 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.912077 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf241951835b6885f961341296a8be3ada1a92e8940a3bd7fb566597434a0044"} err="failed to get container status \"bf241951835b6885f961341296a8be3ada1a92e8940a3bd7fb566597434a0044\": rpc error: code = NotFound desc = could not find container \"bf241951835b6885f961341296a8be3ada1a92e8940a3bd7fb566597434a0044\": container with ID starting with bf241951835b6885f961341296a8be3ada1a92e8940a3bd7fb566597434a0044 not found: ID does not exist" Oct 30 18:20:54.912122 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.912101 4442 scope.go:122] "RemoveContainer" containerID="75a87049b35ed90e6b4f9816ff095725eb0e94da27e1eb67ec98d3341aeb5de4" Oct 30 18:20:54.916267 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.916243 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-011d6f-6747d9ddd4-bhn8d"] Oct 30 18:20:54.920501 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.920479 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-011d6f-6747d9ddd4-bhn8d"] Oct 30 18:20:54.923328 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.923313 4442 scope.go:122] "RemoveContainer" containerID="75a87049b35ed90e6b4f9816ff095725eb0e94da27e1eb67ec98d3341aeb5de4" Oct 30 18:20:54.923615 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:20:54.923595 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75a87049b35ed90e6b4f9816ff095725eb0e94da27e1eb67ec98d3341aeb5de4\": container with ID starting with 75a87049b35ed90e6b4f9816ff095725eb0e94da27e1eb67ec98d3341aeb5de4 not found: ID does not exist" containerID="75a87049b35ed90e6b4f9816ff095725eb0e94da27e1eb67ec98d3341aeb5de4" Oct 30 18:20:54.923668 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.923623 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75a87049b35ed90e6b4f9816ff095725eb0e94da27e1eb67ec98d3341aeb5de4"} err="failed to get container status \"75a87049b35ed90e6b4f9816ff095725eb0e94da27e1eb67ec98d3341aeb5de4\": rpc error: code = NotFound desc = could not find container \"75a87049b35ed90e6b4f9816ff095725eb0e94da27e1eb67ec98d3341aeb5de4\": container with ID starting with 75a87049b35ed90e6b4f9816ff095725eb0e94da27e1eb67ec98d3341aeb5de4 not found: ID does not exist" Oct 30 18:20:54.966857 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.966828 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-7d8df6544c-mlnfd"] Oct 30 18:20:54.970882 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:54.970857 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-7d8df6544c-mlnfd"] Oct 30 18:20:55.066564 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:55.066485 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3c3d410-8315-4ab0-a021-9e51cd8cf579" path="/var/lib/kubelet/pods/e3c3d410-8315-4ab0-a021-9e51cd8cf579/volumes" Oct 30 18:20:55.066977 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:55.066963 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efec0626-054b-45be-a516-f0f8b4329b52" path="/var/lib/kubelet/pods/efec0626-054b-45be-a516-f0f8b4329b52/volumes" Oct 30 18:20:55.314933 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:55.314895 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34802091-fbcd-4b48-90da-96ee88ec3aae-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f826dd-7cf496bf8b-qwhl9\" (UID: \"34802091-fbcd-4b48-90da-96ee88ec3aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f826dd-7cf496bf8b-qwhl9" Oct 30 18:20:55.317527 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:55.317462 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34802091-fbcd-4b48-90da-96ee88ec3aae-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f826dd-7cf496bf8b-qwhl9\" (UID: \"34802091-fbcd-4b48-90da-96ee88ec3aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f826dd-7cf496bf8b-qwhl9" Oct 30 18:20:55.458752 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:55.458711 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f826dd-7cf496bf8b-qwhl9" Oct 30 18:20:55.861293 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:55.861259 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f826dd-7cf496bf8b-qwhl9"] Oct 30 18:20:55.895462 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:55.895432 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f826dd-7cf496bf8b-qwhl9" event={"ID":"34802091-fbcd-4b48-90da-96ee88ec3aae","Type":"ContainerStarted","Data":"c505402738ab30ec52106e27c67fb27b14d8a870b85a46e051391af93d868b09"} Oct 30 18:20:56.904787 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:56.904747 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f826dd-7cf496bf8b-qwhl9" event={"ID":"34802091-fbcd-4b48-90da-96ee88ec3aae","Type":"ContainerStarted","Data":"68239458ce1d0966b6543a4fbe1f9ad8fa7c89e8514b52b4fb81fd6865d0f43e"} Oct 30 18:20:56.905239 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:56.904817 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f826dd-7cf496bf8b-qwhl9" Oct 30 18:20:56.947838 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:20:56.947788 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f826dd-7cf496bf8b-qwhl9" podStartSLOduration=2.64123077 podStartE2EDuration="2.947773853s" totalImagesPullingTime="306.543083ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:56.942805285 +0000 UTC m=+1354.032867623" watchObservedRunningTime="2026-10-30 18:20:56.947773853 +0000 UTC m=+1354.037836192" Oct 30 18:21:02.918469 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:02.918444 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f826dd-7cf496bf8b-qwhl9" Oct 30 18:21:04.856463 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:04.856434 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-76135e-759fd4f54d-jcj6v" Oct 30 18:21:04.869109 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:04.869079 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-6b67694855-jkbgw" Oct 30 18:21:14.132657 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:14.132619 4442 generic.go:356] "Generic (PLEG): container finished" podID="024b2852-2e03-4f0b-a7dd-858dc9ed69ed" containerID="cd4569f68051b24db267d3934ac425d3ece913875a2e722cb1082a77b35ce067" exitCode=1 Oct 30 18:21:14.133309 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:14.133284 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67cc46c64-66wvf" event={"ID":"024b2852-2e03-4f0b-a7dd-858dc9ed69ed","Type":"ContainerDied","Data":"cd4569f68051b24db267d3934ac425d3ece913875a2e722cb1082a77b35ce067"} Oct 30 18:21:14.140582 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:14.133932 4442 scope.go:122] "RemoveContainer" containerID="cd4569f68051b24db267d3934ac425d3ece913875a2e722cb1082a77b35ce067" Oct 30 18:21:14.710448 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:14.710414 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 18:21:15.145478 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:15.145443 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67cc46c64-66wvf" event={"ID":"024b2852-2e03-4f0b-a7dd-858dc9ed69ed","Type":"ContainerStarted","Data":"f36234a29770f511950e8a68e532808bb037543b118ed672f2456c012a209610"} Oct 30 18:21:15.145930 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:15.145681 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-67cc46c64-66wvf" Oct 30 18:21:17.566340 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:21:17.566303 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:21:24.649654 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:24.649623 4442 scope.go:122] "RemoveContainer" containerID="3aeeb83180209441d0fcb39b72a72223ee72701904312e66ec34cc0c84c24d6b" Oct 30 18:21:26.157471 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:26.157442 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-67cc46c64-66wvf" Oct 30 18:21:35.040765 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:35.040736 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2ee874-5b975bd4-drq62"] Oct 30 18:21:35.087996 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:35.087964 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t"] Oct 30 18:21:35.088342 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:35.088310 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" podUID="6db69e2c-9e56-4b22-b8e4-3d346b92c062" containerName="istio-proxy" containerID="cri-o://33a4c4b4c4c56e01791bc62d5a1538254f534d969eec70d702ab02df5b99ab52" gracePeriod=30 Oct 30 18:21:35.095226 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:35.095201 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2ee874-69fddfb67-hscz8"] Oct 30 18:21:40.040161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.040111 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789"] Oct 30 18:21:40.040793 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.040344 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" podUID="b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42" containerName="istio-proxy" containerID="cri-o://84431ef14fb17489f30111c3a32731796835c1e6d3fd6cdb900e025731d19026" gracePeriod=30 Oct 30 18:21:40.041223 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.041085 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-2ee874-5b975bd4-drq62" podUID="6f233a98-d41a-4a25-9901-418e48cebadd" containerName="payload-processing" containerID="cri-o://965942e71f9d587a54dcb01a2e5d7b9b841bc766d8534a317f6a99d27da6926e" gracePeriod=25 Oct 30 18:21:40.055890 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.055859 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m"] Oct 30 18:21:40.056206 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.056179 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" podUID="d9b6ff41-9b4a-43e1-8a09-b99b2014530c" containerName="istio-proxy" containerID="cri-o://b04013d4b8552b82935627b34ab847c949a27d0038b27a31e64e4676e7c67653" gracePeriod=30 Oct 30 18:21:40.061089 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.061060 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx"] Oct 30 18:21:40.061517 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.061266 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" podUID="f6a0e550-b203-433c-8991-387f2179e0de" containerName="istio-proxy" containerID="cri-o://4bed610050ec6cda902438aae0dcc950d53cc9ebee5302bf95a529cc8f48426f" gracePeriod=30 Oct 30 18:21:40.097318 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.095846 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ee874-69fddfb67-hscz8" podUID="e295ee2a-45e5-485d-9418-b046b37095d2" containerName="payload-pre-processing" containerID="cri-o://ac06a0977b9a226ebefb88de50e0e34679f723b72fd4f867967c4e8eb8266e6a" gracePeriod=25 Oct 30 18:21:40.417684 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.415883 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2ee874-5b975bd4-drq62" Oct 30 18:21:40.449205 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.448109 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f233a98-d41a-4a25-9901-418e48cebadd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f233a98-d41a-4a25-9901-418e48cebadd-plugins-config-volume\") pod \"6f233a98-d41a-4a25-9901-418e48cebadd\" (UID: \"6f233a98-d41a-4a25-9901-418e48cebadd\") " Oct 30 18:21:40.449205 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.448334 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f233a98-d41a-4a25-9901-418e48cebadd-kube-api-access-mln4d\" (UniqueName: \"kubernetes.io/projected/6f233a98-d41a-4a25-9901-418e48cebadd-kube-api-access-mln4d\") pod \"6f233a98-d41a-4a25-9901-418e48cebadd\" (UID: \"6f233a98-d41a-4a25-9901-418e48cebadd\") " Oct 30 18:21:40.449205 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.448751 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f233a98-d41a-4a25-9901-418e48cebadd-plugins-config-volume" pod "6f233a98-d41a-4a25-9901-418e48cebadd" (UID: "6f233a98-d41a-4a25-9901-418e48cebadd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:21:40.449205 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.449154 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f233a98-d41a-4a25-9901-418e48cebadd-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:21:40.454350 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.454317 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f233a98-d41a-4a25-9901-418e48cebadd-kube-api-access-mln4d" pod "6f233a98-d41a-4a25-9901-418e48cebadd" (UID: "6f233a98-d41a-4a25-9901-418e48cebadd"). InnerVolumeSpecName "kube-api-access-mln4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:40.467685 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.467308 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" Oct 30 18:21:40.469220 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.469071 4442 generic.go:356] "Generic (PLEG): container finished" podID="6db69e2c-9e56-4b22-b8e4-3d346b92c062" containerID="33a4c4b4c4c56e01791bc62d5a1538254f534d969eec70d702ab02df5b99ab52" exitCode=0 Oct 30 18:21:40.470963 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.469489 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" event={"ID":"6db69e2c-9e56-4b22-b8e4-3d346b92c062","Type":"ContainerDied","Data":"33a4c4b4c4c56e01791bc62d5a1538254f534d969eec70d702ab02df5b99ab52"} Oct 30 18:21:40.470963 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.469550 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" event={"ID":"6db69e2c-9e56-4b22-b8e4-3d346b92c062","Type":"ContainerDied","Data":"2a6d224e7181600427ae783ea6df3fe7b119e8ed4ee6f83d52ef56179b3ec418"} Oct 30 18:21:40.470963 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.469581 4442 scope.go:122] "RemoveContainer" containerID="33a4c4b4c4c56e01791bc62d5a1538254f534d969eec70d702ab02df5b99ab52" Oct 30 18:21:40.477341 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.473386 4442 generic.go:356] "Generic (PLEG): container finished" podID="6f233a98-d41a-4a25-9901-418e48cebadd" containerID="965942e71f9d587a54dcb01a2e5d7b9b841bc766d8534a317f6a99d27da6926e" exitCode=0 Oct 30 18:21:40.477341 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.473474 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2ee874-5b975bd4-drq62" event={"ID":"6f233a98-d41a-4a25-9901-418e48cebadd","Type":"ContainerDied","Data":"965942e71f9d587a54dcb01a2e5d7b9b841bc766d8534a317f6a99d27da6926e"} Oct 30 18:21:40.477341 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.473519 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2ee874-5b975bd4-drq62" event={"ID":"6f233a98-d41a-4a25-9901-418e48cebadd","Type":"ContainerDied","Data":"7def7b0d39048a1ef21e513cda8bd58a86a2133d03faafd92ebb1ae0d77beee3"} Oct 30 18:21:40.477341 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.473601 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2ee874-5b975bd4-drq62" Oct 30 18:21:40.477341 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.476853 4442 generic.go:356] "Generic (PLEG): container finished" podID="e295ee2a-45e5-485d-9418-b046b37095d2" containerID="ac06a0977b9a226ebefb88de50e0e34679f723b72fd4f867967c4e8eb8266e6a" exitCode=0 Oct 30 18:21:40.477341 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.476966 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ee874-69fddfb67-hscz8" event={"ID":"e295ee2a-45e5-485d-9418-b046b37095d2","Type":"ContainerDied","Data":"ac06a0977b9a226ebefb88de50e0e34679f723b72fd4f867967c4e8eb8266e6a"} Oct 30 18:21:40.483320 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.483286 4442 scope.go:122] "RemoveContainer" containerID="33a4c4b4c4c56e01791bc62d5a1538254f534d969eec70d702ab02df5b99ab52" Oct 30 18:21:40.483634 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:21:40.483609 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33a4c4b4c4c56e01791bc62d5a1538254f534d969eec70d702ab02df5b99ab52\": container with ID starting with 33a4c4b4c4c56e01791bc62d5a1538254f534d969eec70d702ab02df5b99ab52 not found: ID does not exist" containerID="33a4c4b4c4c56e01791bc62d5a1538254f534d969eec70d702ab02df5b99ab52" Oct 30 18:21:40.483729 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.483645 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33a4c4b4c4c56e01791bc62d5a1538254f534d969eec70d702ab02df5b99ab52"} err="failed to get container status \"33a4c4b4c4c56e01791bc62d5a1538254f534d969eec70d702ab02df5b99ab52\": rpc error: code = NotFound desc = could not find container \"33a4c4b4c4c56e01791bc62d5a1538254f534d969eec70d702ab02df5b99ab52\": container with ID starting with 33a4c4b4c4c56e01791bc62d5a1538254f534d969eec70d702ab02df5b99ab52 not found: ID does not exist" Oct 30 18:21:40.483729 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.483669 4442 scope.go:122] "RemoveContainer" containerID="965942e71f9d587a54dcb01a2e5d7b9b841bc766d8534a317f6a99d27da6926e" Oct 30 18:21:40.499769 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.499746 4442 scope.go:122] "RemoveContainer" containerID="965942e71f9d587a54dcb01a2e5d7b9b841bc766d8534a317f6a99d27da6926e" Oct 30 18:21:40.500230 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:21:40.500199 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"965942e71f9d587a54dcb01a2e5d7b9b841bc766d8534a317f6a99d27da6926e\": container with ID starting with 965942e71f9d587a54dcb01a2e5d7b9b841bc766d8534a317f6a99d27da6926e not found: ID does not exist" containerID="965942e71f9d587a54dcb01a2e5d7b9b841bc766d8534a317f6a99d27da6926e" Oct 30 18:21:40.500333 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.500241 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"965942e71f9d587a54dcb01a2e5d7b9b841bc766d8534a317f6a99d27da6926e"} err="failed to get container status \"965942e71f9d587a54dcb01a2e5d7b9b841bc766d8534a317f6a99d27da6926e\": rpc error: code = NotFound desc = could not find container \"965942e71f9d587a54dcb01a2e5d7b9b841bc766d8534a317f6a99d27da6926e\": container with ID starting with 965942e71f9d587a54dcb01a2e5d7b9b841bc766d8534a317f6a99d27da6926e not found: ID does not exist" Oct 30 18:21:40.536412 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.536374 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ee874-69fddfb67-hscz8" Oct 30 18:21:40.550618 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.550511 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6db69e2c-9e56-4b22-b8e4-3d346b92c062-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db69e2c-9e56-4b22-b8e4-3d346b92c062-istiod-ca-cert\") pod \"6db69e2c-9e56-4b22-b8e4-3d346b92c062\" (UID: \"6db69e2c-9e56-4b22-b8e4-3d346b92c062\") " Oct 30 18:21:40.550795 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.550630 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6db69e2c-9e56-4b22-b8e4-3d346b92c062-kube-api-access-44gjh\" (UniqueName: \"kubernetes.io/projected/6db69e2c-9e56-4b22-b8e4-3d346b92c062-kube-api-access-44gjh\") pod \"6db69e2c-9e56-4b22-b8e4-3d346b92c062\" (UID: \"6db69e2c-9e56-4b22-b8e4-3d346b92c062\") " Oct 30 18:21:40.550795 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.550669 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db69e2c-9e56-4b22-b8e4-3d346b92c062-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db69e2c-9e56-4b22-b8e4-3d346b92c062-credential-socket\") pod \"6db69e2c-9e56-4b22-b8e4-3d346b92c062\" (UID: \"6db69e2c-9e56-4b22-b8e4-3d346b92c062\") " Oct 30 18:21:40.550795 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.550761 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db69e2c-9e56-4b22-b8e4-3d346b92c062-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db69e2c-9e56-4b22-b8e4-3d346b92c062-workload-certs\") pod \"6db69e2c-9e56-4b22-b8e4-3d346b92c062\" (UID: \"6db69e2c-9e56-4b22-b8e4-3d346b92c062\") " Oct 30 18:21:40.550982 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.550811 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db69e2c-9e56-4b22-b8e4-3d346b92c062-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db69e2c-9e56-4b22-b8e4-3d346b92c062-istio-data\") pod \"6db69e2c-9e56-4b22-b8e4-3d346b92c062\" (UID: \"6db69e2c-9e56-4b22-b8e4-3d346b92c062\") " Oct 30 18:21:40.550982 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.550845 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6db69e2c-9e56-4b22-b8e4-3d346b92c062-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db69e2c-9e56-4b22-b8e4-3d346b92c062-istio-podinfo\") pod \"6db69e2c-9e56-4b22-b8e4-3d346b92c062\" (UID: \"6db69e2c-9e56-4b22-b8e4-3d346b92c062\") " Oct 30 18:21:40.550982 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.550912 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db69e2c-9e56-4b22-b8e4-3d346b92c062-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db69e2c-9e56-4b22-b8e4-3d346b92c062-istio-envoy\") pod \"6db69e2c-9e56-4b22-b8e4-3d346b92c062\" (UID: \"6db69e2c-9e56-4b22-b8e4-3d346b92c062\") " Oct 30 18:21:40.551223 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.551200 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db69e2c-9e56-4b22-b8e4-3d346b92c062-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db69e2c-9e56-4b22-b8e4-3d346b92c062-workload-socket\") pod \"6db69e2c-9e56-4b22-b8e4-3d346b92c062\" (UID: \"6db69e2c-9e56-4b22-b8e4-3d346b92c062\") " Oct 30 18:21:40.551352 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.551335 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6db69e2c-9e56-4b22-b8e4-3d346b92c062-istio-token\" (UniqueName: \"kubernetes.io/projected/6db69e2c-9e56-4b22-b8e4-3d346b92c062-istio-token\") pod \"6db69e2c-9e56-4b22-b8e4-3d346b92c062\" (UID: \"6db69e2c-9e56-4b22-b8e4-3d346b92c062\") " Oct 30 18:21:40.552178 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.552128 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mln4d\" (UniqueName: \"kubernetes.io/projected/6f233a98-d41a-4a25-9901-418e48cebadd-kube-api-access-mln4d\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:21:40.552856 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.552829 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db69e2c-9e56-4b22-b8e4-3d346b92c062-workload-certs" pod "6db69e2c-9e56-4b22-b8e4-3d346b92c062" (UID: "6db69e2c-9e56-4b22-b8e4-3d346b92c062"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:40.553233 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.553208 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6db69e2c-9e56-4b22-b8e4-3d346b92c062-istiod-ca-cert" pod "6db69e2c-9e56-4b22-b8e4-3d346b92c062" (UID: "6db69e2c-9e56-4b22-b8e4-3d346b92c062"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:21:40.554533 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.554495 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db69e2c-9e56-4b22-b8e4-3d346b92c062-workload-socket" pod "6db69e2c-9e56-4b22-b8e4-3d346b92c062" (UID: "6db69e2c-9e56-4b22-b8e4-3d346b92c062"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:40.555013 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.554570 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db69e2c-9e56-4b22-b8e4-3d346b92c062-istio-data" pod "6db69e2c-9e56-4b22-b8e4-3d346b92c062" (UID: "6db69e2c-9e56-4b22-b8e4-3d346b92c062"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:40.555013 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.554897 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db69e2c-9e56-4b22-b8e4-3d346b92c062-credential-socket" pod "6db69e2c-9e56-4b22-b8e4-3d346b92c062" (UID: "6db69e2c-9e56-4b22-b8e4-3d346b92c062"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:40.556019 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.555973 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db69e2c-9e56-4b22-b8e4-3d346b92c062-istio-token" pod "6db69e2c-9e56-4b22-b8e4-3d346b92c062" (UID: "6db69e2c-9e56-4b22-b8e4-3d346b92c062"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:40.556598 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.556571 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db69e2c-9e56-4b22-b8e4-3d346b92c062-kube-api-access-44gjh" pod "6db69e2c-9e56-4b22-b8e4-3d346b92c062" (UID: "6db69e2c-9e56-4b22-b8e4-3d346b92c062"). InnerVolumeSpecName "kube-api-access-44gjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:40.558339 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.558314 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6db69e2c-9e56-4b22-b8e4-3d346b92c062-istio-podinfo" pod "6db69e2c-9e56-4b22-b8e4-3d346b92c062" (UID: "6db69e2c-9e56-4b22-b8e4-3d346b92c062"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:21:40.558494 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.558473 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db69e2c-9e56-4b22-b8e4-3d346b92c062-istio-envoy" pod "6db69e2c-9e56-4b22-b8e4-3d346b92c062" (UID: "6db69e2c-9e56-4b22-b8e4-3d346b92c062"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:40.580283 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.580236 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2ee874-5b975bd4-drq62"] Oct 30 18:21:40.586902 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.586873 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2ee874-5b975bd4-drq62"] Oct 30 18:21:40.653781 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.653750 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e295ee2a-45e5-485d-9418-b046b37095d2-kube-api-access-mm282\" (UniqueName: \"kubernetes.io/projected/e295ee2a-45e5-485d-9418-b046b37095d2-kube-api-access-mm282\") pod \"e295ee2a-45e5-485d-9418-b046b37095d2\" (UID: \"e295ee2a-45e5-485d-9418-b046b37095d2\") " Oct 30 18:21:40.653969 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.653926 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e295ee2a-45e5-485d-9418-b046b37095d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e295ee2a-45e5-485d-9418-b046b37095d2-plugins-config-volume\") pod \"e295ee2a-45e5-485d-9418-b046b37095d2\" (UID: \"e295ee2a-45e5-485d-9418-b046b37095d2\") " Oct 30 18:21:40.654443 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.654409 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e295ee2a-45e5-485d-9418-b046b37095d2-plugins-config-volume" pod "e295ee2a-45e5-485d-9418-b046b37095d2" (UID: "e295ee2a-45e5-485d-9418-b046b37095d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:21:40.654874 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.654850 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db69e2c-9e56-4b22-b8e4-3d346b92c062-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:21:40.654966 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.654890 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db69e2c-9e56-4b22-b8e4-3d346b92c062-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:21:40.654966 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.654902 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db69e2c-9e56-4b22-b8e4-3d346b92c062-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:21:40.654966 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.654915 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db69e2c-9e56-4b22-b8e4-3d346b92c062-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:21:40.654966 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.654928 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e295ee2a-45e5-485d-9418-b046b37095d2-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:21:40.654966 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.654943 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db69e2c-9e56-4b22-b8e4-3d346b92c062-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:21:40.654966 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.654956 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6db69e2c-9e56-4b22-b8e4-3d346b92c062-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:21:40.654966 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.654968 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db69e2c-9e56-4b22-b8e4-3d346b92c062-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:21:40.655441 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.654981 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44gjh\" (UniqueName: \"kubernetes.io/projected/6db69e2c-9e56-4b22-b8e4-3d346b92c062-kube-api-access-44gjh\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:21:40.655441 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.654993 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db69e2c-9e56-4b22-b8e4-3d346b92c062-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:21:40.656479 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.656450 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e295ee2a-45e5-485d-9418-b046b37095d2-kube-api-access-mm282" pod "e295ee2a-45e5-485d-9418-b046b37095d2" (UID: "e295ee2a-45e5-485d-9418-b046b37095d2"). InnerVolumeSpecName "kube-api-access-mm282". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:40.756516 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:40.756473 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm282\" (UniqueName: \"kubernetes.io/projected/e295ee2a-45e5-485d-9418-b046b37095d2-kube-api-access-mm282\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:21:41.069524 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:41.069480 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f233a98-d41a-4a25-9901-418e48cebadd" path="/var/lib/kubelet/pods/6f233a98-d41a-4a25-9901-418e48cebadd/volumes" Oct 30 18:21:41.485401 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:41.485371 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t" Oct 30 18:21:41.488205 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:41.488178 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ee874-69fddfb67-hscz8" event={"ID":"e295ee2a-45e5-485d-9418-b046b37095d2","Type":"ContainerDied","Data":"c45f95ff72dce7eb3a6ec9380fc971d2c02abdb43c290e4c24e7d6b7ddd90acd"} Oct 30 18:21:41.488357 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:41.488229 4442 scope.go:122] "RemoveContainer" containerID="ac06a0977b9a226ebefb88de50e0e34679f723b72fd4f867967c4e8eb8266e6a" Oct 30 18:21:41.488436 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:41.488419 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ee874-69fddfb67-hscz8" Oct 30 18:21:41.538315 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:41.538278 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t"] Oct 30 18:21:41.551128 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:41.551100 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-f257t"] Oct 30 18:21:41.575116 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:41.575088 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2ee874-69fddfb67-hscz8"] Oct 30 18:21:41.579418 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:41.579394 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2ee874-69fddfb67-hscz8"] Oct 30 18:21:43.102638 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:43.102587 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6db69e2c-9e56-4b22-b8e4-3d346b92c062" path="/var/lib/kubelet/pods/6db69e2c-9e56-4b22-b8e4-3d346b92c062/volumes" Oct 30 18:21:43.103785 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:43.103762 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e295ee2a-45e5-485d-9418-b046b37095d2" path="/var/lib/kubelet/pods/e295ee2a-45e5-485d-9418-b046b37095d2/volumes" Oct 30 18:21:43.697546 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:43.696629 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx"] Oct 30 18:21:43.697893 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:43.697783 4442 cpu_manager.go:414] "containerMap: removing container" podUID="e295ee2a-45e5-485d-9418-b046b37095d2" containerName="payload-pre-processing" Oct 30 18:21:43.697893 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:43.697814 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e295ee2a-45e5-485d-9418-b046b37095d2" containerName="payload-pre-processing" Oct 30 18:21:43.697893 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:43.697874 4442 cpu_manager.go:414] "containerMap: removing container" podUID="6f233a98-d41a-4a25-9901-418e48cebadd" containerName="payload-processing" Oct 30 18:21:43.697893 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:43.697882 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f233a98-d41a-4a25-9901-418e48cebadd" containerName="payload-processing" Oct 30 18:21:43.697893 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:43.697896 4442 cpu_manager.go:414] "containerMap: removing container" podUID="6db69e2c-9e56-4b22-b8e4-3d346b92c062" containerName="istio-proxy" Oct 30 18:21:43.698388 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:43.697903 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6db69e2c-9e56-4b22-b8e4-3d346b92c062" containerName="istio-proxy" Oct 30 18:21:43.698388 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:43.698061 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="e295ee2a-45e5-485d-9418-b046b37095d2" containerName="payload-pre-processing" Oct 30 18:21:43.698388 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:43.698083 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f233a98-d41a-4a25-9901-418e48cebadd" containerName="payload-processing" Oct 30 18:21:43.698388 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:43.698095 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="6db69e2c-9e56-4b22-b8e4-3d346b92c062" containerName="istio-proxy" Oct 30 18:21:43.704051 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:43.704025 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" Oct 30 18:21:43.780172 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:43.775936 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-10f1a3-openshift-default-dockercfg-2hlrx\"" Oct 30 18:21:43.781507 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:43.781421 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx"] Oct 30 18:21:43.853161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:43.847278 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-istio-envoy\") pod \"e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx\" (UID: \"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214\") " pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" Oct 30 18:21:43.853161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:43.847365 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-istio-token\") pod \"e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx\" (UID: \"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214\") " pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" Oct 30 18:21:43.853161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:43.847448 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-workload-certs\") pod \"e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx\" (UID: \"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214\") " pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" Oct 30 18:21:43.853161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:43.847490 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-credential-socket\") pod \"e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx\" (UID: \"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214\") " pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" Oct 30 18:21:43.853161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:43.847584 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t7sj\" (UniqueName: \"kubernetes.io/projected/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-kube-api-access-9t7sj\") pod \"e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx\" (UID: \"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214\") " pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" Oct 30 18:21:43.853161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:43.847665 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-istio-podinfo\") pod \"e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx\" (UID: \"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214\") " pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" Oct 30 18:21:43.853161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:43.847778 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-istio-data\") pod \"e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx\" (UID: \"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214\") " pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" Oct 30 18:21:43.853161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:43.847816 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-workload-socket\") pod \"e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx\" (UID: \"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214\") " pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" Oct 30 18:21:43.853161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:43.847889 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-istiod-ca-cert\") pod \"e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx\" (UID: \"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214\") " pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" Oct 30 18:21:44.029213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:44.016119 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-istio-data\") pod \"e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx\" (UID: \"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214\") " pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" Oct 30 18:21:44.029213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:44.016200 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-workload-socket\") pod \"e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx\" (UID: \"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214\") " pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" Oct 30 18:21:44.029213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:44.016257 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-istiod-ca-cert\") pod \"e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx\" (UID: \"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214\") " pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" Oct 30 18:21:44.029213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:44.016306 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-istio-envoy\") pod \"e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx\" (UID: \"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214\") " pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" Oct 30 18:21:44.029213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:44.016346 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-istio-token\") pod \"e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx\" (UID: \"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214\") " pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" Oct 30 18:21:44.029213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:44.016408 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-workload-certs\") pod \"e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx\" (UID: \"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214\") " pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" Oct 30 18:21:44.029213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:44.016447 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-credential-socket\") pod \"e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx\" (UID: \"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214\") " pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" Oct 30 18:21:44.029213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:44.016529 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t7sj\" (UniqueName: \"kubernetes.io/projected/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-kube-api-access-9t7sj\") pod \"e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx\" (UID: \"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214\") " pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" Oct 30 18:21:44.029213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:44.016607 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-istio-podinfo\") pod \"e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx\" (UID: \"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214\") " pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" Oct 30 18:21:44.063455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:44.044775 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-istio-data\") pod \"e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx\" (UID: \"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214\") " pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" Oct 30 18:21:44.063455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:44.044901 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-credential-socket\") pod \"e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx\" (UID: \"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214\") " pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" Oct 30 18:21:44.063455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:44.045257 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-workload-certs\") pod \"e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx\" (UID: \"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214\") " pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" Oct 30 18:21:44.063455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:44.045622 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-workload-socket\") pod \"e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx\" (UID: \"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214\") " pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" Oct 30 18:21:44.063455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:44.045714 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-istiod-ca-cert\") pod \"e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx\" (UID: \"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214\") " pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" Oct 30 18:21:44.063455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:44.059932 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-istio-podinfo\") pod \"e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx\" (UID: \"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214\") " pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" Oct 30 18:21:44.063455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:44.060145 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-istio-envoy\") pod \"e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx\" (UID: \"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214\") " pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" Oct 30 18:21:44.083703 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:44.083662 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t7sj\" (UniqueName: \"kubernetes.io/projected/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-kube-api-access-9t7sj\") pod \"e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx\" (UID: \"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214\") " pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" Oct 30 18:21:44.087759 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:44.087728 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-istio-token\") pod \"e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx\" (UID: \"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214\") " pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" Oct 30 18:21:44.341925 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:44.341826 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" Oct 30 18:21:44.715572 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:44.715544 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx"] Oct 30 18:21:44.722575 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:21:44.722540 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9e1bc7b_1615_4d9d_9a73_ee8a58cf4214.slice/crio-37beac749322b5d49a607bfe9c0095d4ebf13dbb436838b83ca71e5a0fa39161 WatchSource:0}: Error finding container 37beac749322b5d49a607bfe9c0095d4ebf13dbb436838b83ca71e5a0fa39161: Status 404 returned error can't find the container with id 37beac749322b5d49a607bfe9c0095d4ebf13dbb436838b83ca71e5a0fa39161 Oct 30 18:21:44.725567 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:44.725522 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:21:44.725694 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:44.725619 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:21:44.725818 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:44.725690 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:21:45.514279 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.514251 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" Oct 30 18:21:45.558808 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.558774 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-istio-token\" (UniqueName: \"kubernetes.io/projected/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-istio-token\") pod \"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42\" (UID: \"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42\") " Oct 30 18:21:45.558955 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.558852 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-istio-podinfo\") pod \"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42\" (UID: \"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42\") " Oct 30 18:21:45.558955 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.558908 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-kube-api-access-z4dgk\" (UniqueName: \"kubernetes.io/projected/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-kube-api-access-z4dgk\") pod \"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42\" (UID: \"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42\") " Oct 30 18:21:45.558955 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.558949 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-workload-socket\") pod \"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42\" (UID: \"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42\") " Oct 30 18:21:45.559225 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.559002 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-istiod-ca-cert\") pod \"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42\" (UID: \"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42\") " Oct 30 18:21:45.559225 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.559029 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-istio-data\") pod \"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42\" (UID: \"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42\") " Oct 30 18:21:45.559225 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.559187 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-credential-socket\") pod \"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42\" (UID: \"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42\") " Oct 30 18:21:45.559383 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.559229 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-workload-certs\") pod \"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42\" (UID: \"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42\") " Oct 30 18:21:45.559383 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.559264 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-istio-envoy\") pod \"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42\" (UID: \"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42\") " Oct 30 18:21:45.560439 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.560248 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-workload-socket" pod "b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42" (UID: "b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:45.560573 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.560444 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-credential-socket" pod "b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42" (UID: "b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:45.560673 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.560649 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-istio-data" pod "b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42" (UID: "b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:45.560903 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.560884 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" Oct 30 18:21:45.561076 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.560930 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-istiod-ca-cert" pod "b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42" (UID: "b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:21:45.561212 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.560981 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-workload-certs" pod "b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42" (UID: "b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:45.564177 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.564012 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-istio-podinfo" pod "b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42" (UID: "b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:21:45.564177 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.564077 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-istio-envoy" pod "b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42" (UID: "b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:45.564335 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.564299 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-istio-token" pod "b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42" (UID: "b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:45.565578 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.565553 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-kube-api-access-z4dgk" pod "b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42" (UID: "b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42"). InnerVolumeSpecName "kube-api-access-z4dgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:45.569007 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.568984 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" Oct 30 18:21:45.588024 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.587985 4442 generic.go:356] "Generic (PLEG): container finished" podID="b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42" containerID="84431ef14fb17489f30111c3a32731796835c1e6d3fd6cdb900e025731d19026" exitCode=0 Oct 30 18:21:45.588208 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.588055 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" Oct 30 18:21:45.588208 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.588070 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" event={"ID":"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42","Type":"ContainerDied","Data":"84431ef14fb17489f30111c3a32731796835c1e6d3fd6cdb900e025731d19026"} Oct 30 18:21:45.588208 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.588168 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789" event={"ID":"b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42","Type":"ContainerDied","Data":"8140051ce436ca7bf3c3fa3836c182cc404b4617f6e29e58ec072189940bcf22"} Oct 30 18:21:45.588208 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.588201 4442 scope.go:122] "RemoveContainer" containerID="84431ef14fb17489f30111c3a32731796835c1e6d3fd6cdb900e025731d19026" Oct 30 18:21:45.591860 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.590978 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" event={"ID":"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214","Type":"ContainerStarted","Data":"fe437537e5a1ea841df80e36da3e51f00839aa35a49ad76b6cf07e0236736778"} Oct 30 18:21:45.591860 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.591026 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" event={"ID":"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214","Type":"ContainerStarted","Data":"37beac749322b5d49a607bfe9c0095d4ebf13dbb436838b83ca71e5a0fa39161"} Oct 30 18:21:45.597630 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.597597 4442 generic.go:356] "Generic (PLEG): container finished" podID="f6a0e550-b203-433c-8991-387f2179e0de" containerID="4bed610050ec6cda902438aae0dcc950d53cc9ebee5302bf95a529cc8f48426f" exitCode=0 Oct 30 18:21:45.597767 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.597728 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" event={"ID":"f6a0e550-b203-433c-8991-387f2179e0de","Type":"ContainerDied","Data":"4bed610050ec6cda902438aae0dcc950d53cc9ebee5302bf95a529cc8f48426f"} Oct 30 18:21:45.597840 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.597772 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" event={"ID":"f6a0e550-b203-433c-8991-387f2179e0de","Type":"ContainerDied","Data":"0cdfd330ac649e8573daf5611ee8c2af451c8e42f68bc780d20da3c7bb1d2165"} Oct 30 18:21:45.597901 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.597855 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx" Oct 30 18:21:45.599819 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.599793 4442 generic.go:356] "Generic (PLEG): container finished" podID="d9b6ff41-9b4a-43e1-8a09-b99b2014530c" containerID="b04013d4b8552b82935627b34ab847c949a27d0038b27a31e64e4676e7c67653" exitCode=0 Oct 30 18:21:45.599910 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.599843 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" event={"ID":"d9b6ff41-9b4a-43e1-8a09-b99b2014530c","Type":"ContainerDied","Data":"b04013d4b8552b82935627b34ab847c949a27d0038b27a31e64e4676e7c67653"} Oct 30 18:21:45.599910 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.599881 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" event={"ID":"d9b6ff41-9b4a-43e1-8a09-b99b2014530c","Type":"ContainerDied","Data":"758513067df1c18571fd8b40d45ed3101e86a794e1d0615e3a62a886d0b3bad7"} Oct 30 18:21:45.600019 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.599952 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m" Oct 30 18:21:45.605535 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.605516 4442 scope.go:122] "RemoveContainer" containerID="84431ef14fb17489f30111c3a32731796835c1e6d3fd6cdb900e025731d19026" Oct 30 18:21:45.605888 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:21:45.605866 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84431ef14fb17489f30111c3a32731796835c1e6d3fd6cdb900e025731d19026\": container with ID starting with 84431ef14fb17489f30111c3a32731796835c1e6d3fd6cdb900e025731d19026 not found: ID does not exist" containerID="84431ef14fb17489f30111c3a32731796835c1e6d3fd6cdb900e025731d19026" Oct 30 18:21:45.605969 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.605898 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84431ef14fb17489f30111c3a32731796835c1e6d3fd6cdb900e025731d19026"} err="failed to get container status \"84431ef14fb17489f30111c3a32731796835c1e6d3fd6cdb900e025731d19026\": rpc error: code = NotFound desc = could not find container \"84431ef14fb17489f30111c3a32731796835c1e6d3fd6cdb900e025731d19026\": container with ID starting with 84431ef14fb17489f30111c3a32731796835c1e6d3fd6cdb900e025731d19026 not found: ID does not exist" Oct 30 18:21:45.605969 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.605921 4442 scope.go:122] "RemoveContainer" containerID="4bed610050ec6cda902438aae0dcc950d53cc9ebee5302bf95a529cc8f48426f" Oct 30 18:21:45.625446 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.624280 4442 scope.go:122] "RemoveContainer" containerID="4bed610050ec6cda902438aae0dcc950d53cc9ebee5302bf95a529cc8f48426f" Oct 30 18:21:45.625446 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:21:45.624731 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bed610050ec6cda902438aae0dcc950d53cc9ebee5302bf95a529cc8f48426f\": container with ID starting with 4bed610050ec6cda902438aae0dcc950d53cc9ebee5302bf95a529cc8f48426f not found: ID does not exist" containerID="4bed610050ec6cda902438aae0dcc950d53cc9ebee5302bf95a529cc8f48426f" Oct 30 18:21:45.625446 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.624764 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bed610050ec6cda902438aae0dcc950d53cc9ebee5302bf95a529cc8f48426f"} err="failed to get container status \"4bed610050ec6cda902438aae0dcc950d53cc9ebee5302bf95a529cc8f48426f\": rpc error: code = NotFound desc = could not find container \"4bed610050ec6cda902438aae0dcc950d53cc9ebee5302bf95a529cc8f48426f\": container with ID starting with 4bed610050ec6cda902438aae0dcc950d53cc9ebee5302bf95a529cc8f48426f not found: ID does not exist" Oct 30 18:21:45.625446 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.624789 4442 scope.go:122] "RemoveContainer" containerID="b04013d4b8552b82935627b34ab847c949a27d0038b27a31e64e4676e7c67653" Oct 30 18:21:45.640940 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.640916 4442 scope.go:122] "RemoveContainer" containerID="b04013d4b8552b82935627b34ab847c949a27d0038b27a31e64e4676e7c67653" Oct 30 18:21:45.641376 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:21:45.641331 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b04013d4b8552b82935627b34ab847c949a27d0038b27a31e64e4676e7c67653\": container with ID starting with b04013d4b8552b82935627b34ab847c949a27d0038b27a31e64e4676e7c67653 not found: ID does not exist" containerID="b04013d4b8552b82935627b34ab847c949a27d0038b27a31e64e4676e7c67653" Oct 30 18:21:45.641499 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.641377 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b04013d4b8552b82935627b34ab847c949a27d0038b27a31e64e4676e7c67653"} err="failed to get container status \"b04013d4b8552b82935627b34ab847c949a27d0038b27a31e64e4676e7c67653\": rpc error: code = NotFound desc = could not find container \"b04013d4b8552b82935627b34ab847c949a27d0038b27a31e64e4676e7c67653\": container with ID starting with b04013d4b8552b82935627b34ab847c949a27d0038b27a31e64e4676e7c67653 not found: ID does not exist" Oct 30 18:21:45.660447 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.660416 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6a0e550-b203-433c-8991-387f2179e0de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6a0e550-b203-433c-8991-387f2179e0de-istio-data\") pod \"f6a0e550-b203-433c-8991-387f2179e0de\" (UID: \"f6a0e550-b203-433c-8991-387f2179e0de\") " Oct 30 18:21:45.660629 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.660456 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6a0e550-b203-433c-8991-387f2179e0de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6a0e550-b203-433c-8991-387f2179e0de-istiod-ca-cert\") pod \"f6a0e550-b203-433c-8991-387f2179e0de\" (UID: \"f6a0e550-b203-433c-8991-387f2179e0de\") " Oct 30 18:21:45.660629 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.660526 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6a0e550-b203-433c-8991-387f2179e0de-kube-api-access-wh5zd\" (UniqueName: \"kubernetes.io/projected/f6a0e550-b203-433c-8991-387f2179e0de-kube-api-access-wh5zd\") pod \"f6a0e550-b203-433c-8991-387f2179e0de\" (UID: \"f6a0e550-b203-433c-8991-387f2179e0de\") " Oct 30 18:21:45.660629 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.660547 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-kube-api-access-q92v4\" (UniqueName: \"kubernetes.io/projected/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-kube-api-access-q92v4\") pod \"d9b6ff41-9b4a-43e1-8a09-b99b2014530c\" (UID: \"d9b6ff41-9b4a-43e1-8a09-b99b2014530c\") " Oct 30 18:21:45.660629 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.660576 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-istio-podinfo\") pod \"d9b6ff41-9b4a-43e1-8a09-b99b2014530c\" (UID: \"d9b6ff41-9b4a-43e1-8a09-b99b2014530c\") " Oct 30 18:21:45.660629 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.660602 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-istio-data\") pod \"d9b6ff41-9b4a-43e1-8a09-b99b2014530c\" (UID: \"d9b6ff41-9b4a-43e1-8a09-b99b2014530c\") " Oct 30 18:21:45.660629 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.660625 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6a0e550-b203-433c-8991-387f2179e0de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6a0e550-b203-433c-8991-387f2179e0de-credential-socket\") pod \"f6a0e550-b203-433c-8991-387f2179e0de\" (UID: \"f6a0e550-b203-433c-8991-387f2179e0de\") " Oct 30 18:21:45.660951 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.660658 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6a0e550-b203-433c-8991-387f2179e0de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6a0e550-b203-433c-8991-387f2179e0de-workload-certs\") pod \"f6a0e550-b203-433c-8991-387f2179e0de\" (UID: \"f6a0e550-b203-433c-8991-387f2179e0de\") " Oct 30 18:21:45.660951 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.660688 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-istiod-ca-cert\") pod \"d9b6ff41-9b4a-43e1-8a09-b99b2014530c\" (UID: \"d9b6ff41-9b4a-43e1-8a09-b99b2014530c\") " Oct 30 18:21:45.660951 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.660735 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-istio-token\" (UniqueName: \"kubernetes.io/projected/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-istio-token\") pod \"d9b6ff41-9b4a-43e1-8a09-b99b2014530c\" (UID: \"d9b6ff41-9b4a-43e1-8a09-b99b2014530c\") " Oct 30 18:21:45.660951 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.660758 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6a0e550-b203-433c-8991-387f2179e0de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6a0e550-b203-433c-8991-387f2179e0de-istio-envoy\") pod \"f6a0e550-b203-433c-8991-387f2179e0de\" (UID: \"f6a0e550-b203-433c-8991-387f2179e0de\") " Oct 30 18:21:45.660951 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.660787 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6a0e550-b203-433c-8991-387f2179e0de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6a0e550-b203-433c-8991-387f2179e0de-workload-socket\") pod \"f6a0e550-b203-433c-8991-387f2179e0de\" (UID: \"f6a0e550-b203-433c-8991-387f2179e0de\") " Oct 30 18:21:45.660951 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.660858 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-credential-socket\") pod \"d9b6ff41-9b4a-43e1-8a09-b99b2014530c\" (UID: \"d9b6ff41-9b4a-43e1-8a09-b99b2014530c\") " Oct 30 18:21:45.660951 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.660883 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-workload-certs\") pod \"d9b6ff41-9b4a-43e1-8a09-b99b2014530c\" (UID: \"d9b6ff41-9b4a-43e1-8a09-b99b2014530c\") " Oct 30 18:21:45.660951 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.660932 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6a0e550-b203-433c-8991-387f2179e0de-istio-token\" (UniqueName: \"kubernetes.io/projected/f6a0e550-b203-433c-8991-387f2179e0de-istio-token\") pod \"f6a0e550-b203-433c-8991-387f2179e0de\" (UID: \"f6a0e550-b203-433c-8991-387f2179e0de\") " Oct 30 18:21:45.660951 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.660953 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-istio-envoy\") pod \"d9b6ff41-9b4a-43e1-8a09-b99b2014530c\" (UID: \"d9b6ff41-9b4a-43e1-8a09-b99b2014530c\") " Oct 30 18:21:45.661455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.660977 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-workload-socket\") pod \"d9b6ff41-9b4a-43e1-8a09-b99b2014530c\" (UID: \"d9b6ff41-9b4a-43e1-8a09-b99b2014530c\") " Oct 30 18:21:45.661455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.660997 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6a0e550-b203-433c-8991-387f2179e0de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6a0e550-b203-433c-8991-387f2179e0de-istio-podinfo\") pod \"f6a0e550-b203-433c-8991-387f2179e0de\" (UID: \"f6a0e550-b203-433c-8991-387f2179e0de\") " Oct 30 18:21:45.661590 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.661573 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:21:45.661648 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.661593 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:21:45.661648 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.661603 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:21:45.661648 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.661616 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:21:45.661648 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.661628 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:21:45.661648 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.661635 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4dgk\" (UniqueName: \"kubernetes.io/projected/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-kube-api-access-z4dgk\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:21:45.661648 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.661644 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:21:45.661952 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.661652 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:21:45.661952 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.661660 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:21:45.663467 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.662579 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a0e550-b203-433c-8991-387f2179e0de-workload-certs" pod "f6a0e550-b203-433c-8991-387f2179e0de" (UID: "f6a0e550-b203-433c-8991-387f2179e0de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:45.663467 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.662868 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a0e550-b203-433c-8991-387f2179e0de-credential-socket" pod "f6a0e550-b203-433c-8991-387f2179e0de" (UID: "f6a0e550-b203-433c-8991-387f2179e0de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:45.663467 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.662909 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-istiod-ca-cert" pod "d9b6ff41-9b4a-43e1-8a09-b99b2014530c" (UID: "d9b6ff41-9b4a-43e1-8a09-b99b2014530c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:21:45.663467 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.663120 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-workload-socket" pod "d9b6ff41-9b4a-43e1-8a09-b99b2014530c" (UID: "d9b6ff41-9b4a-43e1-8a09-b99b2014530c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:45.663467 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.663286 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a0e550-b203-433c-8991-387f2179e0de-workload-socket" pod "f6a0e550-b203-433c-8991-387f2179e0de" (UID: "f6a0e550-b203-433c-8991-387f2179e0de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:45.663467 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.663396 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-credential-socket" pod "d9b6ff41-9b4a-43e1-8a09-b99b2014530c" (UID: "d9b6ff41-9b4a-43e1-8a09-b99b2014530c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:45.665179 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.663568 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6a0e550-b203-433c-8991-387f2179e0de-istiod-ca-cert" pod "f6a0e550-b203-433c-8991-387f2179e0de" (UID: "f6a0e550-b203-433c-8991-387f2179e0de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:21:45.665179 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.663694 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-workload-certs" pod "d9b6ff41-9b4a-43e1-8a09-b99b2014530c" (UID: "d9b6ff41-9b4a-43e1-8a09-b99b2014530c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:45.665179 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.663766 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-istio-data" pod "d9b6ff41-9b4a-43e1-8a09-b99b2014530c" (UID: "d9b6ff41-9b4a-43e1-8a09-b99b2014530c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:45.665179 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.663864 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a0e550-b203-433c-8991-387f2179e0de-istio-data" pod "f6a0e550-b203-433c-8991-387f2179e0de" (UID: "f6a0e550-b203-433c-8991-387f2179e0de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:45.671547 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.671397 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-istio-podinfo" pod "d9b6ff41-9b4a-43e1-8a09-b99b2014530c" (UID: "d9b6ff41-9b4a-43e1-8a09-b99b2014530c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:21:45.671547 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.671507 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6a0e550-b203-433c-8991-387f2179e0de-istio-token" pod "f6a0e550-b203-433c-8991-387f2179e0de" (UID: "f6a0e550-b203-433c-8991-387f2179e0de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:45.671748 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.671553 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a0e550-b203-433c-8991-387f2179e0de-istio-envoy" pod "f6a0e550-b203-433c-8991-387f2179e0de" (UID: "f6a0e550-b203-433c-8991-387f2179e0de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:45.671748 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.671648 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6a0e550-b203-433c-8991-387f2179e0de-istio-podinfo" pod "f6a0e550-b203-433c-8991-387f2179e0de" (UID: "f6a0e550-b203-433c-8991-387f2179e0de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:21:45.672338 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.672296 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-istio-token" pod "d9b6ff41-9b4a-43e1-8a09-b99b2014530c" (UID: "d9b6ff41-9b4a-43e1-8a09-b99b2014530c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:45.672338 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.672297 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6a0e550-b203-433c-8991-387f2179e0de-kube-api-access-wh5zd" pod "f6a0e550-b203-433c-8991-387f2179e0de" (UID: "f6a0e550-b203-433c-8991-387f2179e0de"). InnerVolumeSpecName "kube-api-access-wh5zd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:45.673089 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.673057 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-istio-envoy" pod "d9b6ff41-9b4a-43e1-8a09-b99b2014530c" (UID: "d9b6ff41-9b4a-43e1-8a09-b99b2014530c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:45.675462 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.675438 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-kube-api-access-q92v4" pod "d9b6ff41-9b4a-43e1-8a09-b99b2014530c" (UID: "d9b6ff41-9b4a-43e1-8a09-b99b2014530c"). InnerVolumeSpecName "kube-api-access-q92v4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:45.700266 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.700228 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789"] Oct 30 18:21:45.701901 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.701873 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wd789"] Oct 30 18:21:45.763536 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.763484 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh5zd\" (UniqueName: \"kubernetes.io/projected/f6a0e550-b203-433c-8991-387f2179e0de-kube-api-access-wh5zd\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:21:45.763536 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.763524 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q92v4\" (UniqueName: \"kubernetes.io/projected/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-kube-api-access-q92v4\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:21:45.763536 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.763539 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:21:45.763860 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.763553 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:21:45.763860 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.763570 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6a0e550-b203-433c-8991-387f2179e0de-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:21:45.763860 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.763584 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6a0e550-b203-433c-8991-387f2179e0de-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:21:45.763860 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.763597 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:21:45.763860 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.763609 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:21:45.763860 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.763621 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6a0e550-b203-433c-8991-387f2179e0de-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:21:45.763860 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.763633 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6a0e550-b203-433c-8991-387f2179e0de-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:21:45.763860 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.763649 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:21:45.763860 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.763662 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:21:45.763860 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.763675 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6a0e550-b203-433c-8991-387f2179e0de-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:21:45.763860 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.763688 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:21:45.763860 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.763700 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b6ff41-9b4a-43e1-8a09-b99b2014530c-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:21:45.763860 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.763713 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6a0e550-b203-433c-8991-387f2179e0de-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:21:45.763860 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.763727 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6a0e550-b203-433c-8991-387f2179e0de-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:21:45.763860 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.763742 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6a0e550-b203-433c-8991-387f2179e0de-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:21:45.764600 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:45.764128 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" podStartSLOduration=2.764114814 podStartE2EDuration="2.764114814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:21:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:21:45.761350433 +0000 UTC m=+1402.851412795" watchObservedRunningTime="2026-10-30 18:21:45.764114814 +0000 UTC m=+1402.854177224" Oct 30 18:21:46.017948 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:46.017918 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx"] Oct 30 18:21:46.024270 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:46.024171 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rnhxx"] Oct 30 18:21:46.057117 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:46.057081 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m"] Oct 30 18:21:46.061672 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:46.061647 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-g9j6m"] Oct 30 18:21:46.343046 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:46.342961 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" Oct 30 18:21:46.350769 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:46.350741 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" Oct 30 18:21:46.610084 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:46.610003 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" Oct 30 18:21:46.610905 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:46.610888 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" Oct 30 18:21:47.067255 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:47.067221 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42" path="/var/lib/kubelet/pods/b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42/volumes" Oct 30 18:21:47.067748 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:47.067733 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9b6ff41-9b4a-43e1-8a09-b99b2014530c" path="/var/lib/kubelet/pods/d9b6ff41-9b4a-43e1-8a09-b99b2014530c/volumes" Oct 30 18:21:47.068434 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:47.068410 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6a0e550-b203-433c-8991-387f2179e0de" path="/var/lib/kubelet/pods/f6a0e550-b203-433c-8991-387f2179e0de/volumes" Oct 30 18:21:47.564231 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:21:47.564194 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:21:50.065240 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:50.065207 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-10f1a3-d55fc9547-kj7nd"] Oct 30 18:21:50.065857 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:50.065844 4442 cpu_manager.go:414] "containerMap: removing container" podUID="b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42" containerName="istio-proxy" Oct 30 18:21:50.065905 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:50.065859 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42" containerName="istio-proxy" Oct 30 18:21:50.065905 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:50.065890 4442 cpu_manager.go:414] "containerMap: removing container" podUID="d9b6ff41-9b4a-43e1-8a09-b99b2014530c" containerName="istio-proxy" Oct 30 18:21:50.065905 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:50.065896 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9b6ff41-9b4a-43e1-8a09-b99b2014530c" containerName="istio-proxy" Oct 30 18:21:50.066000 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:50.065921 4442 cpu_manager.go:414] "containerMap: removing container" podUID="f6a0e550-b203-433c-8991-387f2179e0de" containerName="istio-proxy" Oct 30 18:21:50.066000 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:50.065926 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6a0e550-b203-433c-8991-387f2179e0de" containerName="istio-proxy" Oct 30 18:21:50.066069 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:50.066014 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9b6ff41-9b4a-43e1-8a09-b99b2014530c" containerName="istio-proxy" Oct 30 18:21:50.066069 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:50.066026 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6a0e550-b203-433c-8991-387f2179e0de" containerName="istio-proxy" Oct 30 18:21:50.066069 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:50.066034 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2dd0cbf-8c17-4435-930a-aa4ea4b4bc42" containerName="istio-proxy" Oct 30 18:21:50.071445 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:50.071425 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-10f1a3-d55fc9547-kj7nd" Oct 30 18:21:50.089902 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:50.089872 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-10f1a3-d55fc9547-kj7nd"] Oct 30 18:21:50.095228 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:50.095206 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-10f1a3-dockercfg-6f6nm\"" Oct 30 18:21:50.095228 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:50.095208 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-10f1a3\"" Oct 30 18:21:50.138521 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:50.138469 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hvs5\" (UniqueName: \"kubernetes.io/projected/288df6a1-5215-4b1a-9013-4aa4beae9d23-kube-api-access-5hvs5\") pod \"payload-pre-processing-e2e-api-a-10f1a3-d55fc9547-kj7nd\" (UID: \"288df6a1-5215-4b1a-9013-4aa4beae9d23\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-10f1a3-d55fc9547-kj7nd" Oct 30 18:21:50.138753 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:50.138625 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/288df6a1-5215-4b1a-9013-4aa4beae9d23-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-10f1a3-d55fc9547-kj7nd\" (UID: \"288df6a1-5215-4b1a-9013-4aa4beae9d23\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-10f1a3-d55fc9547-kj7nd" Oct 30 18:21:50.258160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:50.248198 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hvs5\" (UniqueName: \"kubernetes.io/projected/288df6a1-5215-4b1a-9013-4aa4beae9d23-kube-api-access-5hvs5\") pod \"payload-pre-processing-e2e-api-a-10f1a3-d55fc9547-kj7nd\" (UID: \"288df6a1-5215-4b1a-9013-4aa4beae9d23\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-10f1a3-d55fc9547-kj7nd" Oct 30 18:21:50.258160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:50.248306 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/288df6a1-5215-4b1a-9013-4aa4beae9d23-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-10f1a3-d55fc9547-kj7nd\" (UID: \"288df6a1-5215-4b1a-9013-4aa4beae9d23\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-10f1a3-d55fc9547-kj7nd" Oct 30 18:21:50.323296 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:50.272547 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/288df6a1-5215-4b1a-9013-4aa4beae9d23-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-10f1a3-d55fc9547-kj7nd\" (UID: \"288df6a1-5215-4b1a-9013-4aa4beae9d23\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-10f1a3-d55fc9547-kj7nd" Oct 30 18:21:50.448761 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:50.448710 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hvs5\" (UniqueName: \"kubernetes.io/projected/288df6a1-5215-4b1a-9013-4aa4beae9d23-kube-api-access-5hvs5\") pod \"payload-pre-processing-e2e-api-a-10f1a3-d55fc9547-kj7nd\" (UID: \"288df6a1-5215-4b1a-9013-4aa4beae9d23\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-10f1a3-d55fc9547-kj7nd" Oct 30 18:21:50.452255 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:50.452222 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-10f1a3-6f6f779d4b-95h9b"] Oct 30 18:21:50.459832 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:50.459495 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-10f1a3-6f6f779d4b-95h9b"] Oct 30 18:21:50.459832 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:50.459604 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-10f1a3-6f6f779d4b-95h9b" Oct 30 18:21:50.555768 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:50.555726 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bce19dc-a5e8-4487-b9c7-b67a03135319-plugins-config-volume\") pod \"payload-processing-e2e-api-a-10f1a3-6f6f779d4b-95h9b\" (UID: \"4bce19dc-a5e8-4487-b9c7-b67a03135319\") " pod="openshift-ingress/payload-processing-e2e-api-a-10f1a3-6f6f779d4b-95h9b" Oct 30 18:21:50.555988 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:50.555975 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhszx\" (UniqueName: \"kubernetes.io/projected/4bce19dc-a5e8-4487-b9c7-b67a03135319-kube-api-access-xhszx\") pod \"payload-processing-e2e-api-a-10f1a3-6f6f779d4b-95h9b\" (UID: \"4bce19dc-a5e8-4487-b9c7-b67a03135319\") " pod="openshift-ingress/payload-processing-e2e-api-a-10f1a3-6f6f779d4b-95h9b" Oct 30 18:21:50.657673 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:50.657634 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhszx\" (UniqueName: \"kubernetes.io/projected/4bce19dc-a5e8-4487-b9c7-b67a03135319-kube-api-access-xhszx\") pod \"payload-processing-e2e-api-a-10f1a3-6f6f779d4b-95h9b\" (UID: \"4bce19dc-a5e8-4487-b9c7-b67a03135319\") " pod="openshift-ingress/payload-processing-e2e-api-a-10f1a3-6f6f779d4b-95h9b" Oct 30 18:21:50.657855 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:50.657768 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bce19dc-a5e8-4487-b9c7-b67a03135319-plugins-config-volume\") pod \"payload-processing-e2e-api-a-10f1a3-6f6f779d4b-95h9b\" (UID: \"4bce19dc-a5e8-4487-b9c7-b67a03135319\") " pod="openshift-ingress/payload-processing-e2e-api-a-10f1a3-6f6f779d4b-95h9b" Oct 30 18:21:50.658782 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:50.658755 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bce19dc-a5e8-4487-b9c7-b67a03135319-plugins-config-volume\") pod \"payload-processing-e2e-api-a-10f1a3-6f6f779d4b-95h9b\" (UID: \"4bce19dc-a5e8-4487-b9c7-b67a03135319\") " pod="openshift-ingress/payload-processing-e2e-api-a-10f1a3-6f6f779d4b-95h9b" Oct 30 18:21:50.677972 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:50.677942 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhszx\" (UniqueName: \"kubernetes.io/projected/4bce19dc-a5e8-4487-b9c7-b67a03135319-kube-api-access-xhszx\") pod \"payload-processing-e2e-api-a-10f1a3-6f6f779d4b-95h9b\" (UID: \"4bce19dc-a5e8-4487-b9c7-b67a03135319\") " pod="openshift-ingress/payload-processing-e2e-api-a-10f1a3-6f6f779d4b-95h9b" Oct 30 18:21:50.687107 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:50.687081 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-10f1a3-d55fc9547-kj7nd" Oct 30 18:21:50.783248 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:50.782925 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-10f1a3-6f6f779d4b-95h9b" Oct 30 18:21:50.993532 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:50.993500 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-10f1a3-d55fc9547-kj7nd"] Oct 30 18:21:50.994527 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:21:50.994501 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod288df6a1_5215_4b1a_9013_4aa4beae9d23.slice/crio-d3e09f2fc310390e98a5bcd1e832f366fd55292504cf42fb07a541fdb7b049d4 WatchSource:0}: Error finding container d3e09f2fc310390e98a5bcd1e832f366fd55292504cf42fb07a541fdb7b049d4: Status 404 returned error can't find the container with id d3e09f2fc310390e98a5bcd1e832f366fd55292504cf42fb07a541fdb7b049d4 Oct 30 18:21:51.099093 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:51.099070 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-10f1a3-6f6f779d4b-95h9b"] Oct 30 18:21:51.100477 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:21:51.100448 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bce19dc_a5e8_4487_b9c7_b67a03135319.slice/crio-ceaeae6ca3debbdc1d7ff4fdd6a0693612702d6da60e59751480763d2925142e WatchSource:0}: Error finding container ceaeae6ca3debbdc1d7ff4fdd6a0693612702d6da60e59751480763d2925142e: Status 404 returned error can't find the container with id ceaeae6ca3debbdc1d7ff4fdd6a0693612702d6da60e59751480763d2925142e Oct 30 18:21:51.227480 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:51.227445 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2ee874-fcdcfc47c-ztm82"] Oct 30 18:21:51.227688 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:51.227668 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ee874-fcdcfc47c-ztm82" podUID="93c780c7-a1ca-4b63-86bb-7b59dd49745a" containerName="maas-api" containerID="cri-o://9a573d62f13e740c78fdbcbd9604f16ba7d9372de24290431104fffb16938cf4" gracePeriod=30 Oct 30 18:21:51.688161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:51.679310 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-10f1a3-6f6f779d4b-95h9b" event={"ID":"4bce19dc-a5e8-4487-b9c7-b67a03135319","Type":"ContainerStarted","Data":"d2909bf8ed1faa60a0aa95baa169a175b053ee500bb4750de466a81ddbd36cc6"} Oct 30 18:21:51.688161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:51.679382 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-10f1a3-6f6f779d4b-95h9b" event={"ID":"4bce19dc-a5e8-4487-b9c7-b67a03135319","Type":"ContainerStarted","Data":"ceaeae6ca3debbdc1d7ff4fdd6a0693612702d6da60e59751480763d2925142e"} Oct 30 18:21:51.694404 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:51.688713 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-10f1a3-6f6f779d4b-95h9b" Oct 30 18:21:51.700184 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:51.700004 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-10f1a3-d55fc9547-kj7nd" event={"ID":"288df6a1-5215-4b1a-9013-4aa4beae9d23","Type":"ContainerStarted","Data":"0fcc4387e5db6c987d7395945ec76ec193274173be432efe9d09d9293de9b5d5"} Oct 30 18:21:51.700184 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:51.700089 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-10f1a3-d55fc9547-kj7nd" event={"ID":"288df6a1-5215-4b1a-9013-4aa4beae9d23","Type":"ContainerStarted","Data":"d3e09f2fc310390e98a5bcd1e832f366fd55292504cf42fb07a541fdb7b049d4"} Oct 30 18:21:51.700399 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:51.700209 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-10f1a3-d55fc9547-kj7nd" Oct 30 18:21:51.703192 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:51.703035 4442 generic.go:356] "Generic (PLEG): container finished" podID="93c780c7-a1ca-4b63-86bb-7b59dd49745a" containerID="9a573d62f13e740c78fdbcbd9604f16ba7d9372de24290431104fffb16938cf4" exitCode=0 Oct 30 18:21:51.703192 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:51.703088 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ee874-fcdcfc47c-ztm82" event={"ID":"93c780c7-a1ca-4b63-86bb-7b59dd49745a","Type":"ContainerDied","Data":"9a573d62f13e740c78fdbcbd9604f16ba7d9372de24290431104fffb16938cf4"} Oct 30 18:21:51.763786 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:51.763759 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ee874-fcdcfc47c-ztm82" Oct 30 18:21:51.792156 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:51.783999 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-10f1a3-6f6f779d4b-95h9b" podStartSLOduration=1.7839803939999999 podStartE2EDuration="1.783980394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:21:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:21:51.737688068 +0000 UTC m=+1408.827750407" watchObservedRunningTime="2026-10-30 18:21:51.783980394 +0000 UTC m=+1408.874042734" Oct 30 18:21:51.792156 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:51.784763 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-10f1a3-d55fc9547-kj7nd" podStartSLOduration=1.784748859 podStartE2EDuration="1.784748859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:21:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:21:51.783411757 +0000 UTC m=+1408.873474100" watchObservedRunningTime="2026-10-30 18:21:51.784748859 +0000 UTC m=+1408.874811199" Oct 30 18:21:51.856217 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:51.855208 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93c780c7-a1ca-4b63-86bb-7b59dd49745a-kube-api-access-6mqwb\" (UniqueName: \"kubernetes.io/projected/93c780c7-a1ca-4b63-86bb-7b59dd49745a-kube-api-access-6mqwb\") pod \"93c780c7-a1ca-4b63-86bb-7b59dd49745a\" (UID: \"93c780c7-a1ca-4b63-86bb-7b59dd49745a\") " Oct 30 18:21:51.856217 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:51.855352 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93c780c7-a1ca-4b63-86bb-7b59dd49745a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/93c780c7-a1ca-4b63-86bb-7b59dd49745a-metrics-certs\") pod \"93c780c7-a1ca-4b63-86bb-7b59dd49745a\" (UID: \"93c780c7-a1ca-4b63-86bb-7b59dd49745a\") " Oct 30 18:21:51.856217 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:51.855514 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93c780c7-a1ca-4b63-86bb-7b59dd49745a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93c780c7-a1ca-4b63-86bb-7b59dd49745a-openshift-service-ca\") pod \"93c780c7-a1ca-4b63-86bb-7b59dd49745a\" (UID: \"93c780c7-a1ca-4b63-86bb-7b59dd49745a\") " Oct 30 18:21:51.856217 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:51.855576 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93c780c7-a1ca-4b63-86bb-7b59dd49745a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93c780c7-a1ca-4b63-86bb-7b59dd49745a-maas-api-tls\") pod \"93c780c7-a1ca-4b63-86bb-7b59dd49745a\" (UID: \"93c780c7-a1ca-4b63-86bb-7b59dd49745a\") " Oct 30 18:21:51.866949 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:51.858168 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93c780c7-a1ca-4b63-86bb-7b59dd49745a-openshift-service-ca" pod "93c780c7-a1ca-4b63-86bb-7b59dd49745a" (UID: "93c780c7-a1ca-4b63-86bb-7b59dd49745a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:21:51.880190 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:51.871685 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93c780c7-a1ca-4b63-86bb-7b59dd49745a-kube-api-access-6mqwb" pod "93c780c7-a1ca-4b63-86bb-7b59dd49745a" (UID: "93c780c7-a1ca-4b63-86bb-7b59dd49745a"). InnerVolumeSpecName "kube-api-access-6mqwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:51.880190 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:51.878753 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93c780c7-a1ca-4b63-86bb-7b59dd49745a-maas-api-tls" pod "93c780c7-a1ca-4b63-86bb-7b59dd49745a" (UID: "93c780c7-a1ca-4b63-86bb-7b59dd49745a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:21:51.910161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:51.902860 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93c780c7-a1ca-4b63-86bb-7b59dd49745a-metrics-certs" pod "93c780c7-a1ca-4b63-86bb-7b59dd49745a" (UID: "93c780c7-a1ca-4b63-86bb-7b59dd49745a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:21:51.964065 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:51.963952 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93c780c7-a1ca-4b63-86bb-7b59dd49745a-openshift-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:21:51.964065 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:51.963993 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93c780c7-a1ca-4b63-86bb-7b59dd49745a-maas-api-tls\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:21:51.964065 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:51.964009 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mqwb\" (UniqueName: \"kubernetes.io/projected/93c780c7-a1ca-4b63-86bb-7b59dd49745a-kube-api-access-6mqwb\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:21:51.964065 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:51.964021 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93c780c7-a1ca-4b63-86bb-7b59dd49745a-metrics-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:21:52.712971 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:52.712880 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ee874-fcdcfc47c-ztm82" event={"ID":"93c780c7-a1ca-4b63-86bb-7b59dd49745a","Type":"ContainerDied","Data":"cfa475aa72ddc9512b6bc4b5921a9ad46e88daf9672116b47de272c3ae776eb5"} Oct 30 18:21:52.712971 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:52.712958 4442 scope.go:122] "RemoveContainer" containerID="9a573d62f13e740c78fdbcbd9604f16ba7d9372de24290431104fffb16938cf4" Oct 30 18:21:52.713463 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:52.712900 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ee874-fcdcfc47c-ztm82" Oct 30 18:21:52.765886 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:52.765849 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2ee874-fcdcfc47c-ztm82"] Oct 30 18:21:52.776189 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:52.776158 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2ee874-fcdcfc47c-ztm82"] Oct 30 18:21:53.066786 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:53.066690 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93c780c7-a1ca-4b63-86bb-7b59dd49745a" path="/var/lib/kubelet/pods/93c780c7-a1ca-4b63-86bb-7b59dd49745a/volumes" Oct 30 18:21:54.835693 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:54.835659 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-10f1a3-698744b678-gk8pv"] Oct 30 18:21:54.836468 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:54.836450 4442 cpu_manager.go:414] "containerMap: removing container" podUID="93c780c7-a1ca-4b63-86bb-7b59dd49745a" containerName="maas-api" Oct 30 18:21:54.836468 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:54.836467 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93c780c7-a1ca-4b63-86bb-7b59dd49745a" containerName="maas-api" Oct 30 18:21:54.836617 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:54.836596 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="93c780c7-a1ca-4b63-86bb-7b59dd49745a" containerName="maas-api" Oct 30 18:21:54.841726 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:54.841701 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-10f1a3-698744b678-gk8pv" Oct 30 18:21:54.856910 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:54.856884 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-10f1a3-serving-cert\"" Oct 30 18:21:54.858850 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:54.858827 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-10f1a3-698744b678-gk8pv"] Oct 30 18:21:54.911784 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:54.911747 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de1d297f-6a7a-4466-ab3e-7582d6c63612-openshift-service-ca\") pod \"maas-api-e2e-api-a-10f1a3-698744b678-gk8pv\" (UID: \"de1d297f-6a7a-4466-ab3e-7582d6c63612\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-10f1a3-698744b678-gk8pv" Oct 30 18:21:54.911958 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:54.911882 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngxgp\" (UniqueName: \"kubernetes.io/projected/de1d297f-6a7a-4466-ab3e-7582d6c63612-kube-api-access-ngxgp\") pod \"maas-api-e2e-api-a-10f1a3-698744b678-gk8pv\" (UID: \"de1d297f-6a7a-4466-ab3e-7582d6c63612\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-10f1a3-698744b678-gk8pv" Oct 30 18:21:54.912031 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:54.911959 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de1d297f-6a7a-4466-ab3e-7582d6c63612-maas-api-tls\") pod \"maas-api-e2e-api-a-10f1a3-698744b678-gk8pv\" (UID: \"de1d297f-6a7a-4466-ab3e-7582d6c63612\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-10f1a3-698744b678-gk8pv" Oct 30 18:21:54.912214 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:54.912192 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de1d297f-6a7a-4466-ab3e-7582d6c63612-metrics-certs\") pod \"maas-api-e2e-api-a-10f1a3-698744b678-gk8pv\" (UID: \"de1d297f-6a7a-4466-ab3e-7582d6c63612\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-10f1a3-698744b678-gk8pv" Oct 30 18:21:55.014078 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:55.014048 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de1d297f-6a7a-4466-ab3e-7582d6c63612-maas-api-tls\") pod \"maas-api-e2e-api-a-10f1a3-698744b678-gk8pv\" (UID: \"de1d297f-6a7a-4466-ab3e-7582d6c63612\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-10f1a3-698744b678-gk8pv" Oct 30 18:21:55.014283 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:55.014197 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de1d297f-6a7a-4466-ab3e-7582d6c63612-metrics-certs\") pod \"maas-api-e2e-api-a-10f1a3-698744b678-gk8pv\" (UID: \"de1d297f-6a7a-4466-ab3e-7582d6c63612\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-10f1a3-698744b678-gk8pv" Oct 30 18:21:55.014362 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:55.014346 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de1d297f-6a7a-4466-ab3e-7582d6c63612-openshift-service-ca\") pod \"maas-api-e2e-api-a-10f1a3-698744b678-gk8pv\" (UID: \"de1d297f-6a7a-4466-ab3e-7582d6c63612\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-10f1a3-698744b678-gk8pv" Oct 30 18:21:55.014423 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:55.014407 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngxgp\" (UniqueName: \"kubernetes.io/projected/de1d297f-6a7a-4466-ab3e-7582d6c63612-kube-api-access-ngxgp\") pod \"maas-api-e2e-api-a-10f1a3-698744b678-gk8pv\" (UID: \"de1d297f-6a7a-4466-ab3e-7582d6c63612\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-10f1a3-698744b678-gk8pv" Oct 30 18:21:55.015062 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:55.015037 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de1d297f-6a7a-4466-ab3e-7582d6c63612-openshift-service-ca\") pod \"maas-api-e2e-api-a-10f1a3-698744b678-gk8pv\" (UID: \"de1d297f-6a7a-4466-ab3e-7582d6c63612\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-10f1a3-698744b678-gk8pv" Oct 30 18:21:55.016942 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:55.016909 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de1d297f-6a7a-4466-ab3e-7582d6c63612-metrics-certs\") pod \"maas-api-e2e-api-a-10f1a3-698744b678-gk8pv\" (UID: \"de1d297f-6a7a-4466-ab3e-7582d6c63612\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-10f1a3-698744b678-gk8pv" Oct 30 18:21:55.017038 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:55.016967 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de1d297f-6a7a-4466-ab3e-7582d6c63612-maas-api-tls\") pod \"maas-api-e2e-api-a-10f1a3-698744b678-gk8pv\" (UID: \"de1d297f-6a7a-4466-ab3e-7582d6c63612\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-10f1a3-698744b678-gk8pv" Oct 30 18:21:55.033847 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:55.033819 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngxgp\" (UniqueName: \"kubernetes.io/projected/de1d297f-6a7a-4466-ab3e-7582d6c63612-kube-api-access-ngxgp\") pod \"maas-api-e2e-api-a-10f1a3-698744b678-gk8pv\" (UID: \"de1d297f-6a7a-4466-ab3e-7582d6c63612\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-10f1a3-698744b678-gk8pv" Oct 30 18:21:55.212690 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:55.207841 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-10f1a3-698744b678-gk8pv" Oct 30 18:21:55.549289 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:55.549263 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-10f1a3-698744b678-gk8pv"] Oct 30 18:21:55.550881 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:21:55.550855 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde1d297f_6a7a_4466_ab3e_7582d6c63612.slice/crio-f9faa8e85cc50a5a9db5d14aeda57ca0c328a0f930a5c82c0226eb85b182a756 WatchSource:0}: Error finding container f9faa8e85cc50a5a9db5d14aeda57ca0c328a0f930a5c82c0226eb85b182a756: Status 404 returned error can't find the container with id f9faa8e85cc50a5a9db5d14aeda57ca0c328a0f930a5c82c0226eb85b182a756 Oct 30 18:21:55.739241 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:55.739208 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-10f1a3-698744b678-gk8pv" event={"ID":"de1d297f-6a7a-4466-ab3e-7582d6c63612","Type":"ContainerStarted","Data":"f9faa8e85cc50a5a9db5d14aeda57ca0c328a0f930a5c82c0226eb85b182a756"} Oct 30 18:21:56.748797 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:56.748759 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-10f1a3-698744b678-gk8pv" event={"ID":"de1d297f-6a7a-4466-ab3e-7582d6c63612","Type":"ContainerStarted","Data":"4a2cdb6dc6b4cd87986764150ef4eaa243a2c37fe39beae8177f86d1353e5761"} Oct 30 18:21:56.749299 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:56.748849 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-10f1a3-698744b678-gk8pv" Oct 30 18:21:56.787463 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:56.787411 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-10f1a3-698744b678-gk8pv" podStartSLOduration=2.357609354 podStartE2EDuration="2.78739641s" totalImagesPullingTime="429.787056ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:21:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:21:56.787301863 +0000 UTC m=+1413.877364202" watchObservedRunningTime="2026-10-30 18:21:56.78739641 +0000 UTC m=+1413.877458759" Oct 30 18:21:58.265251 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:58.265206 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft"] Oct 30 18:21:58.265754 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:58.265493 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" podUID="034e6427-0b95-4d7d-b6c9-05673f97a48a" containerName="istio-proxy" containerID="cri-o://dab9e5fecb7301368eaf8a03e5780bcf7583815446f8dd2a6945826d897b96f5" gracePeriod=30 Oct 30 18:21:58.838939 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:58.838893 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b"] Oct 30 18:21:58.839615 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:21:58.839580 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" podUID="64bde5b5-0bc5-42b8-9aa9-fc6495afca20" containerName="istio-proxy" containerID="cri-o://163fe66746c6646fae1b65c5d4cfb6430eeb035a852193945dece5d1a36c1b29" gracePeriod=30 Oct 30 18:22:02.716213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:02.716175 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-10f1a3-d55fc9547-kj7nd" Oct 30 18:22:02.770278 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:02.765804 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-10f1a3-698744b678-gk8pv" Oct 30 18:22:03.550287 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:03.550263 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" Oct 30 18:22:03.641609 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:03.641567 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/034e6427-0b95-4d7d-b6c9-05673f97a48a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/034e6427-0b95-4d7d-b6c9-05673f97a48a-istio-data\") pod \"034e6427-0b95-4d7d-b6c9-05673f97a48a\" (UID: \"034e6427-0b95-4d7d-b6c9-05673f97a48a\") " Oct 30 18:22:03.641825 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:03.641737 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/034e6427-0b95-4d7d-b6c9-05673f97a48a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/034e6427-0b95-4d7d-b6c9-05673f97a48a-workload-certs\") pod \"034e6427-0b95-4d7d-b6c9-05673f97a48a\" (UID: \"034e6427-0b95-4d7d-b6c9-05673f97a48a\") " Oct 30 18:22:03.641825 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:03.641790 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/034e6427-0b95-4d7d-b6c9-05673f97a48a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/034e6427-0b95-4d7d-b6c9-05673f97a48a-istio-podinfo\") pod \"034e6427-0b95-4d7d-b6c9-05673f97a48a\" (UID: \"034e6427-0b95-4d7d-b6c9-05673f97a48a\") " Oct 30 18:22:03.641920 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:03.641826 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/034e6427-0b95-4d7d-b6c9-05673f97a48a-istio-token\" (UniqueName: \"kubernetes.io/projected/034e6427-0b95-4d7d-b6c9-05673f97a48a-istio-token\") pod \"034e6427-0b95-4d7d-b6c9-05673f97a48a\" (UID: \"034e6427-0b95-4d7d-b6c9-05673f97a48a\") " Oct 30 18:22:03.641920 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:03.641912 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/034e6427-0b95-4d7d-b6c9-05673f97a48a-kube-api-access-nxp5n\" (UniqueName: \"kubernetes.io/projected/034e6427-0b95-4d7d-b6c9-05673f97a48a-kube-api-access-nxp5n\") pod \"034e6427-0b95-4d7d-b6c9-05673f97a48a\" (UID: \"034e6427-0b95-4d7d-b6c9-05673f97a48a\") " Oct 30 18:22:03.641985 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:03.641961 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/034e6427-0b95-4d7d-b6c9-05673f97a48a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/034e6427-0b95-4d7d-b6c9-05673f97a48a-credential-socket\") pod \"034e6427-0b95-4d7d-b6c9-05673f97a48a\" (UID: \"034e6427-0b95-4d7d-b6c9-05673f97a48a\") " Oct 30 18:22:03.642029 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:03.642012 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/034e6427-0b95-4d7d-b6c9-05673f97a48a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/034e6427-0b95-4d7d-b6c9-05673f97a48a-istiod-ca-cert\") pod \"034e6427-0b95-4d7d-b6c9-05673f97a48a\" (UID: \"034e6427-0b95-4d7d-b6c9-05673f97a48a\") " Oct 30 18:22:03.642069 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:03.642056 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/034e6427-0b95-4d7d-b6c9-05673f97a48a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/034e6427-0b95-4d7d-b6c9-05673f97a48a-istio-envoy\") pod \"034e6427-0b95-4d7d-b6c9-05673f97a48a\" (UID: \"034e6427-0b95-4d7d-b6c9-05673f97a48a\") " Oct 30 18:22:03.642115 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:03.642102 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/034e6427-0b95-4d7d-b6c9-05673f97a48a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/034e6427-0b95-4d7d-b6c9-05673f97a48a-workload-socket\") pod \"034e6427-0b95-4d7d-b6c9-05673f97a48a\" (UID: \"034e6427-0b95-4d7d-b6c9-05673f97a48a\") " Oct 30 18:22:03.643559 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:03.643389 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/034e6427-0b95-4d7d-b6c9-05673f97a48a-credential-socket" pod "034e6427-0b95-4d7d-b6c9-05673f97a48a" (UID: "034e6427-0b95-4d7d-b6c9-05673f97a48a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:03.643559 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:03.643403 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/034e6427-0b95-4d7d-b6c9-05673f97a48a-workload-certs" pod "034e6427-0b95-4d7d-b6c9-05673f97a48a" (UID: "034e6427-0b95-4d7d-b6c9-05673f97a48a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:03.643739 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:03.643591 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/034e6427-0b95-4d7d-b6c9-05673f97a48a-istio-data" pod "034e6427-0b95-4d7d-b6c9-05673f97a48a" (UID: "034e6427-0b95-4d7d-b6c9-05673f97a48a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:03.643739 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:03.643652 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/034e6427-0b95-4d7d-b6c9-05673f97a48a-workload-socket" pod "034e6427-0b95-4d7d-b6c9-05673f97a48a" (UID: "034e6427-0b95-4d7d-b6c9-05673f97a48a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:03.644180 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:03.644098 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/034e6427-0b95-4d7d-b6c9-05673f97a48a-istiod-ca-cert" pod "034e6427-0b95-4d7d-b6c9-05673f97a48a" (UID: "034e6427-0b95-4d7d-b6c9-05673f97a48a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:03.646318 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:03.646292 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/034e6427-0b95-4d7d-b6c9-05673f97a48a-kube-api-access-nxp5n" pod "034e6427-0b95-4d7d-b6c9-05673f97a48a" (UID: "034e6427-0b95-4d7d-b6c9-05673f97a48a"). InnerVolumeSpecName "kube-api-access-nxp5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:03.647783 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:03.647725 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/034e6427-0b95-4d7d-b6c9-05673f97a48a-istio-envoy" pod "034e6427-0b95-4d7d-b6c9-05673f97a48a" (UID: "034e6427-0b95-4d7d-b6c9-05673f97a48a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:03.656296 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:03.656246 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/034e6427-0b95-4d7d-b6c9-05673f97a48a-istio-token" pod "034e6427-0b95-4d7d-b6c9-05673f97a48a" (UID: "034e6427-0b95-4d7d-b6c9-05673f97a48a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:03.658667 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:03.658635 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/034e6427-0b95-4d7d-b6c9-05673f97a48a-istio-podinfo" pod "034e6427-0b95-4d7d-b6c9-05673f97a48a" (UID: "034e6427-0b95-4d7d-b6c9-05673f97a48a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:22:03.733695 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:03.733668 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-10f1a3-6f6f779d4b-95h9b" Oct 30 18:22:03.745151 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:03.745106 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxp5n\" (UniqueName: \"kubernetes.io/projected/034e6427-0b95-4d7d-b6c9-05673f97a48a-kube-api-access-nxp5n\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:22:03.745298 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:03.745179 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/034e6427-0b95-4d7d-b6c9-05673f97a48a-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:22:03.745298 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:03.745197 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/034e6427-0b95-4d7d-b6c9-05673f97a48a-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:22:03.745298 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:03.745210 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/034e6427-0b95-4d7d-b6c9-05673f97a48a-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:22:03.745298 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:03.745225 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/034e6427-0b95-4d7d-b6c9-05673f97a48a-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:22:03.745298 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:03.745236 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/034e6427-0b95-4d7d-b6c9-05673f97a48a-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:22:03.745298 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:03.745248 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/034e6427-0b95-4d7d-b6c9-05673f97a48a-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:22:03.745298 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:03.745261 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/034e6427-0b95-4d7d-b6c9-05673f97a48a-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:22:03.745298 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:03.745272 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/034e6427-0b95-4d7d-b6c9-05673f97a48a-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:22:03.798650 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:03.798596 4442 patch_prober.go:29] interesting pod/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:22:03.798849 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:03.798663 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" podUID="64bde5b5-0bc5-42b8-9aa9-fc6495afca20" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:22:03.813403 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:03.813363 4442 generic.go:356] "Generic (PLEG): container finished" podID="034e6427-0b95-4d7d-b6c9-05673f97a48a" containerID="dab9e5fecb7301368eaf8a03e5780bcf7583815446f8dd2a6945826d897b96f5" exitCode=0 Oct 30 18:22:03.813578 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:03.813425 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" Oct 30 18:22:03.813578 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:03.813447 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" event={"ID":"034e6427-0b95-4d7d-b6c9-05673f97a48a","Type":"ContainerDied","Data":"dab9e5fecb7301368eaf8a03e5780bcf7583815446f8dd2a6945826d897b96f5"} Oct 30 18:22:03.813578 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:03.813529 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft" event={"ID":"034e6427-0b95-4d7d-b6c9-05673f97a48a","Type":"ContainerDied","Data":"610b21867b62745678032adfd2d9901772a68c0ae8c93c00e6bd98826ce13aab"} Oct 30 18:22:03.813578 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:03.813562 4442 scope.go:122] "RemoveContainer" containerID="dab9e5fecb7301368eaf8a03e5780bcf7583815446f8dd2a6945826d897b96f5" Oct 30 18:22:03.835094 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:03.835071 4442 scope.go:122] "RemoveContainer" containerID="dab9e5fecb7301368eaf8a03e5780bcf7583815446f8dd2a6945826d897b96f5" Oct 30 18:22:03.835529 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:22:03.835495 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dab9e5fecb7301368eaf8a03e5780bcf7583815446f8dd2a6945826d897b96f5\": container with ID starting with dab9e5fecb7301368eaf8a03e5780bcf7583815446f8dd2a6945826d897b96f5 not found: ID does not exist" containerID="dab9e5fecb7301368eaf8a03e5780bcf7583815446f8dd2a6945826d897b96f5" Oct 30 18:22:03.835659 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:03.835539 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dab9e5fecb7301368eaf8a03e5780bcf7583815446f8dd2a6945826d897b96f5"} err="failed to get container status \"dab9e5fecb7301368eaf8a03e5780bcf7583815446f8dd2a6945826d897b96f5\": rpc error: code = NotFound desc = could not find container \"dab9e5fecb7301368eaf8a03e5780bcf7583815446f8dd2a6945826d897b96f5\": container with ID starting with dab9e5fecb7301368eaf8a03e5780bcf7583815446f8dd2a6945826d897b96f5 not found: ID does not exist" Oct 30 18:22:03.867506 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:03.867483 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft"] Oct 30 18:22:03.871935 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:03.871912 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-77zft"] Oct 30 18:22:04.140625 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:04.140594 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" Oct 30 18:22:04.252337 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:04.252240 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-istio-token\" (UniqueName: \"kubernetes.io/projected/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-istio-token\") pod \"64bde5b5-0bc5-42b8-9aa9-fc6495afca20\" (UID: \"64bde5b5-0bc5-42b8-9aa9-fc6495afca20\") " Oct 30 18:22:04.252337 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:04.252314 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-credential-socket\") pod \"64bde5b5-0bc5-42b8-9aa9-fc6495afca20\" (UID: \"64bde5b5-0bc5-42b8-9aa9-fc6495afca20\") " Oct 30 18:22:04.252573 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:04.252346 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-istiod-ca-cert\") pod \"64bde5b5-0bc5-42b8-9aa9-fc6495afca20\" (UID: \"64bde5b5-0bc5-42b8-9aa9-fc6495afca20\") " Oct 30 18:22:04.252573 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:04.252401 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-workload-socket\") pod \"64bde5b5-0bc5-42b8-9aa9-fc6495afca20\" (UID: \"64bde5b5-0bc5-42b8-9aa9-fc6495afca20\") " Oct 30 18:22:04.252573 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:04.252442 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-workload-certs\") pod \"64bde5b5-0bc5-42b8-9aa9-fc6495afca20\" (UID: \"64bde5b5-0bc5-42b8-9aa9-fc6495afca20\") " Oct 30 18:22:04.252573 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:04.252524 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-istio-podinfo\") pod \"64bde5b5-0bc5-42b8-9aa9-fc6495afca20\" (UID: \"64bde5b5-0bc5-42b8-9aa9-fc6495afca20\") " Oct 30 18:22:04.252573 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:04.252555 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-istio-envoy\") pod \"64bde5b5-0bc5-42b8-9aa9-fc6495afca20\" (UID: \"64bde5b5-0bc5-42b8-9aa9-fc6495afca20\") " Oct 30 18:22:04.252817 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:04.252607 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-credential-socket" pod "64bde5b5-0bc5-42b8-9aa9-fc6495afca20" (UID: "64bde5b5-0bc5-42b8-9aa9-fc6495afca20"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:04.252817 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:04.252795 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-kube-api-access-5vbjg\" (UniqueName: \"kubernetes.io/projected/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-kube-api-access-5vbjg\") pod \"64bde5b5-0bc5-42b8-9aa9-fc6495afca20\" (UID: \"64bde5b5-0bc5-42b8-9aa9-fc6495afca20\") " Oct 30 18:22:04.252919 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:04.252850 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-istio-data\") pod \"64bde5b5-0bc5-42b8-9aa9-fc6495afca20\" (UID: \"64bde5b5-0bc5-42b8-9aa9-fc6495afca20\") " Oct 30 18:22:04.252919 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:04.252863 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-istiod-ca-cert" pod "64bde5b5-0bc5-42b8-9aa9-fc6495afca20" (UID: "64bde5b5-0bc5-42b8-9aa9-fc6495afca20"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:04.254333 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:04.254111 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-istio-data" pod "64bde5b5-0bc5-42b8-9aa9-fc6495afca20" (UID: "64bde5b5-0bc5-42b8-9aa9-fc6495afca20"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:04.254333 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:04.254184 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-workload-certs" pod "64bde5b5-0bc5-42b8-9aa9-fc6495afca20" (UID: "64bde5b5-0bc5-42b8-9aa9-fc6495afca20"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:04.254333 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:04.254251 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-workload-socket" pod "64bde5b5-0bc5-42b8-9aa9-fc6495afca20" (UID: "64bde5b5-0bc5-42b8-9aa9-fc6495afca20"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:04.258823 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:04.255690 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-istio-envoy" pod "64bde5b5-0bc5-42b8-9aa9-fc6495afca20" (UID: "64bde5b5-0bc5-42b8-9aa9-fc6495afca20"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:04.258823 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:04.256186 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:22:04.258823 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:04.256210 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:22:04.258823 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:04.256226 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:22:04.258823 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:04.256252 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:22:04.258823 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:04.256275 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:22:04.258823 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:04.257634 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-kube-api-access-5vbjg" pod "64bde5b5-0bc5-42b8-9aa9-fc6495afca20" (UID: "64bde5b5-0bc5-42b8-9aa9-fc6495afca20"). InnerVolumeSpecName "kube-api-access-5vbjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:04.258823 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:04.258190 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-istio-token" pod "64bde5b5-0bc5-42b8-9aa9-fc6495afca20" (UID: "64bde5b5-0bc5-42b8-9aa9-fc6495afca20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:04.258823 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:04.258359 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-istio-podinfo" pod "64bde5b5-0bc5-42b8-9aa9-fc6495afca20" (UID: "64bde5b5-0bc5-42b8-9aa9-fc6495afca20"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:22:04.357518 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:04.357482 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vbjg\" (UniqueName: \"kubernetes.io/projected/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-kube-api-access-5vbjg\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:22:04.357518 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:04.357512 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:22:04.357707 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:04.357527 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:22:04.357707 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:04.357538 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64bde5b5-0bc5-42b8-9aa9-fc6495afca20-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:22:04.824369 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:04.824329 4442 generic.go:356] "Generic (PLEG): container finished" podID="64bde5b5-0bc5-42b8-9aa9-fc6495afca20" containerID="163fe66746c6646fae1b65c5d4cfb6430eeb035a852193945dece5d1a36c1b29" exitCode=0 Oct 30 18:22:04.824874 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:04.824410 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" Oct 30 18:22:04.824874 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:04.824424 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" event={"ID":"64bde5b5-0bc5-42b8-9aa9-fc6495afca20","Type":"ContainerDied","Data":"163fe66746c6646fae1b65c5d4cfb6430eeb035a852193945dece5d1a36c1b29"} Oct 30 18:22:04.824874 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:04.824530 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b" event={"ID":"64bde5b5-0bc5-42b8-9aa9-fc6495afca20","Type":"ContainerDied","Data":"2af47bb18f9225f46d40e3f294ea4a90136127cd35a1ffa9f4a416dff0920001"} Oct 30 18:22:04.824874 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:04.824560 4442 scope.go:122] "RemoveContainer" containerID="163fe66746c6646fae1b65c5d4cfb6430eeb035a852193945dece5d1a36c1b29" Oct 30 18:22:04.838858 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:04.838835 4442 scope.go:122] "RemoveContainer" containerID="267409ebd1bf2a05467b52052560a601115e9a93eb2d80d7effd332260f35161" Oct 30 18:22:04.851848 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:04.851829 4442 scope.go:122] "RemoveContainer" containerID="163fe66746c6646fae1b65c5d4cfb6430eeb035a852193945dece5d1a36c1b29" Oct 30 18:22:04.852279 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:22:04.852251 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"163fe66746c6646fae1b65c5d4cfb6430eeb035a852193945dece5d1a36c1b29\": container with ID starting with 163fe66746c6646fae1b65c5d4cfb6430eeb035a852193945dece5d1a36c1b29 not found: ID does not exist" containerID="163fe66746c6646fae1b65c5d4cfb6430eeb035a852193945dece5d1a36c1b29" Oct 30 18:22:04.852393 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:04.852287 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"163fe66746c6646fae1b65c5d4cfb6430eeb035a852193945dece5d1a36c1b29"} err="failed to get container status \"163fe66746c6646fae1b65c5d4cfb6430eeb035a852193945dece5d1a36c1b29\": rpc error: code = NotFound desc = could not find container \"163fe66746c6646fae1b65c5d4cfb6430eeb035a852193945dece5d1a36c1b29\": container with ID starting with 163fe66746c6646fae1b65c5d4cfb6430eeb035a852193945dece5d1a36c1b29 not found: ID does not exist" Oct 30 18:22:04.852393 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:04.852304 4442 scope.go:122] "RemoveContainer" containerID="267409ebd1bf2a05467b52052560a601115e9a93eb2d80d7effd332260f35161" Oct 30 18:22:04.852649 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:22:04.852624 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"267409ebd1bf2a05467b52052560a601115e9a93eb2d80d7effd332260f35161\": container with ID starting with 267409ebd1bf2a05467b52052560a601115e9a93eb2d80d7effd332260f35161 not found: ID does not exist" containerID="267409ebd1bf2a05467b52052560a601115e9a93eb2d80d7effd332260f35161" Oct 30 18:22:04.852736 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:04.852657 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"267409ebd1bf2a05467b52052560a601115e9a93eb2d80d7effd332260f35161"} err="failed to get container status \"267409ebd1bf2a05467b52052560a601115e9a93eb2d80d7effd332260f35161\": rpc error: code = NotFound desc = could not find container \"267409ebd1bf2a05467b52052560a601115e9a93eb2d80d7effd332260f35161\": container with ID starting with 267409ebd1bf2a05467b52052560a601115e9a93eb2d80d7effd332260f35161 not found: ID does not exist" Oct 30 18:22:04.889213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:04.889183 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b"] Oct 30 18:22:04.900702 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:04.900671 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-2ee874-openshift-default-8457d78d47-lbg4b"] Oct 30 18:22:05.078183 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:05.078075 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="034e6427-0b95-4d7d-b6c9-05673f97a48a" path="/var/lib/kubelet/pods/034e6427-0b95-4d7d-b6c9-05673f97a48a/volumes" Oct 30 18:22:05.078833 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:05.078810 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64bde5b5-0bc5-42b8-9aa9-fc6495afca20" path="/var/lib/kubelet/pods/64bde5b5-0bc5-42b8-9aa9-fc6495afca20/volumes" Oct 30 18:22:15.442158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:15.441943 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-76135e-68d64b77cf-qtc44"] Oct 30 18:22:15.445928 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:15.443044 4442 cpu_manager.go:414] "containerMap: removing container" podUID="034e6427-0b95-4d7d-b6c9-05673f97a48a" containerName="istio-proxy" Oct 30 18:22:15.445928 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:15.443069 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="034e6427-0b95-4d7d-b6c9-05673f97a48a" containerName="istio-proxy" Oct 30 18:22:15.445928 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:15.443099 4442 cpu_manager.go:414] "containerMap: removing container" podUID="64bde5b5-0bc5-42b8-9aa9-fc6495afca20" containerName="istio-proxy" Oct 30 18:22:15.445928 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:15.443107 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64bde5b5-0bc5-42b8-9aa9-fc6495afca20" containerName="istio-proxy" Oct 30 18:22:15.445928 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:15.443144 4442 cpu_manager.go:414] "containerMap: removing container" podUID="64bde5b5-0bc5-42b8-9aa9-fc6495afca20" containerName="istio-proxy" Oct 30 18:22:15.445928 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:15.443153 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64bde5b5-0bc5-42b8-9aa9-fc6495afca20" containerName="istio-proxy" Oct 30 18:22:15.445928 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:15.443321 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="64bde5b5-0bc5-42b8-9aa9-fc6495afca20" containerName="istio-proxy" Oct 30 18:22:15.445928 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:15.443336 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="64bde5b5-0bc5-42b8-9aa9-fc6495afca20" containerName="istio-proxy" Oct 30 18:22:15.445928 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:15.443358 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="034e6427-0b95-4d7d-b6c9-05673f97a48a" containerName="istio-proxy" Oct 30 18:22:15.458864 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:15.454753 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76135e-68d64b77cf-qtc44" Oct 30 18:22:15.472678 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:15.472100 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-76135e-serving-cert\"" Oct 30 18:22:15.485988 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:15.485915 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-76135e-68d64b77cf-qtc44"] Oct 30 18:22:15.554640 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:15.554460 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71aab62b-754d-4bae-8b80-d52633dc0183-maas-api-tls\") pod \"maas-api-e2e-ipp-b-76135e-68d64b77cf-qtc44\" (UID: \"71aab62b-754d-4bae-8b80-d52633dc0183\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76135e-68d64b77cf-qtc44" Oct 30 18:22:15.554640 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:15.554527 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71aab62b-754d-4bae-8b80-d52633dc0183-metrics-certs\") pod \"maas-api-e2e-ipp-b-76135e-68d64b77cf-qtc44\" (UID: \"71aab62b-754d-4bae-8b80-d52633dc0183\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76135e-68d64b77cf-qtc44" Oct 30 18:22:15.554929 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:15.554833 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71aab62b-754d-4bae-8b80-d52633dc0183-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-76135e-68d64b77cf-qtc44\" (UID: \"71aab62b-754d-4bae-8b80-d52633dc0183\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76135e-68d64b77cf-qtc44" Oct 30 18:22:15.554929 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:15.554892 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqfk6\" (UniqueName: \"kubernetes.io/projected/71aab62b-754d-4bae-8b80-d52633dc0183-kube-api-access-pqfk6\") pod \"maas-api-e2e-ipp-b-76135e-68d64b77cf-qtc44\" (UID: \"71aab62b-754d-4bae-8b80-d52633dc0183\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76135e-68d64b77cf-qtc44" Oct 30 18:22:15.667162 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:15.657318 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71aab62b-754d-4bae-8b80-d52633dc0183-maas-api-tls\") pod \"maas-api-e2e-ipp-b-76135e-68d64b77cf-qtc44\" (UID: \"71aab62b-754d-4bae-8b80-d52633dc0183\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76135e-68d64b77cf-qtc44" Oct 30 18:22:15.667162 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:15.657385 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71aab62b-754d-4bae-8b80-d52633dc0183-metrics-certs\") pod \"maas-api-e2e-ipp-b-76135e-68d64b77cf-qtc44\" (UID: \"71aab62b-754d-4bae-8b80-d52633dc0183\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76135e-68d64b77cf-qtc44" Oct 30 18:22:15.667162 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:15.657559 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71aab62b-754d-4bae-8b80-d52633dc0183-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-76135e-68d64b77cf-qtc44\" (UID: \"71aab62b-754d-4bae-8b80-d52633dc0183\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76135e-68d64b77cf-qtc44" Oct 30 18:22:15.667162 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:15.657597 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqfk6\" (UniqueName: \"kubernetes.io/projected/71aab62b-754d-4bae-8b80-d52633dc0183-kube-api-access-pqfk6\") pod \"maas-api-e2e-ipp-b-76135e-68d64b77cf-qtc44\" (UID: \"71aab62b-754d-4bae-8b80-d52633dc0183\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76135e-68d64b77cf-qtc44" Oct 30 18:22:15.667162 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:15.659387 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71aab62b-754d-4bae-8b80-d52633dc0183-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-76135e-68d64b77cf-qtc44\" (UID: \"71aab62b-754d-4bae-8b80-d52633dc0183\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76135e-68d64b77cf-qtc44" Oct 30 18:22:15.667162 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:15.663771 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71aab62b-754d-4bae-8b80-d52633dc0183-maas-api-tls\") pod \"maas-api-e2e-ipp-b-76135e-68d64b77cf-qtc44\" (UID: \"71aab62b-754d-4bae-8b80-d52633dc0183\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76135e-68d64b77cf-qtc44" Oct 30 18:22:15.699273 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:15.695247 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71aab62b-754d-4bae-8b80-d52633dc0183-metrics-certs\") pod \"maas-api-e2e-ipp-b-76135e-68d64b77cf-qtc44\" (UID: \"71aab62b-754d-4bae-8b80-d52633dc0183\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76135e-68d64b77cf-qtc44" Oct 30 18:22:15.722735 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:15.722655 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqfk6\" (UniqueName: \"kubernetes.io/projected/71aab62b-754d-4bae-8b80-d52633dc0183-kube-api-access-pqfk6\") pod \"maas-api-e2e-ipp-b-76135e-68d64b77cf-qtc44\" (UID: \"71aab62b-754d-4bae-8b80-d52633dc0183\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76135e-68d64b77cf-qtc44" Oct 30 18:22:15.793949 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:15.793909 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76135e-68d64b77cf-qtc44" Oct 30 18:22:16.122051 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:22:16.122009 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71aab62b_754d_4bae_8b80_d52633dc0183.slice/crio-adaafe187455b2ec475768afb7a39b54d7c445b40f373fab0b7ad67ba1f5d346 WatchSource:0}: Error finding container adaafe187455b2ec475768afb7a39b54d7c445b40f373fab0b7ad67ba1f5d346: Status 404 returned error can't find the container with id adaafe187455b2ec475768afb7a39b54d7c445b40f373fab0b7ad67ba1f5d346 Oct 30 18:22:16.123395 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:16.123369 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-76135e-68d64b77cf-qtc44"] Oct 30 18:22:17.044126 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:17.044086 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76135e-68d64b77cf-qtc44" event={"ID":"71aab62b-754d-4bae-8b80-d52633dc0183","Type":"ContainerStarted","Data":"68124cbc6cd8c12fcc810c82d8e98a28ac1d801d3e4b1ab18b0f36bcf51bddfc"} Oct 30 18:22:17.044667 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:17.044183 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76135e-68d64b77cf-qtc44" event={"ID":"71aab62b-754d-4bae-8b80-d52633dc0183","Type":"ContainerStarted","Data":"adaafe187455b2ec475768afb7a39b54d7c445b40f373fab0b7ad67ba1f5d346"} Oct 30 18:22:17.044667 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:17.044414 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76135e-68d64b77cf-qtc44" Oct 30 18:22:17.085419 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:17.085368 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76135e-68d64b77cf-qtc44" podStartSLOduration=1.7244254159999999 podStartE2EDuration="2.085353557s" totalImagesPullingTime="360.928141ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:22:17.084964686 +0000 UTC m=+1434.175027026" watchObservedRunningTime="2026-10-30 18:22:17.085353557 +0000 UTC m=+1434.175415896" Oct 30 18:22:17.567233 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:22:17.567191 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:22:17.652350 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:17.652304 4442 patch_prober.go:29] interesting pod/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:22:17.652561 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:17.652384 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" podUID="d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:18.279284 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:18.279251 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw"] Oct 30 18:22:18.279751 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:18.279479 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" podUID="8626d789-257d-46c2-8555-8af451ed6434" containerName="istio-proxy" containerID="cri-o://035b3424b7300cb83f722cfae01122c3113e4f67e6389a1cf0e44c2da1112f6d" gracePeriod=30 Oct 30 18:22:19.064690 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:19.064601 4442 generic.go:356] "Generic (PLEG): container finished" podID="d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214" containerID="fe437537e5a1ea841df80e36da3e51f00839aa35a49ad76b6cf07e0236736778" exitCode=0 Oct 30 18:22:19.068329 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:19.068301 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" event={"ID":"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214","Type":"ContainerDied","Data":"fe437537e5a1ea841df80e36da3e51f00839aa35a49ad76b6cf07e0236736778"} Oct 30 18:22:19.068737 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:19.068722 4442 scope.go:122] "RemoveContainer" containerID="fe437537e5a1ea841df80e36da3e51f00839aa35a49ad76b6cf07e0236736778" Oct 30 18:22:19.070330 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:19.070305 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:22:19.070406 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:19.070367 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:22:19.070406 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:19.070401 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:22:19.962487 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:19.962445 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-de14f2-66c68cfd8-z94hs"] Oct 30 18:22:20.093579 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:20.093524 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" event={"ID":"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214","Type":"ContainerStarted","Data":"83874232eba21933c6e1384dcb9a6353a3c42b3a5e9328db265488473c7cb271"} Oct 30 18:22:20.351215 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:20.350163 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" Oct 30 18:22:20.351215 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:20.350332 4442 patch_prober.go:29] interesting pod/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 18:22:20.351215 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:20.350382 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" podUID="d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 18:22:20.724162 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:20.723350 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f826dd-b9f9b8586-jjsqm"] Oct 30 18:22:20.797176 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:20.794205 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-695db48787-488nc"] Oct 30 18:22:21.357156 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:21.356785 4442 patch_prober.go:29] interesting pod/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 18:22:21.357156 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:21.356847 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" podUID="d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 18:22:22.343163 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:22.342553 4442 patch_prober.go:29] interesting pod/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 18:22:22.343163 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:22.342631 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" podUID="d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 18:22:23.066828 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:23.066801 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76135e-68d64b77cf-qtc44" Oct 30 18:22:23.344193 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:23.343618 4442 patch_prober.go:29] interesting pod/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 18:22:23.344193 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:23.343684 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" podUID="d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 18:22:23.662490 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:23.662112 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" Oct 30 18:22:23.722369 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:23.722323 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8626d789-257d-46c2-8555-8af451ed6434-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8626d789-257d-46c2-8555-8af451ed6434-istio-data\") pod \"8626d789-257d-46c2-8555-8af451ed6434\" (UID: \"8626d789-257d-46c2-8555-8af451ed6434\") " Oct 30 18:22:23.722601 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:23.722396 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8626d789-257d-46c2-8555-8af451ed6434-kube-api-access-dmc8p\" (UniqueName: \"kubernetes.io/projected/8626d789-257d-46c2-8555-8af451ed6434-kube-api-access-dmc8p\") pod \"8626d789-257d-46c2-8555-8af451ed6434\" (UID: \"8626d789-257d-46c2-8555-8af451ed6434\") " Oct 30 18:22:23.722601 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:23.722425 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8626d789-257d-46c2-8555-8af451ed6434-istio-token\" (UniqueName: \"kubernetes.io/projected/8626d789-257d-46c2-8555-8af451ed6434-istio-token\") pod \"8626d789-257d-46c2-8555-8af451ed6434\" (UID: \"8626d789-257d-46c2-8555-8af451ed6434\") " Oct 30 18:22:23.722601 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:23.722477 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8626d789-257d-46c2-8555-8af451ed6434-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8626d789-257d-46c2-8555-8af451ed6434-workload-socket\") pod \"8626d789-257d-46c2-8555-8af451ed6434\" (UID: \"8626d789-257d-46c2-8555-8af451ed6434\") " Oct 30 18:22:23.722783 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:23.722688 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8626d789-257d-46c2-8555-8af451ed6434-istio-data" pod "8626d789-257d-46c2-8555-8af451ed6434" (UID: "8626d789-257d-46c2-8555-8af451ed6434"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:23.723076 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:23.723054 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8626d789-257d-46c2-8555-8af451ed6434-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:22:23.734590 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:23.734530 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8626d789-257d-46c2-8555-8af451ed6434-workload-socket" pod "8626d789-257d-46c2-8555-8af451ed6434" (UID: "8626d789-257d-46c2-8555-8af451ed6434"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:23.764688 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:23.764647 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8626d789-257d-46c2-8555-8af451ed6434-istio-token" pod "8626d789-257d-46c2-8555-8af451ed6434" (UID: "8626d789-257d-46c2-8555-8af451ed6434"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:23.778182 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:23.775123 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8626d789-257d-46c2-8555-8af451ed6434-kube-api-access-dmc8p" pod "8626d789-257d-46c2-8555-8af451ed6434" (UID: "8626d789-257d-46c2-8555-8af451ed6434"). InnerVolumeSpecName "kube-api-access-dmc8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:23.842163 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:23.832581 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8626d789-257d-46c2-8555-8af451ed6434-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8626d789-257d-46c2-8555-8af451ed6434-credential-socket\") pod \"8626d789-257d-46c2-8555-8af451ed6434\" (UID: \"8626d789-257d-46c2-8555-8af451ed6434\") " Oct 30 18:22:23.842163 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:23.832689 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8626d789-257d-46c2-8555-8af451ed6434-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8626d789-257d-46c2-8555-8af451ed6434-istio-envoy\") pod \"8626d789-257d-46c2-8555-8af451ed6434\" (UID: \"8626d789-257d-46c2-8555-8af451ed6434\") " Oct 30 18:22:23.842163 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:23.832768 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8626d789-257d-46c2-8555-8af451ed6434-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8626d789-257d-46c2-8555-8af451ed6434-istiod-ca-cert\") pod \"8626d789-257d-46c2-8555-8af451ed6434\" (UID: \"8626d789-257d-46c2-8555-8af451ed6434\") " Oct 30 18:22:23.842163 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:23.832851 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8626d789-257d-46c2-8555-8af451ed6434-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8626d789-257d-46c2-8555-8af451ed6434-workload-certs\") pod \"8626d789-257d-46c2-8555-8af451ed6434\" (UID: \"8626d789-257d-46c2-8555-8af451ed6434\") " Oct 30 18:22:23.842163 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:23.832887 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8626d789-257d-46c2-8555-8af451ed6434-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8626d789-257d-46c2-8555-8af451ed6434-istio-podinfo\") pod \"8626d789-257d-46c2-8555-8af451ed6434\" (UID: \"8626d789-257d-46c2-8555-8af451ed6434\") " Oct 30 18:22:23.842163 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:23.833631 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmc8p\" (UniqueName: \"kubernetes.io/projected/8626d789-257d-46c2-8555-8af451ed6434-kube-api-access-dmc8p\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:22:23.842163 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:23.833656 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8626d789-257d-46c2-8555-8af451ed6434-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:22:23.842163 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:23.833672 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8626d789-257d-46c2-8555-8af451ed6434-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:22:23.878812 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:23.878770 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8626d789-257d-46c2-8555-8af451ed6434-istiod-ca-cert" pod "8626d789-257d-46c2-8555-8af451ed6434" (UID: "8626d789-257d-46c2-8555-8af451ed6434"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:23.879437 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:23.879385 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8626d789-257d-46c2-8555-8af451ed6434-workload-certs" pod "8626d789-257d-46c2-8555-8af451ed6434" (UID: "8626d789-257d-46c2-8555-8af451ed6434"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:23.882519 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:23.882483 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8626d789-257d-46c2-8555-8af451ed6434-credential-socket" pod "8626d789-257d-46c2-8555-8af451ed6434" (UID: "8626d789-257d-46c2-8555-8af451ed6434"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:23.884620 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:23.884248 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8626d789-257d-46c2-8555-8af451ed6434-istio-envoy" pod "8626d789-257d-46c2-8555-8af451ed6434" (UID: "8626d789-257d-46c2-8555-8af451ed6434"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:23.919723 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:23.919635 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8626d789-257d-46c2-8555-8af451ed6434-istio-podinfo" pod "8626d789-257d-46c2-8555-8af451ed6434" (UID: "8626d789-257d-46c2-8555-8af451ed6434"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:22:23.939998 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:23.939967 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8626d789-257d-46c2-8555-8af451ed6434-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:22:23.940236 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:23.940222 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8626d789-257d-46c2-8555-8af451ed6434-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:22:23.940359 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:23.940348 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8626d789-257d-46c2-8555-8af451ed6434-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:22:23.940437 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:23.940428 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8626d789-257d-46c2-8555-8af451ed6434-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:22:23.940520 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:23.940509 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8626d789-257d-46c2-8555-8af451ed6434-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:22:24.180602 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:24.180517 4442 generic.go:356] "Generic (PLEG): container finished" podID="8626d789-257d-46c2-8555-8af451ed6434" containerID="035b3424b7300cb83f722cfae01122c3113e4f67e6389a1cf0e44c2da1112f6d" exitCode=0 Oct 30 18:22:24.180602 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:24.180563 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" Oct 30 18:22:24.181103 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:24.180603 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" event={"ID":"8626d789-257d-46c2-8555-8af451ed6434","Type":"ContainerDied","Data":"035b3424b7300cb83f722cfae01122c3113e4f67e6389a1cf0e44c2da1112f6d"} Oct 30 18:22:24.181103 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:24.180671 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw" event={"ID":"8626d789-257d-46c2-8555-8af451ed6434","Type":"ContainerDied","Data":"5fca92dc668d0690698b2a5c28a07ce04b5c99d525dba893a1d63e563d1685df"} Oct 30 18:22:24.181103 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:24.180701 4442 scope.go:122] "RemoveContainer" containerID="035b3424b7300cb83f722cfae01122c3113e4f67e6389a1cf0e44c2da1112f6d" Oct 30 18:22:24.192192 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:24.192175 4442 scope.go:122] "RemoveContainer" containerID="035b3424b7300cb83f722cfae01122c3113e4f67e6389a1cf0e44c2da1112f6d" Oct 30 18:22:24.192501 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:22:24.192477 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"035b3424b7300cb83f722cfae01122c3113e4f67e6389a1cf0e44c2da1112f6d\": container with ID starting with 035b3424b7300cb83f722cfae01122c3113e4f67e6389a1cf0e44c2da1112f6d not found: ID does not exist" containerID="035b3424b7300cb83f722cfae01122c3113e4f67e6389a1cf0e44c2da1112f6d" Oct 30 18:22:24.192567 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:24.192509 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"035b3424b7300cb83f722cfae01122c3113e4f67e6389a1cf0e44c2da1112f6d"} err="failed to get container status \"035b3424b7300cb83f722cfae01122c3113e4f67e6389a1cf0e44c2da1112f6d\": rpc error: code = NotFound desc = could not find container \"035b3424b7300cb83f722cfae01122c3113e4f67e6389a1cf0e44c2da1112f6d\": container with ID starting with 035b3424b7300cb83f722cfae01122c3113e4f67e6389a1cf0e44c2da1112f6d not found: ID does not exist" Oct 30 18:22:24.256281 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:24.256243 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw"] Oct 30 18:22:24.266205 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:24.266175 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-m7qjw"] Oct 30 18:22:24.349158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:24.348550 4442 patch_prober.go:29] interesting pod/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 18:22:24.349158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:24.348608 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" podUID="d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 18:22:24.963126 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:24.963073 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-de14f2-66c68cfd8-z94hs" podUID="51d7f087-155a-47a3-b2d5-277a678d5ec0" containerName="payload-processing" containerID="cri-o://fd35c65f7676242bffc9e780e4b095969fad81276330ee59da5ed4da78419920" gracePeriod=25 Oct 30 18:22:25.069608 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:25.069512 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8626d789-257d-46c2-8555-8af451ed6434" path="/var/lib/kubelet/pods/8626d789-257d-46c2-8555-8af451ed6434/volumes" Oct 30 18:22:25.201526 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:25.201326 4442 generic.go:356] "Generic (PLEG): container finished" podID="51d7f087-155a-47a3-b2d5-277a678d5ec0" containerID="fd35c65f7676242bffc9e780e4b095969fad81276330ee59da5ed4da78419920" exitCode=0 Oct 30 18:22:25.201971 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:25.201555 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-de14f2-66c68cfd8-z94hs" event={"ID":"51d7f087-155a-47a3-b2d5-277a678d5ec0","Type":"ContainerDied","Data":"fd35c65f7676242bffc9e780e4b095969fad81276330ee59da5ed4da78419920"} Oct 30 18:22:25.322324 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:25.322247 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-de14f2-66c68cfd8-z94hs" Oct 30 18:22:25.342823 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:25.342787 4442 patch_prober.go:29] interesting pod/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 18:22:25.343028 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:25.342851 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" podUID="d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 18:22:25.378851 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:25.378819 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51d7f087-155a-47a3-b2d5-277a678d5ec0-kube-api-access-9b52t\" (UniqueName: \"kubernetes.io/projected/51d7f087-155a-47a3-b2d5-277a678d5ec0-kube-api-access-9b52t\") pod \"51d7f087-155a-47a3-b2d5-277a678d5ec0\" (UID: \"51d7f087-155a-47a3-b2d5-277a678d5ec0\") " Oct 30 18:22:25.379247 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:25.379222 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51d7f087-155a-47a3-b2d5-277a678d5ec0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51d7f087-155a-47a3-b2d5-277a678d5ec0-plugins-config-volume\") pod \"51d7f087-155a-47a3-b2d5-277a678d5ec0\" (UID: \"51d7f087-155a-47a3-b2d5-277a678d5ec0\") " Oct 30 18:22:25.382595 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:25.381369 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51d7f087-155a-47a3-b2d5-277a678d5ec0-plugins-config-volume" pod "51d7f087-155a-47a3-b2d5-277a678d5ec0" (UID: "51d7f087-155a-47a3-b2d5-277a678d5ec0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:25.392502 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:25.392477 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51d7f087-155a-47a3-b2d5-277a678d5ec0-kube-api-access-9b52t" pod "51d7f087-155a-47a3-b2d5-277a678d5ec0" (UID: "51d7f087-155a-47a3-b2d5-277a678d5ec0"). InnerVolumeSpecName "kube-api-access-9b52t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:25.481768 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:25.481731 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b52t\" (UniqueName: \"kubernetes.io/projected/51d7f087-155a-47a3-b2d5-277a678d5ec0-kube-api-access-9b52t\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:22:25.481768 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:25.481762 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51d7f087-155a-47a3-b2d5-277a678d5ec0-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:22:25.723708 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:25.723658 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f826dd-b9f9b8586-jjsqm" podUID="7cd95b94-2685-45a6-a939-0b93af7457d1" containerName="payload-processing" containerID="cri-o://9bce653a92b49baf5437e2fabe6def341779721ab51220fd5ecadf1dbad45e3b" gracePeriod=25 Oct 30 18:22:25.794821 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:25.794771 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-695db48787-488nc" podUID="e586a1a8-85e4-47df-b62a-72c35b4c5f30" containerName="payload-pre-processing" containerID="cri-o://f6d6fad0054451f82d7dcd04c392a277fa2f347fa27e654c46facdc99a59a0dc" gracePeriod=25 Oct 30 18:22:26.078413 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:26.077644 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f826dd-b9f9b8586-jjsqm" Oct 30 18:22:26.089049 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:26.089019 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cd95b94-2685-45a6-a939-0b93af7457d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cd95b94-2685-45a6-a939-0b93af7457d1-plugins-config-volume\") pod \"7cd95b94-2685-45a6-a939-0b93af7457d1\" (UID: \"7cd95b94-2685-45a6-a939-0b93af7457d1\") " Oct 30 18:22:26.089473 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:26.089449 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cd95b94-2685-45a6-a939-0b93af7457d1-plugins-config-volume" pod "7cd95b94-2685-45a6-a939-0b93af7457d1" (UID: "7cd95b94-2685-45a6-a939-0b93af7457d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:26.089576 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:26.089485 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cd95b94-2685-45a6-a939-0b93af7457d1-kube-api-access-dndwr\" (UniqueName: \"kubernetes.io/projected/7cd95b94-2685-45a6-a939-0b93af7457d1-kube-api-access-dndwr\") pod \"7cd95b94-2685-45a6-a939-0b93af7457d1\" (UID: \"7cd95b94-2685-45a6-a939-0b93af7457d1\") " Oct 30 18:22:26.090183 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:26.090159 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cd95b94-2685-45a6-a939-0b93af7457d1-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:22:26.092319 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:26.092277 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cd95b94-2685-45a6-a939-0b93af7457d1-kube-api-access-dndwr" pod "7cd95b94-2685-45a6-a939-0b93af7457d1" (UID: "7cd95b94-2685-45a6-a939-0b93af7457d1"). InnerVolumeSpecName "kube-api-access-dndwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:26.110327 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:26.110305 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-695db48787-488nc" Oct 30 18:22:26.191587 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:26.191552 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e586a1a8-85e4-47df-b62a-72c35b4c5f30-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e586a1a8-85e4-47df-b62a-72c35b4c5f30-plugins-config-volume\") pod \"e586a1a8-85e4-47df-b62a-72c35b4c5f30\" (UID: \"e586a1a8-85e4-47df-b62a-72c35b4c5f30\") " Oct 30 18:22:26.191795 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:26.191768 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e586a1a8-85e4-47df-b62a-72c35b4c5f30-kube-api-access-xr6rx\" (UniqueName: \"kubernetes.io/projected/e586a1a8-85e4-47df-b62a-72c35b4c5f30-kube-api-access-xr6rx\") pod \"e586a1a8-85e4-47df-b62a-72c35b4c5f30\" (UID: \"e586a1a8-85e4-47df-b62a-72c35b4c5f30\") " Oct 30 18:22:26.192062 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:26.192032 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e586a1a8-85e4-47df-b62a-72c35b4c5f30-plugins-config-volume" pod "e586a1a8-85e4-47df-b62a-72c35b4c5f30" (UID: "e586a1a8-85e4-47df-b62a-72c35b4c5f30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:26.192311 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:26.192285 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e586a1a8-85e4-47df-b62a-72c35b4c5f30-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:22:26.192425 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:26.192316 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dndwr\" (UniqueName: \"kubernetes.io/projected/7cd95b94-2685-45a6-a939-0b93af7457d1-kube-api-access-dndwr\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:22:26.194783 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:26.194754 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e586a1a8-85e4-47df-b62a-72c35b4c5f30-kube-api-access-xr6rx" pod "e586a1a8-85e4-47df-b62a-72c35b4c5f30" (UID: "e586a1a8-85e4-47df-b62a-72c35b4c5f30"). InnerVolumeSpecName "kube-api-access-xr6rx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:26.213064 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:26.213027 4442 generic.go:356] "Generic (PLEG): container finished" podID="e586a1a8-85e4-47df-b62a-72c35b4c5f30" containerID="f6d6fad0054451f82d7dcd04c392a277fa2f347fa27e654c46facdc99a59a0dc" exitCode=0 Oct 30 18:22:26.213523 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:26.213088 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-695db48787-488nc" Oct 30 18:22:26.213523 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:26.213109 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-695db48787-488nc" event={"ID":"e586a1a8-85e4-47df-b62a-72c35b4c5f30","Type":"ContainerDied","Data":"f6d6fad0054451f82d7dcd04c392a277fa2f347fa27e654c46facdc99a59a0dc"} Oct 30 18:22:26.213523 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:26.213189 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-695db48787-488nc" event={"ID":"e586a1a8-85e4-47df-b62a-72c35b4c5f30","Type":"ContainerDied","Data":"a2104f48eaf4c6cbe6043c47b6ef9e42cc06d81c26379cedc120a58f5c556b45"} Oct 30 18:22:26.213523 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:26.213222 4442 scope.go:122] "RemoveContainer" containerID="f6d6fad0054451f82d7dcd04c392a277fa2f347fa27e654c46facdc99a59a0dc" Oct 30 18:22:26.214931 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:26.214907 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-de14f2-66c68cfd8-z94hs" event={"ID":"51d7f087-155a-47a3-b2d5-277a678d5ec0","Type":"ContainerDied","Data":"afaaaa0e6a4d95eca9ed6006606d319e810ca553b2a7ca0b4c541d10ca8c72a7"} Oct 30 18:22:26.215073 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:26.214993 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-de14f2-66c68cfd8-z94hs" Oct 30 18:22:26.217058 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:26.217036 4442 generic.go:356] "Generic (PLEG): container finished" podID="7cd95b94-2685-45a6-a939-0b93af7457d1" containerID="9bce653a92b49baf5437e2fabe6def341779721ab51220fd5ecadf1dbad45e3b" exitCode=0 Oct 30 18:22:26.217186 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:26.217090 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f826dd-b9f9b8586-jjsqm" Oct 30 18:22:26.217186 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:26.217088 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f826dd-b9f9b8586-jjsqm" event={"ID":"7cd95b94-2685-45a6-a939-0b93af7457d1","Type":"ContainerDied","Data":"9bce653a92b49baf5437e2fabe6def341779721ab51220fd5ecadf1dbad45e3b"} Oct 30 18:22:26.217313 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:26.217191 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f826dd-b9f9b8586-jjsqm" event={"ID":"7cd95b94-2685-45a6-a939-0b93af7457d1","Type":"ContainerDied","Data":"dfbaef48748521db352bebef0a998ce578c73b2b5b06b0ac4424466367b51b3c"} Oct 30 18:22:26.234270 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:26.234244 4442 scope.go:122] "RemoveContainer" containerID="f6d6fad0054451f82d7dcd04c392a277fa2f347fa27e654c46facdc99a59a0dc" Oct 30 18:22:26.234709 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:22:26.234674 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6d6fad0054451f82d7dcd04c392a277fa2f347fa27e654c46facdc99a59a0dc\": container with ID starting with f6d6fad0054451f82d7dcd04c392a277fa2f347fa27e654c46facdc99a59a0dc not found: ID does not exist" containerID="f6d6fad0054451f82d7dcd04c392a277fa2f347fa27e654c46facdc99a59a0dc" Oct 30 18:22:26.234793 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:26.234733 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6d6fad0054451f82d7dcd04c392a277fa2f347fa27e654c46facdc99a59a0dc"} err="failed to get container status \"f6d6fad0054451f82d7dcd04c392a277fa2f347fa27e654c46facdc99a59a0dc\": rpc error: code = NotFound desc = could not find container \"f6d6fad0054451f82d7dcd04c392a277fa2f347fa27e654c46facdc99a59a0dc\": container with ID starting with f6d6fad0054451f82d7dcd04c392a277fa2f347fa27e654c46facdc99a59a0dc not found: ID does not exist" Oct 30 18:22:26.234793 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:26.234761 4442 scope.go:122] "RemoveContainer" containerID="fd35c65f7676242bffc9e780e4b095969fad81276330ee59da5ed4da78419920" Oct 30 18:22:26.246487 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:26.246466 4442 scope.go:122] "RemoveContainer" containerID="9bce653a92b49baf5437e2fabe6def341779721ab51220fd5ecadf1dbad45e3b" Oct 30 18:22:26.258174 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:26.258110 4442 scope.go:122] "RemoveContainer" containerID="9bce653a92b49baf5437e2fabe6def341779721ab51220fd5ecadf1dbad45e3b" Oct 30 18:22:26.258514 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:22:26.258493 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bce653a92b49baf5437e2fabe6def341779721ab51220fd5ecadf1dbad45e3b\": container with ID starting with 9bce653a92b49baf5437e2fabe6def341779721ab51220fd5ecadf1dbad45e3b not found: ID does not exist" containerID="9bce653a92b49baf5437e2fabe6def341779721ab51220fd5ecadf1dbad45e3b" Oct 30 18:22:26.258594 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:26.258523 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bce653a92b49baf5437e2fabe6def341779721ab51220fd5ecadf1dbad45e3b"} err="failed to get container status \"9bce653a92b49baf5437e2fabe6def341779721ab51220fd5ecadf1dbad45e3b\": rpc error: code = NotFound desc = could not find container \"9bce653a92b49baf5437e2fabe6def341779721ab51220fd5ecadf1dbad45e3b\": container with ID starting with 9bce653a92b49baf5437e2fabe6def341779721ab51220fd5ecadf1dbad45e3b not found: ID does not exist" Oct 30 18:22:26.279719 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:26.279677 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-695db48787-488nc"] Oct 30 18:22:26.286821 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:26.286794 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-695db48787-488nc"] Oct 30 18:22:26.293944 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:26.293922 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr6rx\" (UniqueName: \"kubernetes.io/projected/e586a1a8-85e4-47df-b62a-72c35b4c5f30-kube-api-access-xr6rx\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:22:26.341829 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:26.341790 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-de14f2-66c68cfd8-z94hs"] Oct 30 18:22:26.347015 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:26.346992 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" Oct 30 18:22:26.347380 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:26.347356 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" Oct 30 18:22:26.347959 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:26.347939 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-de14f2-66c68cfd8-z94hs"] Oct 30 18:22:26.348217 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:26.348204 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" Oct 30 18:22:26.378440 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:26.378406 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f826dd-b9f9b8586-jjsqm"] Oct 30 18:22:26.382331 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:26.382305 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f826dd-b9f9b8586-jjsqm"] Oct 30 18:22:27.069017 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:27.068983 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51d7f087-155a-47a3-b2d5-277a678d5ec0" path="/var/lib/kubelet/pods/51d7f087-155a-47a3-b2d5-277a678d5ec0/volumes" Oct 30 18:22:27.069457 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:27.069440 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cd95b94-2685-45a6-a939-0b93af7457d1" path="/var/lib/kubelet/pods/7cd95b94-2685-45a6-a939-0b93af7457d1/volumes" Oct 30 18:22:27.078806 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:27.078755 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e586a1a8-85e4-47df-b62a-72c35b4c5f30" path="/var/lib/kubelet/pods/e586a1a8-85e4-47df-b62a-72c35b4c5f30/volumes" Oct 30 18:22:35.959120 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:35.959077 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0ac6bb-67fc4d5447-5n5xd"] Oct 30 18:22:35.959813 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:35.959794 4442 cpu_manager.go:414] "containerMap: removing container" podUID="7cd95b94-2685-45a6-a939-0b93af7457d1" containerName="payload-processing" Oct 30 18:22:35.959872 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:35.959817 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cd95b94-2685-45a6-a939-0b93af7457d1" containerName="payload-processing" Oct 30 18:22:35.959872 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:35.959846 4442 cpu_manager.go:414] "containerMap: removing container" podUID="8626d789-257d-46c2-8555-8af451ed6434" containerName="istio-proxy" Oct 30 18:22:35.959872 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:35.959852 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8626d789-257d-46c2-8555-8af451ed6434" containerName="istio-proxy" Oct 30 18:22:35.959982 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:35.959879 4442 cpu_manager.go:414] "containerMap: removing container" podUID="51d7f087-155a-47a3-b2d5-277a678d5ec0" containerName="payload-processing" Oct 30 18:22:35.959982 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:35.959886 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51d7f087-155a-47a3-b2d5-277a678d5ec0" containerName="payload-processing" Oct 30 18:22:35.959982 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:35.959895 4442 cpu_manager.go:414] "containerMap: removing container" podUID="e586a1a8-85e4-47df-b62a-72c35b4c5f30" containerName="payload-pre-processing" Oct 30 18:22:35.959982 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:35.959899 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e586a1a8-85e4-47df-b62a-72c35b4c5f30" containerName="payload-pre-processing" Oct 30 18:22:35.959982 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:35.959984 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="51d7f087-155a-47a3-b2d5-277a678d5ec0" containerName="payload-processing" Oct 30 18:22:35.960157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:35.959992 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="e586a1a8-85e4-47df-b62a-72c35b4c5f30" containerName="payload-pre-processing" Oct 30 18:22:35.960157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:35.959999 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cd95b94-2685-45a6-a939-0b93af7457d1" containerName="payload-processing" Oct 30 18:22:35.960157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:35.960009 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="8626d789-257d-46c2-8555-8af451ed6434" containerName="istio-proxy" Oct 30 18:22:35.963676 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:35.963657 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0ac6bb-67fc4d5447-5n5xd" Oct 30 18:22:35.982507 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:35.982469 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-0ac6bb\"" Oct 30 18:22:35.983002 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:35.982975 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0ac6bb-67fc4d5447-5n5xd"] Oct 30 18:22:35.991109 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:35.991075 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mlf5\" (UniqueName: \"kubernetes.io/projected/35427437-feb0-490d-966e-8978f4cfe11e-kube-api-access-5mlf5\") pod \"payload-pre-processing-e2e-api-b-0ac6bb-67fc4d5447-5n5xd\" (UID: \"35427437-feb0-490d-966e-8978f4cfe11e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0ac6bb-67fc4d5447-5n5xd" Oct 30 18:22:35.991266 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:35.991180 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35427437-feb0-490d-966e-8978f4cfe11e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0ac6bb-67fc4d5447-5n5xd\" (UID: \"35427437-feb0-490d-966e-8978f4cfe11e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0ac6bb-67fc4d5447-5n5xd" Oct 30 18:22:36.118167 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:36.118000 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-0ac6bb-dockercfg-d7mkn\"" Oct 30 18:22:36.152159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:36.149872 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35427437-feb0-490d-966e-8978f4cfe11e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0ac6bb-67fc4d5447-5n5xd\" (UID: \"35427437-feb0-490d-966e-8978f4cfe11e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0ac6bb-67fc4d5447-5n5xd" Oct 30 18:22:36.152159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:36.150167 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mlf5\" (UniqueName: \"kubernetes.io/projected/35427437-feb0-490d-966e-8978f4cfe11e-kube-api-access-5mlf5\") pod \"payload-pre-processing-e2e-api-b-0ac6bb-67fc4d5447-5n5xd\" (UID: \"35427437-feb0-490d-966e-8978f4cfe11e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0ac6bb-67fc4d5447-5n5xd" Oct 30 18:22:36.161189 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:36.157162 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35427437-feb0-490d-966e-8978f4cfe11e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0ac6bb-67fc4d5447-5n5xd\" (UID: \"35427437-feb0-490d-966e-8978f4cfe11e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0ac6bb-67fc4d5447-5n5xd" Oct 30 18:22:36.344860 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:36.344751 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0ac6bb-86c4fbf5cb-x5t46"] Oct 30 18:22:36.365161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:36.365008 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0ac6bb-86c4fbf5cb-x5t46"] Oct 30 18:22:36.365161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:36.365055 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0ac6bb-86c4fbf5cb-x5t46" Oct 30 18:22:36.370728 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:36.370705 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mlf5\" (UniqueName: \"kubernetes.io/projected/35427437-feb0-490d-966e-8978f4cfe11e-kube-api-access-5mlf5\") pod \"payload-pre-processing-e2e-api-b-0ac6bb-67fc4d5447-5n5xd\" (UID: \"35427437-feb0-490d-966e-8978f4cfe11e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0ac6bb-67fc4d5447-5n5xd" Oct 30 18:22:36.469015 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:36.468976 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/141a5be6-7d7b-4199-b58f-9e6831a1ac7b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0ac6bb-86c4fbf5cb-x5t46\" (UID: \"141a5be6-7d7b-4199-b58f-9e6831a1ac7b\") " pod="openshift-ingress/payload-processing-e2e-api-b-0ac6bb-86c4fbf5cb-x5t46" Oct 30 18:22:36.469239 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:36.469055 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tppvd\" (UniqueName: \"kubernetes.io/projected/141a5be6-7d7b-4199-b58f-9e6831a1ac7b-kube-api-access-tppvd\") pod \"payload-processing-e2e-api-b-0ac6bb-86c4fbf5cb-x5t46\" (UID: \"141a5be6-7d7b-4199-b58f-9e6831a1ac7b\") " pod="openshift-ingress/payload-processing-e2e-api-b-0ac6bb-86c4fbf5cb-x5t46" Oct 30 18:22:36.571040 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:36.571004 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/141a5be6-7d7b-4199-b58f-9e6831a1ac7b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0ac6bb-86c4fbf5cb-x5t46\" (UID: \"141a5be6-7d7b-4199-b58f-9e6831a1ac7b\") " pod="openshift-ingress/payload-processing-e2e-api-b-0ac6bb-86c4fbf5cb-x5t46" Oct 30 18:22:36.571299 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:36.571068 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tppvd\" (UniqueName: \"kubernetes.io/projected/141a5be6-7d7b-4199-b58f-9e6831a1ac7b-kube-api-access-tppvd\") pod \"payload-processing-e2e-api-b-0ac6bb-86c4fbf5cb-x5t46\" (UID: \"141a5be6-7d7b-4199-b58f-9e6831a1ac7b\") " pod="openshift-ingress/payload-processing-e2e-api-b-0ac6bb-86c4fbf5cb-x5t46" Oct 30 18:22:36.571977 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:36.571949 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/141a5be6-7d7b-4199-b58f-9e6831a1ac7b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0ac6bb-86c4fbf5cb-x5t46\" (UID: \"141a5be6-7d7b-4199-b58f-9e6831a1ac7b\") " pod="openshift-ingress/payload-processing-e2e-api-b-0ac6bb-86c4fbf5cb-x5t46" Oct 30 18:22:36.582797 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:36.582756 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0ac6bb-67fc4d5447-5n5xd" Oct 30 18:22:36.590268 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:36.590241 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tppvd\" (UniqueName: \"kubernetes.io/projected/141a5be6-7d7b-4199-b58f-9e6831a1ac7b-kube-api-access-tppvd\") pod \"payload-processing-e2e-api-b-0ac6bb-86c4fbf5cb-x5t46\" (UID: \"141a5be6-7d7b-4199-b58f-9e6831a1ac7b\") " pod="openshift-ingress/payload-processing-e2e-api-b-0ac6bb-86c4fbf5cb-x5t46" Oct 30 18:22:36.696259 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:36.694231 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0ac6bb-86c4fbf5cb-x5t46" Oct 30 18:22:36.915211 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:36.915183 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0ac6bb-67fc4d5447-5n5xd"] Oct 30 18:22:36.916738 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:22:36.916708 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35427437_feb0_490d_966e_8978f4cfe11e.slice/crio-b86deff7d5499e86eb0a577c6f88eb5430052ce9f1d51bf2e184fd323a403fd6 WatchSource:0}: Error finding container b86deff7d5499e86eb0a577c6f88eb5430052ce9f1d51bf2e184fd323a403fd6: Status 404 returned error can't find the container with id b86deff7d5499e86eb0a577c6f88eb5430052ce9f1d51bf2e184fd323a403fd6 Oct 30 18:22:37.082330 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:22:37.082287 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod141a5be6_7d7b_4199_b58f_9e6831a1ac7b.slice/crio-6053dca6b85dd59af437ee1e8619428f831c9cc16d05a821d47b235b308e6a6c WatchSource:0}: Error finding container 6053dca6b85dd59af437ee1e8619428f831c9cc16d05a821d47b235b308e6a6c: Status 404 returned error can't find the container with id 6053dca6b85dd59af437ee1e8619428f831c9cc16d05a821d47b235b308e6a6c Oct 30 18:22:37.085042 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:37.085019 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0ac6bb-86c4fbf5cb-x5t46"] Oct 30 18:22:37.387993 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:37.387955 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0ac6bb-67fc4d5447-5n5xd" event={"ID":"35427437-feb0-490d-966e-8978f4cfe11e","Type":"ContainerStarted","Data":"88fbf6a636da6a7cf13c3195268205ca2eff17d7c6b2a1b6d24ef1ed455c0ac5"} Oct 30 18:22:37.388205 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:37.388021 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0ac6bb-67fc4d5447-5n5xd" Oct 30 18:22:37.388205 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:37.388043 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0ac6bb-67fc4d5447-5n5xd" event={"ID":"35427437-feb0-490d-966e-8978f4cfe11e","Type":"ContainerStarted","Data":"b86deff7d5499e86eb0a577c6f88eb5430052ce9f1d51bf2e184fd323a403fd6"} Oct 30 18:22:37.389722 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:37.389695 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0ac6bb-86c4fbf5cb-x5t46" event={"ID":"141a5be6-7d7b-4199-b58f-9e6831a1ac7b","Type":"ContainerStarted","Data":"bea746ed9299c87c0ef6a391cd6664b36646177047d1c3f89c0bf08f893c709a"} Oct 30 18:22:37.389844 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:37.389742 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0ac6bb-86c4fbf5cb-x5t46" event={"ID":"141a5be6-7d7b-4199-b58f-9e6831a1ac7b","Type":"ContainerStarted","Data":"6053dca6b85dd59af437ee1e8619428f831c9cc16d05a821d47b235b308e6a6c"} Oct 30 18:22:37.389844 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:37.389772 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-0ac6bb-86c4fbf5cb-x5t46" Oct 30 18:22:37.487473 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:37.487347 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-0ac6bb-86c4fbf5cb-x5t46" podStartSLOduration=1.487328473 podStartE2EDuration="1.487328473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:22:37.483821349 +0000 UTC m=+1454.573883689" watchObservedRunningTime="2026-10-30 18:22:37.487328473 +0000 UTC m=+1454.577390814" Oct 30 18:22:37.487904 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:37.487864 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0ac6bb-67fc4d5447-5n5xd" podStartSLOduration=2.487836528 podStartE2EDuration="2.487836528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:22:37.447190612 +0000 UTC m=+1454.537252952" watchObservedRunningTime="2026-10-30 18:22:37.487836528 +0000 UTC m=+1454.577898867" Oct 30 18:22:40.803663 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:40.803629 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f826dd-66fc94df7b-vd626"] Oct 30 18:22:40.823223 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:40.823183 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f826dd-66fc94df7b-vd626" Oct 30 18:22:40.842122 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:40.842096 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f826dd-66fc94df7b-vd626"] Oct 30 18:22:40.846602 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:40.846575 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f826dd\"" Oct 30 18:22:40.846801 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:40.846778 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f826dd-dockercfg-hhs7v\"" Oct 30 18:22:40.888814 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:40.888782 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d7a6ab3-2d04-41ec-acc3-df776501aa5e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f826dd-66fc94df7b-vd626\" (UID: \"6d7a6ab3-2d04-41ec-acc3-df776501aa5e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f826dd-66fc94df7b-vd626" Oct 30 18:22:40.888992 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:40.888911 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-982dh\" (UniqueName: \"kubernetes.io/projected/6d7a6ab3-2d04-41ec-acc3-df776501aa5e-kube-api-access-982dh\") pod \"payload-processing-e2e-ipp-b-f826dd-66fc94df7b-vd626\" (UID: \"6d7a6ab3-2d04-41ec-acc3-df776501aa5e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f826dd-66fc94df7b-vd626" Oct 30 18:22:40.909016 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:40.908985 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-675f9fd755-t8pkx"] Oct 30 18:22:40.912599 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:40.912575 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-675f9fd755-t8pkx" Oct 30 18:22:40.932557 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:40.932528 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-675f9fd755-t8pkx"] Oct 30 18:22:40.991270 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:40.991238 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7tkb\" (UniqueName: \"kubernetes.io/projected/9de8d55a-7a7c-4e6d-9ec4-43a1d779f30e-kube-api-access-q7tkb\") pod \"payload-pre-processing-e2e-ipp-b-f826dd-675f9fd755-t8pkx\" (UID: \"9de8d55a-7a7c-4e6d-9ec4-43a1d779f30e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-675f9fd755-t8pkx" Oct 30 18:22:40.991526 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:40.991498 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d7a6ab3-2d04-41ec-acc3-df776501aa5e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f826dd-66fc94df7b-vd626\" (UID: \"6d7a6ab3-2d04-41ec-acc3-df776501aa5e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f826dd-66fc94df7b-vd626" Oct 30 18:22:40.991655 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:40.991637 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9de8d55a-7a7c-4e6d-9ec4-43a1d779f30e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f826dd-675f9fd755-t8pkx\" (UID: \"9de8d55a-7a7c-4e6d-9ec4-43a1d779f30e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-675f9fd755-t8pkx" Oct 30 18:22:40.991712 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:40.991692 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-982dh\" (UniqueName: \"kubernetes.io/projected/6d7a6ab3-2d04-41ec-acc3-df776501aa5e-kube-api-access-982dh\") pod \"payload-processing-e2e-ipp-b-f826dd-66fc94df7b-vd626\" (UID: \"6d7a6ab3-2d04-41ec-acc3-df776501aa5e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f826dd-66fc94df7b-vd626" Oct 30 18:22:40.992399 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:40.992355 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d7a6ab3-2d04-41ec-acc3-df776501aa5e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f826dd-66fc94df7b-vd626\" (UID: \"6d7a6ab3-2d04-41ec-acc3-df776501aa5e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f826dd-66fc94df7b-vd626" Oct 30 18:22:41.012237 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:41.012208 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-982dh\" (UniqueName: \"kubernetes.io/projected/6d7a6ab3-2d04-41ec-acc3-df776501aa5e-kube-api-access-982dh\") pod \"payload-processing-e2e-ipp-b-f826dd-66fc94df7b-vd626\" (UID: \"6d7a6ab3-2d04-41ec-acc3-df776501aa5e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f826dd-66fc94df7b-vd626" Oct 30 18:22:41.093448 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:41.093348 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9de8d55a-7a7c-4e6d-9ec4-43a1d779f30e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f826dd-675f9fd755-t8pkx\" (UID: \"9de8d55a-7a7c-4e6d-9ec4-43a1d779f30e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-675f9fd755-t8pkx" Oct 30 18:22:41.093612 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:41.093485 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7tkb\" (UniqueName: \"kubernetes.io/projected/9de8d55a-7a7c-4e6d-9ec4-43a1d779f30e-kube-api-access-q7tkb\") pod \"payload-pre-processing-e2e-ipp-b-f826dd-675f9fd755-t8pkx\" (UID: \"9de8d55a-7a7c-4e6d-9ec4-43a1d779f30e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-675f9fd755-t8pkx" Oct 30 18:22:41.094273 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:41.094251 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9de8d55a-7a7c-4e6d-9ec4-43a1d779f30e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f826dd-675f9fd755-t8pkx\" (UID: \"9de8d55a-7a7c-4e6d-9ec4-43a1d779f30e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-675f9fd755-t8pkx" Oct 30 18:22:41.113043 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:41.113009 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7tkb\" (UniqueName: \"kubernetes.io/projected/9de8d55a-7a7c-4e6d-9ec4-43a1d779f30e-kube-api-access-q7tkb\") pod \"payload-pre-processing-e2e-ipp-b-f826dd-675f9fd755-t8pkx\" (UID: \"9de8d55a-7a7c-4e6d-9ec4-43a1d779f30e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-675f9fd755-t8pkx" Oct 30 18:22:41.165317 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:41.165280 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f826dd-66fc94df7b-vd626" Oct 30 18:22:41.231475 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:41.231441 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-675f9fd755-t8pkx" Oct 30 18:22:41.480254 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:41.480226 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f826dd-66fc94df7b-vd626"] Oct 30 18:22:41.481181 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:22:41.481123 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d7a6ab3_2d04_41ec_acc3_df776501aa5e.slice/crio-f28da4d374eddaa54a2e279b9d37b181d9601675aedc228c32cb4dc357930a59 WatchSource:0}: Error finding container f28da4d374eddaa54a2e279b9d37b181d9601675aedc228c32cb4dc357930a59: Status 404 returned error can't find the container with id f28da4d374eddaa54a2e279b9d37b181d9601675aedc228c32cb4dc357930a59 Oct 30 18:22:41.532175 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:41.524389 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f826dd-66fc94df7b-vd626" event={"ID":"6d7a6ab3-2d04-41ec-acc3-df776501aa5e","Type":"ContainerStarted","Data":"f28da4d374eddaa54a2e279b9d37b181d9601675aedc228c32cb4dc357930a59"} Oct 30 18:22:41.583328 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:22:41.583240 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9de8d55a_7a7c_4e6d_9ec4_43a1d779f30e.slice/crio-55756afdf229a126e2d4bc7ca669a00dd5fffc273a9ac4307655fe40582f5eb5 WatchSource:0}: Error finding container 55756afdf229a126e2d4bc7ca669a00dd5fffc273a9ac4307655fe40582f5eb5: Status 404 returned error can't find the container with id 55756afdf229a126e2d4bc7ca669a00dd5fffc273a9ac4307655fe40582f5eb5 Oct 30 18:22:41.583328 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:41.583254 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-675f9fd755-t8pkx"] Oct 30 18:22:42.538288 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:42.538249 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f826dd-66fc94df7b-vd626" event={"ID":"6d7a6ab3-2d04-41ec-acc3-df776501aa5e","Type":"ContainerStarted","Data":"f48281dac0ece7e15609356abc48e21c3bc35a5a76dbc031707142b299ae5638"} Oct 30 18:22:42.538810 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:42.538400 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f826dd-66fc94df7b-vd626" Oct 30 18:22:42.540729 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:42.540699 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-675f9fd755-t8pkx" event={"ID":"9de8d55a-7a7c-4e6d-9ec4-43a1d779f30e","Type":"ContainerStarted","Data":"c1ee8edb689eb9ca12f5cebde8e8b080b4bba4fec1e67a722344a2734c370287"} Oct 30 18:22:42.540866 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:42.540752 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-675f9fd755-t8pkx" event={"ID":"9de8d55a-7a7c-4e6d-9ec4-43a1d779f30e","Type":"ContainerStarted","Data":"55756afdf229a126e2d4bc7ca669a00dd5fffc273a9ac4307655fe40582f5eb5"} Oct 30 18:22:42.540866 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:42.540855 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-675f9fd755-t8pkx" Oct 30 18:22:42.581804 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:42.581751 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f826dd-66fc94df7b-vd626" podStartSLOduration=2.581735969 podStartE2EDuration="2.581735969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:22:42.57668285 +0000 UTC m=+1459.666745196" watchObservedRunningTime="2026-10-30 18:22:42.581735969 +0000 UTC m=+1459.671798308" Oct 30 18:22:42.616810 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:42.616729 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-675f9fd755-t8pkx" podStartSLOduration=2.616713776 podStartE2EDuration="2.616713776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:22:42.614815234 +0000 UTC m=+1459.704877576" watchObservedRunningTime="2026-10-30 18:22:42.616713776 +0000 UTC m=+1459.706776112" Oct 30 18:22:44.560582 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:44.560486 4442 generic.go:356] "Generic (PLEG): container finished" podID="22d32885-e0c5-475e-b497-d10d09015d22" containerID="77ad92f64f90785547d2304549fc0b52597b278094c6e0174453e4fab2ed6f21" exitCode=0 Oct 30 18:22:44.560582 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:44.560562 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" event={"ID":"22d32885-e0c5-475e-b497-d10d09015d22","Type":"ContainerDied","Data":"77ad92f64f90785547d2304549fc0b52597b278094c6e0174453e4fab2ed6f21"} Oct 30 18:22:44.561025 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:44.561005 4442 scope.go:122] "RemoveContainer" containerID="77ad92f64f90785547d2304549fc0b52597b278094c6e0174453e4fab2ed6f21" Oct 30 18:22:44.562785 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:44.562752 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:22:44.562905 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:44.562842 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:22:44.562905 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:44.562892 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:22:45.569930 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:45.569892 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" event={"ID":"22d32885-e0c5-475e-b497-d10d09015d22","Type":"ContainerStarted","Data":"2101fe5edac5c2232d74426cce15c60f246e9922cca1f16dacf8c95827c658f6"} Oct 30 18:22:46.538160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:46.536682 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" Oct 30 18:22:46.548162 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:46.544263 4442 patch_prober.go:29] interesting pod/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 18:22:46.548162 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:46.544318 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" podUID="22d32885-e0c5-475e-b497-d10d09015d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 18:22:47.190569 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:47.190530 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-76135e-759fd4f54d-jcj6v"] Oct 30 18:22:47.228160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:47.223184 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-6b67694855-jkbgw"] Oct 30 18:22:47.531898 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:47.528576 4442 patch_prober.go:29] interesting pod/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 18:22:47.531898 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:47.528635 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" podUID="22d32885-e0c5-475e-b497-d10d09015d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 18:22:47.704467 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:22:47.704428 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:22:48.432696 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:48.430616 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" Oct 30 18:22:48.444164 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:48.442347 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-0ac6bb-86c4fbf5cb-x5t46" Oct 30 18:22:48.452389 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:48.452359 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0ac6bb-67fc4d5447-5n5xd" Oct 30 18:22:48.528226 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:48.528182 4442 patch_prober.go:29] interesting pod/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 18:22:48.528484 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:48.528455 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" podUID="22d32885-e0c5-475e-b497-d10d09015d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 18:22:49.545159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:49.543541 4442 patch_prober.go:29] interesting pod/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 18:22:49.545159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:49.543604 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" podUID="22d32885-e0c5-475e-b497-d10d09015d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 18:22:50.536564 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:50.536525 4442 patch_prober.go:29] interesting pod/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 18:22:50.536854 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:50.536826 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" podUID="22d32885-e0c5-475e-b497-d10d09015d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 18:22:51.530222 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:51.530190 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" Oct 30 18:22:51.531518 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:51.531497 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" Oct 30 18:22:52.191383 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:52.191329 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-76135e-759fd4f54d-jcj6v" podUID="528c3010-3850-4dad-9695-9e05bacd97b8" containerName="payload-processing" containerID="cri-o://93168e7e5566feb6d2fce5a6ae1bbdc31e8a9c9faab8f0564bf4900a96c29919" gracePeriod=25 Oct 30 18:22:52.223872 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:52.223702 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-6b67694855-jkbgw" podUID="b4115d55-b178-4ab3-89d2-f2bc728b8183" containerName="payload-pre-processing" containerID="cri-o://247f87d4a309f2b2ff251f1edaf22bfe795b1eaa5bf96653b8b2748f460a55cc" gracePeriod=25 Oct 30 18:22:52.712783 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:52.712750 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-76135e-759fd4f54d-jcj6v" Oct 30 18:22:52.749499 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:52.749374 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-6b67694855-jkbgw" Oct 30 18:22:52.794504 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:52.794475 4442 generic.go:356] "Generic (PLEG): container finished" podID="b4115d55-b178-4ab3-89d2-f2bc728b8183" containerID="247f87d4a309f2b2ff251f1edaf22bfe795b1eaa5bf96653b8b2748f460a55cc" exitCode=0 Oct 30 18:22:52.794681 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:52.794523 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-6b67694855-jkbgw" event={"ID":"b4115d55-b178-4ab3-89d2-f2bc728b8183","Type":"ContainerDied","Data":"247f87d4a309f2b2ff251f1edaf22bfe795b1eaa5bf96653b8b2748f460a55cc"} Oct 30 18:22:52.794681 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:52.794579 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-6b67694855-jkbgw" event={"ID":"b4115d55-b178-4ab3-89d2-f2bc728b8183","Type":"ContainerDied","Data":"94c1299e8be12ba613adc22dd0551aa636123cc5885a8fb7f675bfabeff228a4"} Oct 30 18:22:52.794681 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:52.794605 4442 scope.go:122] "RemoveContainer" containerID="247f87d4a309f2b2ff251f1edaf22bfe795b1eaa5bf96653b8b2748f460a55cc" Oct 30 18:22:52.794681 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:52.794531 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-6b67694855-jkbgw" Oct 30 18:22:52.796652 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:52.796629 4442 generic.go:356] "Generic (PLEG): container finished" podID="528c3010-3850-4dad-9695-9e05bacd97b8" containerID="93168e7e5566feb6d2fce5a6ae1bbdc31e8a9c9faab8f0564bf4900a96c29919" exitCode=0 Oct 30 18:22:52.796887 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:52.796663 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-76135e-759fd4f54d-jcj6v" Oct 30 18:22:52.797020 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:52.796664 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-76135e-759fd4f54d-jcj6v" event={"ID":"528c3010-3850-4dad-9695-9e05bacd97b8","Type":"ContainerDied","Data":"93168e7e5566feb6d2fce5a6ae1bbdc31e8a9c9faab8f0564bf4900a96c29919"} Oct 30 18:22:52.797091 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:52.797050 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-76135e-759fd4f54d-jcj6v" event={"ID":"528c3010-3850-4dad-9695-9e05bacd97b8","Type":"ContainerDied","Data":"f125e6a7bd63d9bb091dfab35bb5f830f30df30cd1fbcfea647be7588af08f6e"} Oct 30 18:22:52.798573 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:52.798547 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4115d55-b178-4ab3-89d2-f2bc728b8183-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4115d55-b178-4ab3-89d2-f2bc728b8183-plugins-config-volume\") pod \"b4115d55-b178-4ab3-89d2-f2bc728b8183\" (UID: \"b4115d55-b178-4ab3-89d2-f2bc728b8183\") " Oct 30 18:22:52.798786 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:52.798765 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/528c3010-3850-4dad-9695-9e05bacd97b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/528c3010-3850-4dad-9695-9e05bacd97b8-plugins-config-volume\") pod \"528c3010-3850-4dad-9695-9e05bacd97b8\" (UID: \"528c3010-3850-4dad-9695-9e05bacd97b8\") " Oct 30 18:22:52.798941 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:52.798924 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4115d55-b178-4ab3-89d2-f2bc728b8183-kube-api-access-b6g48\" (UniqueName: \"kubernetes.io/projected/b4115d55-b178-4ab3-89d2-f2bc728b8183-kube-api-access-b6g48\") pod \"b4115d55-b178-4ab3-89d2-f2bc728b8183\" (UID: \"b4115d55-b178-4ab3-89d2-f2bc728b8183\") " Oct 30 18:22:52.799234 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:52.799214 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/528c3010-3850-4dad-9695-9e05bacd97b8-kube-api-access-7vmt2\" (UniqueName: \"kubernetes.io/projected/528c3010-3850-4dad-9695-9e05bacd97b8-kube-api-access-7vmt2\") pod \"528c3010-3850-4dad-9695-9e05bacd97b8\" (UID: \"528c3010-3850-4dad-9695-9e05bacd97b8\") " Oct 30 18:22:52.801041 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:52.801000 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4115d55-b178-4ab3-89d2-f2bc728b8183-plugins-config-volume" pod "b4115d55-b178-4ab3-89d2-f2bc728b8183" (UID: "b4115d55-b178-4ab3-89d2-f2bc728b8183"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:52.801656 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:52.801632 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/528c3010-3850-4dad-9695-9e05bacd97b8-plugins-config-volume" pod "528c3010-3850-4dad-9695-9e05bacd97b8" (UID: "528c3010-3850-4dad-9695-9e05bacd97b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:52.802498 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:52.802471 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/528c3010-3850-4dad-9695-9e05bacd97b8-kube-api-access-7vmt2" pod "528c3010-3850-4dad-9695-9e05bacd97b8" (UID: "528c3010-3850-4dad-9695-9e05bacd97b8"). InnerVolumeSpecName "kube-api-access-7vmt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:52.830110 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:52.829632 4442 scope.go:122] "RemoveContainer" containerID="247f87d4a309f2b2ff251f1edaf22bfe795b1eaa5bf96653b8b2748f460a55cc" Oct 30 18:22:52.830110 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:52.829761 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4115d55-b178-4ab3-89d2-f2bc728b8183-kube-api-access-b6g48" pod "b4115d55-b178-4ab3-89d2-f2bc728b8183" (UID: "b4115d55-b178-4ab3-89d2-f2bc728b8183"). InnerVolumeSpecName "kube-api-access-b6g48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:52.830929 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:22:52.830561 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"247f87d4a309f2b2ff251f1edaf22bfe795b1eaa5bf96653b8b2748f460a55cc\": container with ID starting with 247f87d4a309f2b2ff251f1edaf22bfe795b1eaa5bf96653b8b2748f460a55cc not found: ID does not exist" containerID="247f87d4a309f2b2ff251f1edaf22bfe795b1eaa5bf96653b8b2748f460a55cc" Oct 30 18:22:52.830929 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:52.830602 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"247f87d4a309f2b2ff251f1edaf22bfe795b1eaa5bf96653b8b2748f460a55cc"} err="failed to get container status \"247f87d4a309f2b2ff251f1edaf22bfe795b1eaa5bf96653b8b2748f460a55cc\": rpc error: code = NotFound desc = could not find container \"247f87d4a309f2b2ff251f1edaf22bfe795b1eaa5bf96653b8b2748f460a55cc\": container with ID starting with 247f87d4a309f2b2ff251f1edaf22bfe795b1eaa5bf96653b8b2748f460a55cc not found: ID does not exist" Oct 30 18:22:52.830929 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:52.830627 4442 scope.go:122] "RemoveContainer" containerID="93168e7e5566feb6d2fce5a6ae1bbdc31e8a9c9faab8f0564bf4900a96c29919" Oct 30 18:22:52.848075 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:52.848047 4442 scope.go:122] "RemoveContainer" containerID="93168e7e5566feb6d2fce5a6ae1bbdc31e8a9c9faab8f0564bf4900a96c29919" Oct 30 18:22:52.848467 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:22:52.848444 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93168e7e5566feb6d2fce5a6ae1bbdc31e8a9c9faab8f0564bf4900a96c29919\": container with ID starting with 93168e7e5566feb6d2fce5a6ae1bbdc31e8a9c9faab8f0564bf4900a96c29919 not found: ID does not exist" containerID="93168e7e5566feb6d2fce5a6ae1bbdc31e8a9c9faab8f0564bf4900a96c29919" Oct 30 18:22:52.848567 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:52.848476 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93168e7e5566feb6d2fce5a6ae1bbdc31e8a9c9faab8f0564bf4900a96c29919"} err="failed to get container status \"93168e7e5566feb6d2fce5a6ae1bbdc31e8a9c9faab8f0564bf4900a96c29919\": rpc error: code = NotFound desc = could not find container \"93168e7e5566feb6d2fce5a6ae1bbdc31e8a9c9faab8f0564bf4900a96c29919\": container with ID starting with 93168e7e5566feb6d2fce5a6ae1bbdc31e8a9c9faab8f0564bf4900a96c29919 not found: ID does not exist" Oct 30 18:22:52.901261 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:52.901234 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4115d55-b178-4ab3-89d2-f2bc728b8183-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:22:52.901261 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:52.901261 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/528c3010-3850-4dad-9695-9e05bacd97b8-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:22:52.901261 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:52.901270 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6g48\" (UniqueName: \"kubernetes.io/projected/b4115d55-b178-4ab3-89d2-f2bc728b8183-kube-api-access-b6g48\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:22:52.901494 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:52.901281 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vmt2\" (UniqueName: \"kubernetes.io/projected/528c3010-3850-4dad-9695-9e05bacd97b8-kube-api-access-7vmt2\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:22:53.135735 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:53.135702 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-6b67694855-jkbgw"] Oct 30 18:22:53.146624 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:53.146596 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-6b67694855-jkbgw"] Oct 30 18:22:53.172677 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:53.172648 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-76135e-759fd4f54d-jcj6v"] Oct 30 18:22:53.176697 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:53.176674 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-76135e-759fd4f54d-jcj6v"] Oct 30 18:22:53.554801 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:53.554775 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-675f9fd755-t8pkx" Oct 30 18:22:53.554980 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:53.554848 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f826dd-66fc94df7b-vd626" Oct 30 18:22:54.106477 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:54.106442 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-784cb9f975-7grmk"] Oct 30 18:22:54.107125 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:54.107111 4442 cpu_manager.go:414] "containerMap: removing container" podUID="528c3010-3850-4dad-9695-9e05bacd97b8" containerName="payload-processing" Oct 30 18:22:54.107125 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:54.107126 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="528c3010-3850-4dad-9695-9e05bacd97b8" containerName="payload-processing" Oct 30 18:22:54.107125 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:54.107152 4442 cpu_manager.go:414] "containerMap: removing container" podUID="b4115d55-b178-4ab3-89d2-f2bc728b8183" containerName="payload-pre-processing" Oct 30 18:22:54.107125 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:54.107158 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4115d55-b178-4ab3-89d2-f2bc728b8183" containerName="payload-pre-processing" Oct 30 18:22:54.107329 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:54.107251 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="528c3010-3850-4dad-9695-9e05bacd97b8" containerName="payload-processing" Oct 30 18:22:54.107329 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:54.107258 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4115d55-b178-4ab3-89d2-f2bc728b8183" containerName="payload-pre-processing" Oct 30 18:22:54.112386 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:54.112367 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-784cb9f975-7grmk" Oct 30 18:22:54.126487 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:54.126453 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-17df15\"" Oct 30 18:22:54.128989 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:54.128966 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-784cb9f975-7grmk"] Oct 30 18:22:54.155488 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:54.155449 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-17df15-dockercfg-5r8qm\"" Oct 30 18:22:54.192670 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:54.192635 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-17df15-dbdfd998b-949th"] Oct 30 18:22:54.197082 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:54.197061 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-17df15-dbdfd998b-949th" Oct 30 18:22:54.237158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:54.222302 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7ksb\" (UniqueName: \"kubernetes.io/projected/cfb0bd5a-c194-4d65-b5ce-10989e213f54-kube-api-access-b7ksb\") pod \"payload-processing-e2e-ipp-b-17df15-dbdfd998b-949th\" (UID: \"cfb0bd5a-c194-4d65-b5ce-10989e213f54\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-17df15-dbdfd998b-949th" Oct 30 18:22:54.237158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:54.222362 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq4d2\" (UniqueName: \"kubernetes.io/projected/fe684747-79a4-4547-810e-591b9c1e6832-kube-api-access-zq4d2\") pod \"payload-pre-processing-e2e-ipp-b-17df15-784cb9f975-7grmk\" (UID: \"fe684747-79a4-4547-810e-591b9c1e6832\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-784cb9f975-7grmk" Oct 30 18:22:54.237158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:54.222590 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe684747-79a4-4547-810e-591b9c1e6832-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-17df15-784cb9f975-7grmk\" (UID: \"fe684747-79a4-4547-810e-591b9c1e6832\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-784cb9f975-7grmk" Oct 30 18:22:54.237158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:54.222672 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfb0bd5a-c194-4d65-b5ce-10989e213f54-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-17df15-dbdfd998b-949th\" (UID: \"cfb0bd5a-c194-4d65-b5ce-10989e213f54\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-17df15-dbdfd998b-949th" Oct 30 18:22:54.237158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:54.223112 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-17df15-dbdfd998b-949th"] Oct 30 18:22:54.360315 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:54.356157 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7ksb\" (UniqueName: \"kubernetes.io/projected/cfb0bd5a-c194-4d65-b5ce-10989e213f54-kube-api-access-b7ksb\") pod \"payload-processing-e2e-ipp-b-17df15-dbdfd998b-949th\" (UID: \"cfb0bd5a-c194-4d65-b5ce-10989e213f54\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-17df15-dbdfd998b-949th" Oct 30 18:22:54.360315 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:54.356221 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq4d2\" (UniqueName: \"kubernetes.io/projected/fe684747-79a4-4547-810e-591b9c1e6832-kube-api-access-zq4d2\") pod \"payload-pre-processing-e2e-ipp-b-17df15-784cb9f975-7grmk\" (UID: \"fe684747-79a4-4547-810e-591b9c1e6832\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-784cb9f975-7grmk" Oct 30 18:22:54.360315 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:54.356474 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe684747-79a4-4547-810e-591b9c1e6832-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-17df15-784cb9f975-7grmk\" (UID: \"fe684747-79a4-4547-810e-591b9c1e6832\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-784cb9f975-7grmk" Oct 30 18:22:54.360315 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:54.356552 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfb0bd5a-c194-4d65-b5ce-10989e213f54-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-17df15-dbdfd998b-949th\" (UID: \"cfb0bd5a-c194-4d65-b5ce-10989e213f54\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-17df15-dbdfd998b-949th" Oct 30 18:22:54.360315 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:54.357453 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfb0bd5a-c194-4d65-b5ce-10989e213f54-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-17df15-dbdfd998b-949th\" (UID: \"cfb0bd5a-c194-4d65-b5ce-10989e213f54\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-17df15-dbdfd998b-949th" Oct 30 18:22:54.399157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:54.397717 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe684747-79a4-4547-810e-591b9c1e6832-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-17df15-784cb9f975-7grmk\" (UID: \"fe684747-79a4-4547-810e-591b9c1e6832\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-784cb9f975-7grmk" Oct 30 18:22:54.464718 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:54.464654 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7ksb\" (UniqueName: \"kubernetes.io/projected/cfb0bd5a-c194-4d65-b5ce-10989e213f54-kube-api-access-b7ksb\") pod \"payload-processing-e2e-ipp-b-17df15-dbdfd998b-949th\" (UID: \"cfb0bd5a-c194-4d65-b5ce-10989e213f54\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-17df15-dbdfd998b-949th" Oct 30 18:22:54.468385 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:54.468352 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq4d2\" (UniqueName: \"kubernetes.io/projected/fe684747-79a4-4547-810e-591b9c1e6832-kube-api-access-zq4d2\") pod \"payload-pre-processing-e2e-ipp-b-17df15-784cb9f975-7grmk\" (UID: \"fe684747-79a4-4547-810e-591b9c1e6832\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-784cb9f975-7grmk" Oct 30 18:22:54.593714 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:54.593673 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-17df15-dbdfd998b-949th" Oct 30 18:22:54.728931 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:54.728893 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-784cb9f975-7grmk" Oct 30 18:22:54.831744 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:54.831704 4442 generic.go:356] "Generic (PLEG): container finished" podID="24752fc1-eb92-452c-8f20-91cd12fc0ab0" containerID="9865e0e5d91816c3ab5b9b74e44ae2731cadc4e7989ef3848aee014bf7e171b2" exitCode=1 Oct 30 18:22:54.831887 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:54.831768 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dv6zs" event={"ID":"24752fc1-eb92-452c-8f20-91cd12fc0ab0","Type":"ContainerDied","Data":"9865e0e5d91816c3ab5b9b74e44ae2731cadc4e7989ef3848aee014bf7e171b2"} Oct 30 18:22:54.832339 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:54.832320 4442 scope.go:122] "RemoveContainer" containerID="9865e0e5d91816c3ab5b9b74e44ae2731cadc4e7989ef3848aee014bf7e171b2" Oct 30 18:22:54.834069 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:54.834046 4442 generic.go:356] "Generic (PLEG): container finished" podID="ec2b6bb1-c8d6-43ee-8234-0556646382a7" containerID="66eb816062a0834ef2fb3583db048e258ac473db243183cf4cc57039b52636c8" exitCode=0 Oct 30 18:22:54.834248 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:54.834121 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-h2877" event={"ID":"ec2b6bb1-c8d6-43ee-8234-0556646382a7","Type":"ContainerDied","Data":"66eb816062a0834ef2fb3583db048e258ac473db243183cf4cc57039b52636c8"} Oct 30 18:22:54.834511 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:54.834494 4442 scope.go:122] "RemoveContainer" containerID="66eb816062a0834ef2fb3583db048e258ac473db243183cf4cc57039b52636c8" Oct 30 18:22:54.928747 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:54.928153 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-17df15-dbdfd998b-949th"] Oct 30 18:22:55.052165 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:55.046837 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-784cb9f975-7grmk"] Oct 30 18:22:55.072229 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:55.072204 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="528c3010-3850-4dad-9695-9e05bacd97b8" path="/var/lib/kubelet/pods/528c3010-3850-4dad-9695-9e05bacd97b8/volumes" Oct 30 18:22:55.072617 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:55.072602 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4115d55-b178-4ab3-89d2-f2bc728b8183" path="/var/lib/kubelet/pods/b4115d55-b178-4ab3-89d2-f2bc728b8183/volumes" Oct 30 18:22:55.506028 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:55.506003 4442 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dv6zs" Oct 30 18:22:55.844192 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:55.844081 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dv6zs" event={"ID":"24752fc1-eb92-452c-8f20-91cd12fc0ab0","Type":"ContainerStarted","Data":"2543b4921bfa9a27a169c7541e3b808638a663b27cf1820c96df429489194838"} Oct 30 18:22:55.844354 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:55.844194 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dv6zs" Oct 30 18:22:55.846549 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:55.846526 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-h2877" event={"ID":"ec2b6bb1-c8d6-43ee-8234-0556646382a7","Type":"ContainerStarted","Data":"eb12d3bb6cb29c0d610e7348e3ea1c40a02b378744e5a036b6516449b447897a"} Oct 30 18:22:55.846762 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:55.846744 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-h2877" Oct 30 18:22:55.848280 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:55.848255 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-17df15-dbdfd998b-949th" event={"ID":"cfb0bd5a-c194-4d65-b5ce-10989e213f54","Type":"ContainerStarted","Data":"eb669969a213f198f76da052286fbb1f8e148fa6d5f34758834b6800715d5407"} Oct 30 18:22:55.848398 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:55.848297 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-17df15-dbdfd998b-949th" Oct 30 18:22:55.848398 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:55.848312 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-17df15-dbdfd998b-949th" event={"ID":"cfb0bd5a-c194-4d65-b5ce-10989e213f54","Type":"ContainerStarted","Data":"e57982dba32716de9ac8ca2840461a30e6d84a97da9124992d7e6e7df86d716b"} Oct 30 18:22:55.849909 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:55.849888 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-784cb9f975-7grmk" event={"ID":"fe684747-79a4-4547-810e-591b9c1e6832","Type":"ContainerStarted","Data":"dd39461f20f67b08d90e0c92824110bbff71083b03db2abdae55e3faefdaf0dc"} Oct 30 18:22:55.850025 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:55.849923 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-784cb9f975-7grmk" event={"ID":"fe684747-79a4-4547-810e-591b9c1e6832","Type":"ContainerStarted","Data":"fa2455cd21d984cce214ad2fd67544435d1b37f547037e0036932cf6dd8f34e8"} Oct 30 18:22:55.850025 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:55.849980 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-784cb9f975-7grmk" Oct 30 18:22:55.851571 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:55.851548 4442 generic.go:356] "Generic (PLEG): container finished" podID="dd0fb59a-f6bb-43fd-95e4-6f740ace1c6e" containerID="43fc76a3115ada38135ee4c071d80fab45528a63640dcfaf0d69cb587c357d20" exitCode=1 Oct 30 18:22:55.851667 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:55.851601 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nxg2g" event={"ID":"dd0fb59a-f6bb-43fd-95e4-6f740ace1c6e","Type":"ContainerDied","Data":"43fc76a3115ada38135ee4c071d80fab45528a63640dcfaf0d69cb587c357d20"} Oct 30 18:22:55.851922 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:55.851909 4442 scope.go:122] "RemoveContainer" containerID="43fc76a3115ada38135ee4c071d80fab45528a63640dcfaf0d69cb587c357d20" Oct 30 18:22:55.934127 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:55.934062 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-784cb9f975-7grmk" podStartSLOduration=1.934040339 podStartE2EDuration="1.934040339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:22:55.929871893 +0000 UTC m=+1473.019934232" watchObservedRunningTime="2026-10-30 18:22:55.934040339 +0000 UTC m=+1473.024102678" Oct 30 18:22:56.015627 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:56.015573 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-17df15-dbdfd998b-949th" podStartSLOduration=2.015558672 podStartE2EDuration="2.015558672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:22:56.012110177 +0000 UTC m=+1473.102172520" watchObservedRunningTime="2026-10-30 18:22:56.015558672 +0000 UTC m=+1473.105621032" Oct 30 18:22:56.861088 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:56.861053 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nxg2g" event={"ID":"dd0fb59a-f6bb-43fd-95e4-6f740ace1c6e","Type":"ContainerStarted","Data":"a0ff2a596ffdda1ce58242ac6080be913f28b4b24b7ba0ab683868d0a02f52f6"} Oct 30 18:22:56.861601 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:56.861371 4442 status_manager.go:462] "Container readiness changed for unknown container" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nxg2g" containerID="cri-o://43fc76a3115ada38135ee4c071d80fab45528a63640dcfaf0d69cb587c357d20" Oct 30 18:22:56.861601 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:56.861400 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nxg2g" Oct 30 18:22:57.868480 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:57.868451 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nxg2g" Oct 30 18:22:59.843987 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:59.843960 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0ac6bb-7f54cddc4c-2vs6c"] Oct 30 18:22:59.848831 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:59.848812 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0ac6bb-7f54cddc4c-2vs6c" Oct 30 18:22:59.865359 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:59.865322 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0ac6bb-7f54cddc4c-2vs6c"] Oct 30 18:22:59.872602 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:59.872570 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-0ac6bb-serving-cert\"" Oct 30 18:22:59.947086 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:59.947051 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ebd21aa-152c-4087-b33b-461678c392a6-metrics-certs\") pod \"maas-api-e2e-api-b-0ac6bb-7f54cddc4c-2vs6c\" (UID: \"1ebd21aa-152c-4087-b33b-461678c392a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0ac6bb-7f54cddc4c-2vs6c" Oct 30 18:22:59.947370 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:59.947350 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ebd21aa-152c-4087-b33b-461678c392a6-maas-api-tls\") pod \"maas-api-e2e-api-b-0ac6bb-7f54cddc4c-2vs6c\" (UID: \"1ebd21aa-152c-4087-b33b-461678c392a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0ac6bb-7f54cddc4c-2vs6c" Oct 30 18:22:59.947417 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:59.947391 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qptp\" (UniqueName: \"kubernetes.io/projected/1ebd21aa-152c-4087-b33b-461678c392a6-kube-api-access-4qptp\") pod \"maas-api-e2e-api-b-0ac6bb-7f54cddc4c-2vs6c\" (UID: \"1ebd21aa-152c-4087-b33b-461678c392a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0ac6bb-7f54cddc4c-2vs6c" Oct 30 18:22:59.947520 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:22:59.947505 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ebd21aa-152c-4087-b33b-461678c392a6-openshift-service-ca\") pod \"maas-api-e2e-api-b-0ac6bb-7f54cddc4c-2vs6c\" (UID: \"1ebd21aa-152c-4087-b33b-461678c392a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0ac6bb-7f54cddc4c-2vs6c" Oct 30 18:23:00.048731 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:00.048696 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ebd21aa-152c-4087-b33b-461678c392a6-maas-api-tls\") pod \"maas-api-e2e-api-b-0ac6bb-7f54cddc4c-2vs6c\" (UID: \"1ebd21aa-152c-4087-b33b-461678c392a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0ac6bb-7f54cddc4c-2vs6c" Oct 30 18:23:00.048897 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:00.048740 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qptp\" (UniqueName: \"kubernetes.io/projected/1ebd21aa-152c-4087-b33b-461678c392a6-kube-api-access-4qptp\") pod \"maas-api-e2e-api-b-0ac6bb-7f54cddc4c-2vs6c\" (UID: \"1ebd21aa-152c-4087-b33b-461678c392a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0ac6bb-7f54cddc4c-2vs6c" Oct 30 18:23:00.048897 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:00.048781 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ebd21aa-152c-4087-b33b-461678c392a6-openshift-service-ca\") pod \"maas-api-e2e-api-b-0ac6bb-7f54cddc4c-2vs6c\" (UID: \"1ebd21aa-152c-4087-b33b-461678c392a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0ac6bb-7f54cddc4c-2vs6c" Oct 30 18:23:00.048897 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:00.048805 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ebd21aa-152c-4087-b33b-461678c392a6-metrics-certs\") pod \"maas-api-e2e-api-b-0ac6bb-7f54cddc4c-2vs6c\" (UID: \"1ebd21aa-152c-4087-b33b-461678c392a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0ac6bb-7f54cddc4c-2vs6c" Oct 30 18:23:00.049567 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:00.049493 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ebd21aa-152c-4087-b33b-461678c392a6-openshift-service-ca\") pod \"maas-api-e2e-api-b-0ac6bb-7f54cddc4c-2vs6c\" (UID: \"1ebd21aa-152c-4087-b33b-461678c392a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0ac6bb-7f54cddc4c-2vs6c" Oct 30 18:23:00.051395 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:00.051368 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ebd21aa-152c-4087-b33b-461678c392a6-maas-api-tls\") pod \"maas-api-e2e-api-b-0ac6bb-7f54cddc4c-2vs6c\" (UID: \"1ebd21aa-152c-4087-b33b-461678c392a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0ac6bb-7f54cddc4c-2vs6c" Oct 30 18:23:00.051522 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:00.051502 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ebd21aa-152c-4087-b33b-461678c392a6-metrics-certs\") pod \"maas-api-e2e-api-b-0ac6bb-7f54cddc4c-2vs6c\" (UID: \"1ebd21aa-152c-4087-b33b-461678c392a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0ac6bb-7f54cddc4c-2vs6c" Oct 30 18:23:00.068909 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:00.068882 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qptp\" (UniqueName: \"kubernetes.io/projected/1ebd21aa-152c-4087-b33b-461678c392a6-kube-api-access-4qptp\") pod \"maas-api-e2e-api-b-0ac6bb-7f54cddc4c-2vs6c\" (UID: \"1ebd21aa-152c-4087-b33b-461678c392a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0ac6bb-7f54cddc4c-2vs6c" Oct 30 18:23:00.174600 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:00.174286 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0ac6bb-7f54cddc4c-2vs6c" Oct 30 18:23:00.491675 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:00.491647 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0ac6bb-7f54cddc4c-2vs6c"] Oct 30 18:23:00.493325 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:23:00.493297 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ebd21aa_152c_4087_b33b_461678c392a6.slice/crio-d0ded2b44fd4a3e35e9b83a445d9ac5076bdfdeaeaadb5a0dce7c048fe95f414 WatchSource:0}: Error finding container d0ded2b44fd4a3e35e9b83a445d9ac5076bdfdeaeaadb5a0dce7c048fe95f414: Status 404 returned error can't find the container with id d0ded2b44fd4a3e35e9b83a445d9ac5076bdfdeaeaadb5a0dce7c048fe95f414 Oct 30 18:23:00.896297 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:00.896253 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0ac6bb-7f54cddc4c-2vs6c" event={"ID":"1ebd21aa-152c-4087-b33b-461678c392a6","Type":"ContainerStarted","Data":"d0ded2b44fd4a3e35e9b83a445d9ac5076bdfdeaeaadb5a0dce7c048fe95f414"} Oct 30 18:23:02.054166 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:02.020681 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0ac6bb-7f54cddc4c-2vs6c" event={"ID":"1ebd21aa-152c-4087-b33b-461678c392a6","Type":"ContainerStarted","Data":"777966f70e8350f1fb48e6c73b097809769a7dc72380a155102be2429459c5dd"} Oct 30 18:23:02.054166 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:02.021235 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0ac6bb-7f54cddc4c-2vs6c" Oct 30 18:23:02.097983 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:02.097912 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0ac6bb-7f54cddc4c-2vs6c" podStartSLOduration=2.656988288 podStartE2EDuration="3.09788965s" totalImagesPullingTime="440.901362ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:23:02.093604294 +0000 UTC m=+1479.183666634" watchObservedRunningTime="2026-10-30 18:23:02.09788965 +0000 UTC m=+1479.187951990" Oct 30 18:23:06.864204 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:06.864114 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dv6zs" Oct 30 18:23:06.864731 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:06.864234 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-h2877" Oct 30 18:23:06.865202 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:06.865178 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-784cb9f975-7grmk" Oct 30 18:23:06.865562 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:06.865545 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-17df15-dbdfd998b-949th" Oct 30 18:23:08.015367 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:08.015338 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0ac6bb-7f54cddc4c-2vs6c" Oct 30 18:23:10.906923 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:10.906885 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-779fd9cbd7-4fqtg"] Oct 30 18:23:10.920748 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:10.920709 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-779fd9cbd7-4fqtg" Oct 30 18:23:10.935185 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:10.935097 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-779fd9cbd7-4fqtg"] Oct 30 18:23:10.935563 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:10.935530 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-76135e-dockercfg-kgd2n\"" Oct 30 18:23:10.941211 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:10.938767 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-76135e-5d8b8684b6-g4jr8"] Oct 30 18:23:10.943702 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:10.943554 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-76135e-5d8b8684b6-g4jr8" Oct 30 18:23:10.944459 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:10.944438 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-76135e\"" Oct 30 18:23:10.947767 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:10.947741 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-76135e-5d8b8684b6-g4jr8"] Oct 30 18:23:11.014570 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:11.014537 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c344a53-a70f-416d-b9c0-0e6156517df9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-76135e-5d8b8684b6-g4jr8\" (UID: \"6c344a53-a70f-416d-b9c0-0e6156517df9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-76135e-5d8b8684b6-g4jr8" Oct 30 18:23:11.014747 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:11.014614 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqj25\" (UniqueName: \"kubernetes.io/projected/4795553f-070d-48df-87c1-db573810c49f-kube-api-access-jqj25\") pod \"payload-pre-processing-e2e-ipp-b-76135e-779fd9cbd7-4fqtg\" (UID: \"4795553f-070d-48df-87c1-db573810c49f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-779fd9cbd7-4fqtg" Oct 30 18:23:11.014889 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:11.014852 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wskjc\" (UniqueName: \"kubernetes.io/projected/6c344a53-a70f-416d-b9c0-0e6156517df9-kube-api-access-wskjc\") pod \"payload-processing-e2e-ipp-b-76135e-5d8b8684b6-g4jr8\" (UID: \"6c344a53-a70f-416d-b9c0-0e6156517df9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-76135e-5d8b8684b6-g4jr8" Oct 30 18:23:11.015013 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:11.014903 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4795553f-070d-48df-87c1-db573810c49f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-76135e-779fd9cbd7-4fqtg\" (UID: \"4795553f-070d-48df-87c1-db573810c49f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-779fd9cbd7-4fqtg" Oct 30 18:23:11.115780 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:11.115749 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c344a53-a70f-416d-b9c0-0e6156517df9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-76135e-5d8b8684b6-g4jr8\" (UID: \"6c344a53-a70f-416d-b9c0-0e6156517df9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-76135e-5d8b8684b6-g4jr8" Oct 30 18:23:11.116032 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:11.115811 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqj25\" (UniqueName: \"kubernetes.io/projected/4795553f-070d-48df-87c1-db573810c49f-kube-api-access-jqj25\") pod \"payload-pre-processing-e2e-ipp-b-76135e-779fd9cbd7-4fqtg\" (UID: \"4795553f-070d-48df-87c1-db573810c49f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-779fd9cbd7-4fqtg" Oct 30 18:23:11.116032 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:11.115995 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wskjc\" (UniqueName: \"kubernetes.io/projected/6c344a53-a70f-416d-b9c0-0e6156517df9-kube-api-access-wskjc\") pod \"payload-processing-e2e-ipp-b-76135e-5d8b8684b6-g4jr8\" (UID: \"6c344a53-a70f-416d-b9c0-0e6156517df9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-76135e-5d8b8684b6-g4jr8" Oct 30 18:23:11.116032 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:11.116027 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4795553f-070d-48df-87c1-db573810c49f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-76135e-779fd9cbd7-4fqtg\" (UID: \"4795553f-070d-48df-87c1-db573810c49f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-779fd9cbd7-4fqtg" Oct 30 18:23:11.116644 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:11.116617 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c344a53-a70f-416d-b9c0-0e6156517df9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-76135e-5d8b8684b6-g4jr8\" (UID: \"6c344a53-a70f-416d-b9c0-0e6156517df9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-76135e-5d8b8684b6-g4jr8" Oct 30 18:23:11.116748 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:11.116718 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4795553f-070d-48df-87c1-db573810c49f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-76135e-779fd9cbd7-4fqtg\" (UID: \"4795553f-070d-48df-87c1-db573810c49f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-779fd9cbd7-4fqtg" Oct 30 18:23:11.134912 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:11.134887 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqj25\" (UniqueName: \"kubernetes.io/projected/4795553f-070d-48df-87c1-db573810c49f-kube-api-access-jqj25\") pod \"payload-pre-processing-e2e-ipp-b-76135e-779fd9cbd7-4fqtg\" (UID: \"4795553f-070d-48df-87c1-db573810c49f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-779fd9cbd7-4fqtg" Oct 30 18:23:11.134912 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:11.134901 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wskjc\" (UniqueName: \"kubernetes.io/projected/6c344a53-a70f-416d-b9c0-0e6156517df9-kube-api-access-wskjc\") pod \"payload-processing-e2e-ipp-b-76135e-5d8b8684b6-g4jr8\" (UID: \"6c344a53-a70f-416d-b9c0-0e6156517df9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-76135e-5d8b8684b6-g4jr8" Oct 30 18:23:11.273639 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:11.273546 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-779fd9cbd7-4fqtg" Oct 30 18:23:11.280910 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:11.280885 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-76135e-5d8b8684b6-g4jr8" Oct 30 18:23:11.587388 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:11.587357 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-779fd9cbd7-4fqtg"] Oct 30 18:23:11.591322 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:23:11.589011 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4795553f_070d_48df_87c1_db573810c49f.slice/crio-9d2ef7eade479a90968cb9c68ff06f8ecf63881b680943e6414a1be1723e5e40 WatchSource:0}: Error finding container 9d2ef7eade479a90968cb9c68ff06f8ecf63881b680943e6414a1be1723e5e40: Status 404 returned error can't find the container with id 9d2ef7eade479a90968cb9c68ff06f8ecf63881b680943e6414a1be1723e5e40 Oct 30 18:23:11.654868 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:23:11.654838 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c344a53_a70f_416d_b9c0_0e6156517df9.slice/crio-5957c9ee2688f219664097e17bad31e6171dc93dd1d404bbef000e709ebd9282 WatchSource:0}: Error finding container 5957c9ee2688f219664097e17bad31e6171dc93dd1d404bbef000e709ebd9282: Status 404 returned error can't find the container with id 5957c9ee2688f219664097e17bad31e6171dc93dd1d404bbef000e709ebd9282 Oct 30 18:23:11.668684 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:11.668653 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-76135e-5d8b8684b6-g4jr8"] Oct 30 18:23:12.098180 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:12.096039 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-76135e-5d8b8684b6-g4jr8" event={"ID":"6c344a53-a70f-416d-b9c0-0e6156517df9","Type":"ContainerStarted","Data":"10452be3e51d807b3b06b98c99556728d92c41437f4ae2d4b2093b354a0839b7"} Oct 30 18:23:12.098180 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:12.096111 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-76135e-5d8b8684b6-g4jr8" Oct 30 18:23:12.098180 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:12.096151 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-76135e-5d8b8684b6-g4jr8" event={"ID":"6c344a53-a70f-416d-b9c0-0e6156517df9","Type":"ContainerStarted","Data":"5957c9ee2688f219664097e17bad31e6171dc93dd1d404bbef000e709ebd9282"} Oct 30 18:23:12.098180 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:12.097865 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-779fd9cbd7-4fqtg" event={"ID":"4795553f-070d-48df-87c1-db573810c49f","Type":"ContainerStarted","Data":"b253673626709c557f82295e22ec21f30949edbf3374f8606ddda1135c941b05"} Oct 30 18:23:12.098180 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:12.097904 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-779fd9cbd7-4fqtg" event={"ID":"4795553f-070d-48df-87c1-db573810c49f","Type":"ContainerStarted","Data":"9d2ef7eade479a90968cb9c68ff06f8ecf63881b680943e6414a1be1723e5e40"} Oct 30 18:23:12.098180 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:12.097931 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-779fd9cbd7-4fqtg" Oct 30 18:23:12.143487 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:12.143424 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-76135e-5d8b8684b6-g4jr8" podStartSLOduration=2.143400061 podStartE2EDuration="2.143400061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:23:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:23:12.137559183 +0000 UTC m=+1489.227621522" watchObservedRunningTime="2026-10-30 18:23:12.143400061 +0000 UTC m=+1489.233462392" Oct 30 18:23:12.182217 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:12.182167 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-779fd9cbd7-4fqtg" podStartSLOduration=2.182122847 podStartE2EDuration="2.182122847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:23:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:23:12.176956119 +0000 UTC m=+1489.267018472" watchObservedRunningTime="2026-10-30 18:23:12.182122847 +0000 UTC m=+1489.272185186" Oct 30 18:23:17.566563 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:23:17.566525 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:23:18.243437 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:18.243405 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a4f3f0-7c77897bf9-7mxbc"] Oct 30 18:23:18.249433 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:18.249408 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a4f3f0-7c77897bf9-7mxbc" Oct 30 18:23:18.274488 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:18.272424 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a4f3f0-7c77897bf9-7mxbc"] Oct 30 18:23:18.276385 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:18.276359 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-a4f3f0\"" Oct 30 18:23:18.467757 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:18.467718 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a4f3f0-594b877768-ctrsw"] Oct 30 18:23:18.520211 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:18.520093 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcv8k\" (UniqueName: \"kubernetes.io/projected/f613d06d-c822-4949-9f01-73d99044a3a5-kube-api-access-gcv8k\") pod \"payload-pre-processing-e2e-praxis-a-a4f3f0-7c77897bf9-7mxbc\" (UID: \"f613d06d-c822-4949-9f01-73d99044a3a5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a4f3f0-7c77897bf9-7mxbc" Oct 30 18:23:18.533251 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:18.533165 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f613d06d-c822-4949-9f01-73d99044a3a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-a4f3f0-7c77897bf9-7mxbc\" (UID: \"f613d06d-c822-4949-9f01-73d99044a3a5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a4f3f0-7c77897bf9-7mxbc" Oct 30 18:23:18.571607 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:18.571480 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a4f3f0-594b877768-ctrsw" Oct 30 18:23:18.581791 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:18.581752 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a4f3f0-594b877768-ctrsw"] Oct 30 18:23:18.592270 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:18.592241 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-a4f3f0-dockercfg-cwv8m\"" Oct 30 18:23:18.658216 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:18.658182 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2816cb76-b775-4ee2-8fe1-063879e21902-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-a4f3f0-594b877768-ctrsw\" (UID: \"2816cb76-b775-4ee2-8fe1-063879e21902\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a4f3f0-594b877768-ctrsw" Oct 30 18:23:18.658391 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:18.658243 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f613d06d-c822-4949-9f01-73d99044a3a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-a4f3f0-7c77897bf9-7mxbc\" (UID: \"f613d06d-c822-4949-9f01-73d99044a3a5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a4f3f0-7c77897bf9-7mxbc" Oct 30 18:23:18.658391 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:18.658328 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcv8k\" (UniqueName: \"kubernetes.io/projected/f613d06d-c822-4949-9f01-73d99044a3a5-kube-api-access-gcv8k\") pod \"payload-pre-processing-e2e-praxis-a-a4f3f0-7c77897bf9-7mxbc\" (UID: \"f613d06d-c822-4949-9f01-73d99044a3a5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a4f3f0-7c77897bf9-7mxbc" Oct 30 18:23:18.658508 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:18.658389 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdkxs\" (UniqueName: \"kubernetes.io/projected/2816cb76-b775-4ee2-8fe1-063879e21902-kube-api-access-fdkxs\") pod \"payload-processing-e2e-praxis-a-a4f3f0-594b877768-ctrsw\" (UID: \"2816cb76-b775-4ee2-8fe1-063879e21902\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a4f3f0-594b877768-ctrsw" Oct 30 18:23:18.659039 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:18.659017 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f613d06d-c822-4949-9f01-73d99044a3a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-a4f3f0-7c77897bf9-7mxbc\" (UID: \"f613d06d-c822-4949-9f01-73d99044a3a5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a4f3f0-7c77897bf9-7mxbc" Oct 30 18:23:18.677470 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:18.677437 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcv8k\" (UniqueName: \"kubernetes.io/projected/f613d06d-c822-4949-9f01-73d99044a3a5-kube-api-access-gcv8k\") pod \"payload-pre-processing-e2e-praxis-a-a4f3f0-7c77897bf9-7mxbc\" (UID: \"f613d06d-c822-4949-9f01-73d99044a3a5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a4f3f0-7c77897bf9-7mxbc" Oct 30 18:23:18.759916 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:18.759880 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdkxs\" (UniqueName: \"kubernetes.io/projected/2816cb76-b775-4ee2-8fe1-063879e21902-kube-api-access-fdkxs\") pod \"payload-processing-e2e-praxis-a-a4f3f0-594b877768-ctrsw\" (UID: \"2816cb76-b775-4ee2-8fe1-063879e21902\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a4f3f0-594b877768-ctrsw" Oct 30 18:23:18.760178 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:18.760068 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2816cb76-b775-4ee2-8fe1-063879e21902-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-a4f3f0-594b877768-ctrsw\" (UID: \"2816cb76-b775-4ee2-8fe1-063879e21902\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a4f3f0-594b877768-ctrsw" Oct 30 18:23:18.760781 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:18.760762 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2816cb76-b775-4ee2-8fe1-063879e21902-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-a4f3f0-594b877768-ctrsw\" (UID: \"2816cb76-b775-4ee2-8fe1-063879e21902\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a4f3f0-594b877768-ctrsw" Oct 30 18:23:18.779432 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:18.779364 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdkxs\" (UniqueName: \"kubernetes.io/projected/2816cb76-b775-4ee2-8fe1-063879e21902-kube-api-access-fdkxs\") pod \"payload-processing-e2e-praxis-a-a4f3f0-594b877768-ctrsw\" (UID: \"2816cb76-b775-4ee2-8fe1-063879e21902\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a4f3f0-594b877768-ctrsw" Oct 30 18:23:18.870109 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:18.870073 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a4f3f0-7c77897bf9-7mxbc" Oct 30 18:23:18.878176 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:18.878115 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nxg2g" Oct 30 18:23:18.896359 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:18.896336 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a4f3f0-594b877768-ctrsw" Oct 30 18:23:19.181063 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:19.181022 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a4f3f0-7c77897bf9-7mxbc"] Oct 30 18:23:19.187583 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:23:19.187558 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf613d06d_c822_4949_9f01_73d99044a3a5.slice/crio-694cea101d665717e61b2a4e20dbdde86ff5d2d24245e92200ea47d6a0cfbd7e WatchSource:0}: Error finding container 694cea101d665717e61b2a4e20dbdde86ff5d2d24245e92200ea47d6a0cfbd7e: Status 404 returned error can't find the container with id 694cea101d665717e61b2a4e20dbdde86ff5d2d24245e92200ea47d6a0cfbd7e Oct 30 18:23:19.213113 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:19.213081 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a4f3f0-7c77897bf9-7mxbc" event={"ID":"f613d06d-c822-4949-9f01-73d99044a3a5","Type":"ContainerStarted","Data":"694cea101d665717e61b2a4e20dbdde86ff5d2d24245e92200ea47d6a0cfbd7e"} Oct 30 18:23:19.235013 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:19.234989 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a4f3f0-594b877768-ctrsw"] Oct 30 18:23:19.243321 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:23:19.243292 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2816cb76_b775_4ee2_8fe1_063879e21902.slice/crio-a622a7728af9fbf32e4137e6dee912f913c2efa12134eea1b672f4ecd2ff43fa WatchSource:0}: Error finding container a622a7728af9fbf32e4137e6dee912f913c2efa12134eea1b672f4ecd2ff43fa: Status 404 returned error can't find the container with id a622a7728af9fbf32e4137e6dee912f913c2efa12134eea1b672f4ecd2ff43fa Oct 30 18:23:20.144408 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:20.144367 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-17df15-79665f588d-dxcz6"] Oct 30 18:23:20.150048 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:20.150026 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-17df15-79665f588d-dxcz6" Oct 30 18:23:20.167219 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:20.167191 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-17df15-79665f588d-dxcz6"] Oct 30 18:23:20.173318 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:20.173277 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-17df15-serving-cert\"" Oct 30 18:23:20.190098 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:20.190062 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fb3df6c-8dde-409c-985e-0bc963a96a11-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-17df15-79665f588d-dxcz6\" (UID: \"1fb3df6c-8dde-409c-985e-0bc963a96a11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-17df15-79665f588d-dxcz6" Oct 30 18:23:20.190313 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:20.190210 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fb3df6c-8dde-409c-985e-0bc963a96a11-metrics-certs\") pod \"maas-api-e2e-ipp-b-17df15-79665f588d-dxcz6\" (UID: \"1fb3df6c-8dde-409c-985e-0bc963a96a11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-17df15-79665f588d-dxcz6" Oct 30 18:23:20.190624 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:20.190590 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87wzs\" (UniqueName: \"kubernetes.io/projected/1fb3df6c-8dde-409c-985e-0bc963a96a11-kube-api-access-87wzs\") pod \"maas-api-e2e-ipp-b-17df15-79665f588d-dxcz6\" (UID: \"1fb3df6c-8dde-409c-985e-0bc963a96a11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-17df15-79665f588d-dxcz6" Oct 30 18:23:20.190765 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:20.190724 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fb3df6c-8dde-409c-985e-0bc963a96a11-maas-api-tls\") pod \"maas-api-e2e-ipp-b-17df15-79665f588d-dxcz6\" (UID: \"1fb3df6c-8dde-409c-985e-0bc963a96a11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-17df15-79665f588d-dxcz6" Oct 30 18:23:20.226039 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:20.225957 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a4f3f0-7c77897bf9-7mxbc" event={"ID":"f613d06d-c822-4949-9f01-73d99044a3a5","Type":"ContainerStarted","Data":"d8e2b9b14300e5caf3c7933bd357e4ddbbe4ffcc108aeeaa230d1cc74eab7b34"} Oct 30 18:23:20.226250 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:20.226061 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a4f3f0-7c77897bf9-7mxbc" Oct 30 18:23:20.228599 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:20.228572 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a4f3f0-594b877768-ctrsw" event={"ID":"2816cb76-b775-4ee2-8fe1-063879e21902","Type":"ContainerStarted","Data":"22402fadba1d91990f046a6b8fbb18e552d656be12f061039f60cf471e7fcbcf"} Oct 30 18:23:20.228753 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:20.228631 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-a4f3f0-594b877768-ctrsw" Oct 30 18:23:20.228753 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:20.228655 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a4f3f0-594b877768-ctrsw" event={"ID":"2816cb76-b775-4ee2-8fe1-063879e21902","Type":"ContainerStarted","Data":"a622a7728af9fbf32e4137e6dee912f913c2efa12134eea1b672f4ecd2ff43fa"} Oct 30 18:23:20.270753 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:20.270660 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a4f3f0-7c77897bf9-7mxbc" podStartSLOduration=2.270644811 podStartE2EDuration="2.270644811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:23:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:23:20.263216049 +0000 UTC m=+1497.353278388" watchObservedRunningTime="2026-10-30 18:23:20.270644811 +0000 UTC m=+1497.360707151" Oct 30 18:23:20.292430 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:20.292391 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fb3df6c-8dde-409c-985e-0bc963a96a11-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-17df15-79665f588d-dxcz6\" (UID: \"1fb3df6c-8dde-409c-985e-0bc963a96a11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-17df15-79665f588d-dxcz6" Oct 30 18:23:20.292621 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:20.292486 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fb3df6c-8dde-409c-985e-0bc963a96a11-metrics-certs\") pod \"maas-api-e2e-ipp-b-17df15-79665f588d-dxcz6\" (UID: \"1fb3df6c-8dde-409c-985e-0bc963a96a11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-17df15-79665f588d-dxcz6" Oct 30 18:23:20.292621 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:20.292602 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87wzs\" (UniqueName: \"kubernetes.io/projected/1fb3df6c-8dde-409c-985e-0bc963a96a11-kube-api-access-87wzs\") pod \"maas-api-e2e-ipp-b-17df15-79665f588d-dxcz6\" (UID: \"1fb3df6c-8dde-409c-985e-0bc963a96a11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-17df15-79665f588d-dxcz6" Oct 30 18:23:20.292748 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:20.292648 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fb3df6c-8dde-409c-985e-0bc963a96a11-maas-api-tls\") pod \"maas-api-e2e-ipp-b-17df15-79665f588d-dxcz6\" (UID: \"1fb3df6c-8dde-409c-985e-0bc963a96a11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-17df15-79665f588d-dxcz6" Oct 30 18:23:20.293125 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:20.293102 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fb3df6c-8dde-409c-985e-0bc963a96a11-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-17df15-79665f588d-dxcz6\" (UID: \"1fb3df6c-8dde-409c-985e-0bc963a96a11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-17df15-79665f588d-dxcz6" Oct 30 18:23:20.295279 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:20.295254 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fb3df6c-8dde-409c-985e-0bc963a96a11-maas-api-tls\") pod \"maas-api-e2e-ipp-b-17df15-79665f588d-dxcz6\" (UID: \"1fb3df6c-8dde-409c-985e-0bc963a96a11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-17df15-79665f588d-dxcz6" Oct 30 18:23:20.295398 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:20.295286 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fb3df6c-8dde-409c-985e-0bc963a96a11-metrics-certs\") pod \"maas-api-e2e-ipp-b-17df15-79665f588d-dxcz6\" (UID: \"1fb3df6c-8dde-409c-985e-0bc963a96a11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-17df15-79665f588d-dxcz6" Oct 30 18:23:20.311391 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:20.311347 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-a4f3f0-594b877768-ctrsw" podStartSLOduration=2.311333486 podStartE2EDuration="2.311333486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:23:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:23:20.306736668 +0000 UTC m=+1497.396799007" watchObservedRunningTime="2026-10-30 18:23:20.311333486 +0000 UTC m=+1497.401395824" Oct 30 18:23:20.314538 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:20.314515 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87wzs\" (UniqueName: \"kubernetes.io/projected/1fb3df6c-8dde-409c-985e-0bc963a96a11-kube-api-access-87wzs\") pod \"maas-api-e2e-ipp-b-17df15-79665f588d-dxcz6\" (UID: \"1fb3df6c-8dde-409c-985e-0bc963a96a11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-17df15-79665f588d-dxcz6" Oct 30 18:23:20.487232 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:20.473404 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-17df15-79665f588d-dxcz6" Oct 30 18:23:20.856537 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:23:20.856489 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fb3df6c_8dde_409c_985e_0bc963a96a11.slice/crio-a893a8808dbed08a3b5e1eca9a6e49f03b8c1244a1b8b945bab4c410ae4d5e14 WatchSource:0}: Error finding container a893a8808dbed08a3b5e1eca9a6e49f03b8c1244a1b8b945bab4c410ae4d5e14: Status 404 returned error can't find the container with id a893a8808dbed08a3b5e1eca9a6e49f03b8c1244a1b8b945bab4c410ae4d5e14 Oct 30 18:23:20.857043 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:20.856996 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-17df15-79665f588d-dxcz6"] Oct 30 18:23:21.241314 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:21.241282 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-17df15-79665f588d-dxcz6" event={"ID":"1fb3df6c-8dde-409c-985e-0bc963a96a11","Type":"ContainerStarted","Data":"a893a8808dbed08a3b5e1eca9a6e49f03b8c1244a1b8b945bab4c410ae4d5e14"} Oct 30 18:23:22.258764 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:22.258727 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-17df15-79665f588d-dxcz6" event={"ID":"1fb3df6c-8dde-409c-985e-0bc963a96a11","Type":"ContainerStarted","Data":"dd529ba1c6afd0841c42f5fd2c856f09ecd533cd8be828141f3e449b51a762e5"} Oct 30 18:23:22.259356 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:22.258993 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-17df15-79665f588d-dxcz6" Oct 30 18:23:22.302339 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:22.302279 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-17df15-79665f588d-dxcz6" podStartSLOduration=1.94615604 podStartE2EDuration="2.302262339s" totalImagesPullingTime="356.106299ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:23:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:23:22.298510038 +0000 UTC m=+1499.388572376" watchObservedRunningTime="2026-10-30 18:23:22.302262339 +0000 UTC m=+1499.392324708" Oct 30 18:23:23.151637 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:23.151599 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-779fd9cbd7-4fqtg" Oct 30 18:23:23.169854 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:23.169823 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-76135e-5d8b8684b6-g4jr8" Oct 30 18:23:23.319084 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:23.319057 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52rn6_516c6b6c-fc40-49cf-93b5-1c4471b02cf0/ovn-acl-logging/0.log" Oct 30 18:23:23.323629 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:23.323603 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52rn6_516c6b6c-fc40-49cf-93b5-1c4471b02cf0/ovn-acl-logging/0.log" Oct 30 18:23:28.276920 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:28.276883 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-17df15-79665f588d-dxcz6" podUID="1fb3df6c-8dde-409c-985e-0bc963a96a11" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.151:8443/health\": dial tcp 10.134.0.151:8443: connect: connection refused" Oct 30 18:23:31.266288 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:31.266249 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a4f3f0-7c77897bf9-7mxbc" Oct 30 18:23:31.281155 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:31.273432 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-a4f3f0-594b877768-ctrsw" Oct 30 18:23:32.348804 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:32.348768 4442 generic.go:356] "Generic (PLEG): container finished" podID="1fb3df6c-8dde-409c-985e-0bc963a96a11" containerID="dd529ba1c6afd0841c42f5fd2c856f09ecd533cd8be828141f3e449b51a762e5" exitCode=1 Oct 30 18:23:32.349359 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:32.348848 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-17df15-79665f588d-dxcz6" event={"ID":"1fb3df6c-8dde-409c-985e-0bc963a96a11","Type":"ContainerDied","Data":"dd529ba1c6afd0841c42f5fd2c856f09ecd533cd8be828141f3e449b51a762e5"} Oct 30 18:23:32.349446 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:32.349427 4442 scope.go:122] "RemoveContainer" containerID="dd529ba1c6afd0841c42f5fd2c856f09ecd533cd8be828141f3e449b51a762e5" Oct 30 18:23:33.361533 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:33.361491 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-17df15-79665f588d-dxcz6" event={"ID":"1fb3df6c-8dde-409c-985e-0bc963a96a11","Type":"ContainerStarted","Data":"38f69f783658c677dd5fb88a854a701447835ccff1a6b50b9a46491f29e14036"} Oct 30 18:23:33.362314 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:33.361704 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-17df15-79665f588d-dxcz6" Oct 30 18:23:39.374795 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:39.374764 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-17df15-79665f588d-dxcz6" Oct 30 18:23:47.653439 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:23:47.653379 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:23:53.427287 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:53.427251 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-17df15-dbdfd998b-949th"] Oct 30 18:23:53.458207 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:53.458174 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-784cb9f975-7grmk"] Oct 30 18:23:58.428527 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:58.428480 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-17df15-dbdfd998b-949th" podUID="cfb0bd5a-c194-4d65-b5ce-10989e213f54" containerName="payload-processing" containerID="cri-o://eb669969a213f198f76da052286fbb1f8e148fa6d5f34758834b6800715d5407" gracePeriod=25 Oct 30 18:23:58.459522 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:58.459489 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-784cb9f975-7grmk" podUID="fe684747-79a4-4547-810e-591b9c1e6832" containerName="payload-pre-processing" containerID="cri-o://dd39461f20f67b08d90e0c92824110bbff71083b03db2abdae55e3faefdaf0dc" gracePeriod=25 Oct 30 18:23:58.608905 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:58.608866 4442 generic.go:356] "Generic (PLEG): container finished" podID="cfb0bd5a-c194-4d65-b5ce-10989e213f54" containerID="eb669969a213f198f76da052286fbb1f8e148fa6d5f34758834b6800715d5407" exitCode=0 Oct 30 18:23:58.609093 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:58.609014 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-17df15-dbdfd998b-949th" event={"ID":"cfb0bd5a-c194-4d65-b5ce-10989e213f54","Type":"ContainerDied","Data":"eb669969a213f198f76da052286fbb1f8e148fa6d5f34758834b6800715d5407"} Oct 30 18:23:58.611120 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:58.611020 4442 generic.go:356] "Generic (PLEG): container finished" podID="fe684747-79a4-4547-810e-591b9c1e6832" containerID="dd39461f20f67b08d90e0c92824110bbff71083b03db2abdae55e3faefdaf0dc" exitCode=0 Oct 30 18:23:58.611120 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:58.611096 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-784cb9f975-7grmk" event={"ID":"fe684747-79a4-4547-810e-591b9c1e6832","Type":"ContainerDied","Data":"dd39461f20f67b08d90e0c92824110bbff71083b03db2abdae55e3faefdaf0dc"} Oct 30 18:23:58.802501 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:58.802473 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-17df15-dbdfd998b-949th" Oct 30 18:23:58.840344 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:58.840231 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-784cb9f975-7grmk" Oct 30 18:23:58.978914 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:58.978829 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe684747-79a4-4547-810e-591b9c1e6832-kube-api-access-zq4d2\" (UniqueName: \"kubernetes.io/projected/fe684747-79a4-4547-810e-591b9c1e6832-kube-api-access-zq4d2\") pod \"fe684747-79a4-4547-810e-591b9c1e6832\" (UID: \"fe684747-79a4-4547-810e-591b9c1e6832\") " Oct 30 18:23:58.978914 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:58.978910 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfb0bd5a-c194-4d65-b5ce-10989e213f54-kube-api-access-b7ksb\" (UniqueName: \"kubernetes.io/projected/cfb0bd5a-c194-4d65-b5ce-10989e213f54-kube-api-access-b7ksb\") pod \"cfb0bd5a-c194-4d65-b5ce-10989e213f54\" (UID: \"cfb0bd5a-c194-4d65-b5ce-10989e213f54\") " Oct 30 18:23:58.979197 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:58.978992 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe684747-79a4-4547-810e-591b9c1e6832-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe684747-79a4-4547-810e-591b9c1e6832-plugins-config-volume\") pod \"fe684747-79a4-4547-810e-591b9c1e6832\" (UID: \"fe684747-79a4-4547-810e-591b9c1e6832\") " Oct 30 18:23:58.979197 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:58.979165 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfb0bd5a-c194-4d65-b5ce-10989e213f54-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfb0bd5a-c194-4d65-b5ce-10989e213f54-plugins-config-volume\") pod \"cfb0bd5a-c194-4d65-b5ce-10989e213f54\" (UID: \"cfb0bd5a-c194-4d65-b5ce-10989e213f54\") " Oct 30 18:23:58.979435 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:58.979395 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe684747-79a4-4547-810e-591b9c1e6832-plugins-config-volume" pod "fe684747-79a4-4547-810e-591b9c1e6832" (UID: "fe684747-79a4-4547-810e-591b9c1e6832"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:23:58.979656 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:58.979623 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfb0bd5a-c194-4d65-b5ce-10989e213f54-plugins-config-volume" pod "cfb0bd5a-c194-4d65-b5ce-10989e213f54" (UID: "cfb0bd5a-c194-4d65-b5ce-10989e213f54"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:23:58.979949 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:58.979911 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe684747-79a4-4547-810e-591b9c1e6832-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:23:58.979949 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:58.979928 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfb0bd5a-c194-4d65-b5ce-10989e213f54-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:23:58.981545 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:58.981518 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfb0bd5a-c194-4d65-b5ce-10989e213f54-kube-api-access-b7ksb" pod "cfb0bd5a-c194-4d65-b5ce-10989e213f54" (UID: "cfb0bd5a-c194-4d65-b5ce-10989e213f54"). InnerVolumeSpecName "kube-api-access-b7ksb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:58.981639 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:58.981605 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe684747-79a4-4547-810e-591b9c1e6832-kube-api-access-zq4d2" pod "fe684747-79a4-4547-810e-591b9c1e6832" (UID: "fe684747-79a4-4547-810e-591b9c1e6832"). InnerVolumeSpecName "kube-api-access-zq4d2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:59.080833 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:59.080808 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq4d2\" (UniqueName: \"kubernetes.io/projected/fe684747-79a4-4547-810e-591b9c1e6832-kube-api-access-zq4d2\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:23:59.080945 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:59.080838 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7ksb\" (UniqueName: \"kubernetes.io/projected/cfb0bd5a-c194-4d65-b5ce-10989e213f54-kube-api-access-b7ksb\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:23:59.624032 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:59.622421 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-784cb9f975-7grmk" Oct 30 18:23:59.624032 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:59.622446 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-784cb9f975-7grmk" event={"ID":"fe684747-79a4-4547-810e-591b9c1e6832","Type":"ContainerDied","Data":"fa2455cd21d984cce214ad2fd67544435d1b37f547037e0036932cf6dd8f34e8"} Oct 30 18:23:59.624032 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:59.622608 4442 scope.go:122] "RemoveContainer" containerID="dd39461f20f67b08d90e0c92824110bbff71083b03db2abdae55e3faefdaf0dc" Oct 30 18:23:59.634745 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:59.634685 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-17df15-dbdfd998b-949th" event={"ID":"cfb0bd5a-c194-4d65-b5ce-10989e213f54","Type":"ContainerDied","Data":"e57982dba32716de9ac8ca2840461a30e6d84a97da9124992d7e6e7df86d716b"} Oct 30 18:23:59.635655 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:59.634854 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-17df15-dbdfd998b-949th" Oct 30 18:23:59.648985 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:59.648957 4442 scope.go:122] "RemoveContainer" containerID="eb669969a213f198f76da052286fbb1f8e148fa6d5f34758834b6800715d5407" Oct 30 18:23:59.677575 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:59.677547 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-784cb9f975-7grmk"] Oct 30 18:23:59.686845 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:59.686818 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-784cb9f975-7grmk"] Oct 30 18:23:59.727684 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:59.727590 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-17df15-dbdfd998b-949th"] Oct 30 18:23:59.734246 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:23:59.734208 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-17df15-dbdfd998b-949th"] Oct 30 18:24:01.068450 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:01.068413 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfb0bd5a-c194-4d65-b5ce-10989e213f54" path="/var/lib/kubelet/pods/cfb0bd5a-c194-4d65-b5ce-10989e213f54/volumes" Oct 30 18:24:01.069030 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:01.069009 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe684747-79a4-4547-810e-591b9c1e6832" path="/var/lib/kubelet/pods/fe684747-79a4-4547-810e-591b9c1e6832/volumes" Oct 30 18:24:06.447073 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:06.447039 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-10f1a3-6f6f779d4b-95h9b"] Oct 30 18:24:06.492538 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:06.492505 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-10f1a3-d55fc9547-kj7nd"] Oct 30 18:24:11.448036 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:11.447978 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-10f1a3-6f6f779d4b-95h9b" podUID="4bce19dc-a5e8-4487-b9c7-b67a03135319" containerName="payload-processing" containerID="cri-o://d2909bf8ed1faa60a0aa95baa169a175b053ee500bb4750de466a81ddbd36cc6" gracePeriod=25 Oct 30 18:24:11.492917 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:11.492881 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-10f1a3-d55fc9547-kj7nd" podUID="288df6a1-5215-4b1a-9013-4aa4beae9d23" containerName="payload-pre-processing" containerID="cri-o://0fcc4387e5db6c987d7395945ec76ec193274173be432efe9d09d9293de9b5d5" gracePeriod=25 Oct 30 18:24:11.775973 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:11.775877 4442 generic.go:356] "Generic (PLEG): container finished" podID="4bce19dc-a5e8-4487-b9c7-b67a03135319" containerID="d2909bf8ed1faa60a0aa95baa169a175b053ee500bb4750de466a81ddbd36cc6" exitCode=0 Oct 30 18:24:11.776076 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:11.776048 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-10f1a3-6f6f779d4b-95h9b" event={"ID":"4bce19dc-a5e8-4487-b9c7-b67a03135319","Type":"ContainerDied","Data":"d2909bf8ed1faa60a0aa95baa169a175b053ee500bb4750de466a81ddbd36cc6"} Oct 30 18:24:11.778769 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:11.778742 4442 generic.go:356] "Generic (PLEG): container finished" podID="288df6a1-5215-4b1a-9013-4aa4beae9d23" containerID="0fcc4387e5db6c987d7395945ec76ec193274173be432efe9d09d9293de9b5d5" exitCode=0 Oct 30 18:24:11.778915 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:11.778828 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-10f1a3-d55fc9547-kj7nd" event={"ID":"288df6a1-5215-4b1a-9013-4aa4beae9d23","Type":"ContainerDied","Data":"0fcc4387e5db6c987d7395945ec76ec193274173be432efe9d09d9293de9b5d5"} Oct 30 18:24:11.823430 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:11.823400 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-10f1a3-6f6f779d4b-95h9b" Oct 30 18:24:11.869568 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:11.869538 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-10f1a3-d55fc9547-kj7nd" Oct 30 18:24:11.967032 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:11.966998 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bce19dc-a5e8-4487-b9c7-b67a03135319-kube-api-access-xhszx\" (UniqueName: \"kubernetes.io/projected/4bce19dc-a5e8-4487-b9c7-b67a03135319-kube-api-access-xhszx\") pod \"4bce19dc-a5e8-4487-b9c7-b67a03135319\" (UID: \"4bce19dc-a5e8-4487-b9c7-b67a03135319\") " Oct 30 18:24:11.967263 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:11.967049 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bce19dc-a5e8-4487-b9c7-b67a03135319-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bce19dc-a5e8-4487-b9c7-b67a03135319-plugins-config-volume\") pod \"4bce19dc-a5e8-4487-b9c7-b67a03135319\" (UID: \"4bce19dc-a5e8-4487-b9c7-b67a03135319\") " Oct 30 18:24:11.967263 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:11.967117 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/288df6a1-5215-4b1a-9013-4aa4beae9d23-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/288df6a1-5215-4b1a-9013-4aa4beae9d23-plugins-config-volume\") pod \"288df6a1-5215-4b1a-9013-4aa4beae9d23\" (UID: \"288df6a1-5215-4b1a-9013-4aa4beae9d23\") " Oct 30 18:24:11.967395 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:11.967356 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/288df6a1-5215-4b1a-9013-4aa4beae9d23-kube-api-access-5hvs5\" (UniqueName: \"kubernetes.io/projected/288df6a1-5215-4b1a-9013-4aa4beae9d23-kube-api-access-5hvs5\") pod \"288df6a1-5215-4b1a-9013-4aa4beae9d23\" (UID: \"288df6a1-5215-4b1a-9013-4aa4beae9d23\") " Oct 30 18:24:11.967539 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:11.967511 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/288df6a1-5215-4b1a-9013-4aa4beae9d23-plugins-config-volume" pod "288df6a1-5215-4b1a-9013-4aa4beae9d23" (UID: "288df6a1-5215-4b1a-9013-4aa4beae9d23"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:11.967539 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:11.967516 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bce19dc-a5e8-4487-b9c7-b67a03135319-plugins-config-volume" pod "4bce19dc-a5e8-4487-b9c7-b67a03135319" (UID: "4bce19dc-a5e8-4487-b9c7-b67a03135319"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:11.968038 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:11.967987 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bce19dc-a5e8-4487-b9c7-b67a03135319-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:11.968038 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:11.968015 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/288df6a1-5215-4b1a-9013-4aa4beae9d23-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:11.969791 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:11.969762 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/288df6a1-5215-4b1a-9013-4aa4beae9d23-kube-api-access-5hvs5" pod "288df6a1-5215-4b1a-9013-4aa4beae9d23" (UID: "288df6a1-5215-4b1a-9013-4aa4beae9d23"). InnerVolumeSpecName "kube-api-access-5hvs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:11.969963 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:11.969941 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bce19dc-a5e8-4487-b9c7-b67a03135319-kube-api-access-xhszx" pod "4bce19dc-a5e8-4487-b9c7-b67a03135319" (UID: "4bce19dc-a5e8-4487-b9c7-b67a03135319"). InnerVolumeSpecName "kube-api-access-xhszx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:12.069874 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:12.069790 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhszx\" (UniqueName: \"kubernetes.io/projected/4bce19dc-a5e8-4487-b9c7-b67a03135319-kube-api-access-xhszx\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:12.069874 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:12.069828 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hvs5\" (UniqueName: \"kubernetes.io/projected/288df6a1-5215-4b1a-9013-4aa4beae9d23-kube-api-access-5hvs5\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:12.136832 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:12.130915 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9096a-7946dfc647-x8bk4"] Oct 30 18:24:12.136832 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:12.131204 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9096a-7946dfc647-x8bk4" podUID="b9770481-db87-4fed-ad1e-3c261452983f" containerName="maas-api" containerID="cri-o://be603036fa2f3e7eff39cbe679a08fa3a90b973b86726dde62f2965763c666fe" gracePeriod=30 Oct 30 18:24:12.193497 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:12.193283 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9096a-7946dfc647-x8bk4" podUID="b9770481-db87-4fed-ad1e-3c261452983f" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.122:8443/health\": dial tcp 10.134.0.122:8443: connect: connection refused" Oct 30 18:24:12.303547 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:12.303519 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e9096a-5dbbbd6d66-zfdxn"] Oct 30 18:24:12.303758 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:12.303738 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9096a-5dbbbd6d66-zfdxn" podUID="a94b8d26-825b-49b0-b1f5-0b5879807918" containerName="payload-processing" containerID="cri-o://39a0af7aa0315475c5497bd64d9ef6a69f67c11cf189faa958faf5c7cd948030" gracePeriod=30 Oct 30 18:24:12.329004 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:12.327882 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-6976b597b7-2kmj8"] Oct 30 18:24:12.329004 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:12.328166 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-6976b597b7-2kmj8" podUID="72007b47-838f-4f2f-97d5-0d93d2d3e3f1" containerName="payload-pre-processing" containerID="cri-o://1cd4db0b0b5c0c9cb344ebae436c65f93f0680228ce0e2a550064dd1e1390faf" gracePeriod=30 Oct 30 18:24:12.551545 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:12.546901 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9096a-7946dfc647-x8bk4" Oct 30 18:24:12.705203 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:12.699807 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9770481-db87-4fed-ad1e-3c261452983f-kube-api-access-68fzg\" (UniqueName: \"kubernetes.io/projected/b9770481-db87-4fed-ad1e-3c261452983f-kube-api-access-68fzg\") pod \"b9770481-db87-4fed-ad1e-3c261452983f\" (UID: \"b9770481-db87-4fed-ad1e-3c261452983f\") " Oct 30 18:24:12.705203 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:12.699934 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9770481-db87-4fed-ad1e-3c261452983f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9770481-db87-4fed-ad1e-3c261452983f-openshift-service-ca\") pod \"b9770481-db87-4fed-ad1e-3c261452983f\" (UID: \"b9770481-db87-4fed-ad1e-3c261452983f\") " Oct 30 18:24:12.742174 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:12.740896 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9770481-db87-4fed-ad1e-3c261452983f-openshift-service-ca" pod "b9770481-db87-4fed-ad1e-3c261452983f" (UID: "b9770481-db87-4fed-ad1e-3c261452983f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:12.742174 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:12.741653 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9770481-db87-4fed-ad1e-3c261452983f-openshift-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:12.800158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:12.798312 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9770481-db87-4fed-ad1e-3c261452983f-kube-api-access-68fzg" pod "b9770481-db87-4fed-ad1e-3c261452983f" (UID: "b9770481-db87-4fed-ad1e-3c261452983f"). InnerVolumeSpecName "kube-api-access-68fzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:12.829411 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:12.822960 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-10f1a3-6f6f779d4b-95h9b" event={"ID":"4bce19dc-a5e8-4487-b9c7-b67a03135319","Type":"ContainerDied","Data":"ceaeae6ca3debbdc1d7ff4fdd6a0693612702d6da60e59751480763d2925142e"} Oct 30 18:24:12.829411 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:12.823032 4442 scope.go:122] "RemoveContainer" containerID="d2909bf8ed1faa60a0aa95baa169a175b053ee500bb4750de466a81ddbd36cc6" Oct 30 18:24:12.829411 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:12.823247 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-10f1a3-6f6f779d4b-95h9b" Oct 30 18:24:12.867118 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:12.856557 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9770481-db87-4fed-ad1e-3c261452983f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9770481-db87-4fed-ad1e-3c261452983f-metrics-certs\") pod \"b9770481-db87-4fed-ad1e-3c261452983f\" (UID: \"b9770481-db87-4fed-ad1e-3c261452983f\") " Oct 30 18:24:12.867118 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:12.857536 4442 generic.go:356] "Generic (PLEG): container finished" podID="a94b8d26-825b-49b0-b1f5-0b5879807918" containerID="39a0af7aa0315475c5497bd64d9ef6a69f67c11cf189faa958faf5c7cd948030" exitCode=0 Oct 30 18:24:12.867118 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:12.857725 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9096a-5dbbbd6d66-zfdxn" event={"ID":"a94b8d26-825b-49b0-b1f5-0b5879807918","Type":"ContainerDied","Data":"39a0af7aa0315475c5497bd64d9ef6a69f67c11cf189faa958faf5c7cd948030"} Oct 30 18:24:12.867118 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:12.858236 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9770481-db87-4fed-ad1e-3c261452983f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9770481-db87-4fed-ad1e-3c261452983f-maas-api-tls\") pod \"b9770481-db87-4fed-ad1e-3c261452983f\" (UID: \"b9770481-db87-4fed-ad1e-3c261452983f\") " Oct 30 18:24:12.867118 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:12.858887 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68fzg\" (UniqueName: \"kubernetes.io/projected/b9770481-db87-4fed-ad1e-3c261452983f-kube-api-access-68fzg\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:12.867118 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:12.860911 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-10f1a3-d55fc9547-kj7nd" Oct 30 18:24:12.867118 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:12.866068 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-10f1a3-d55fc9547-kj7nd" event={"ID":"288df6a1-5215-4b1a-9013-4aa4beae9d23","Type":"ContainerDied","Data":"d3e09f2fc310390e98a5bcd1e832f366fd55292504cf42fb07a541fdb7b049d4"} Oct 30 18:24:12.869806 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:12.869122 4442 generic.go:356] "Generic (PLEG): container finished" podID="b9770481-db87-4fed-ad1e-3c261452983f" containerID="be603036fa2f3e7eff39cbe679a08fa3a90b973b86726dde62f2965763c666fe" exitCode=0 Oct 30 18:24:12.869806 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:12.869318 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9096a-7946dfc647-x8bk4" Oct 30 18:24:12.869806 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:12.869703 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9096a-7946dfc647-x8bk4" event={"ID":"b9770481-db87-4fed-ad1e-3c261452983f","Type":"ContainerDied","Data":"be603036fa2f3e7eff39cbe679a08fa3a90b973b86726dde62f2965763c666fe"} Oct 30 18:24:12.869806 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:12.869759 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9096a-7946dfc647-x8bk4" event={"ID":"b9770481-db87-4fed-ad1e-3c261452983f","Type":"ContainerDied","Data":"aa71781e5f0a94da37bda9e1781641a1199ea457f7e5d044ff7df8a46326f7a8"} Oct 30 18:24:12.873756 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:12.873573 4442 generic.go:356] "Generic (PLEG): container finished" podID="72007b47-838f-4f2f-97d5-0d93d2d3e3f1" containerID="1cd4db0b0b5c0c9cb344ebae436c65f93f0680228ce0e2a550064dd1e1390faf" exitCode=0 Oct 30 18:24:12.873756 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:12.873689 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-6976b597b7-2kmj8" event={"ID":"72007b47-838f-4f2f-97d5-0d93d2d3e3f1","Type":"ContainerDied","Data":"1cd4db0b0b5c0c9cb344ebae436c65f93f0680228ce0e2a550064dd1e1390faf"} Oct 30 18:24:12.900564 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:12.898951 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9096a-5dbbbd6d66-zfdxn" Oct 30 18:24:13.027692 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:13.027625 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9770481-db87-4fed-ad1e-3c261452983f-metrics-certs" pod "b9770481-db87-4fed-ad1e-3c261452983f" (UID: "b9770481-db87-4fed-ad1e-3c261452983f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:24:13.068047 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:13.049277 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9770481-db87-4fed-ad1e-3c261452983f-maas-api-tls" pod "b9770481-db87-4fed-ad1e-3c261452983f" (UID: "b9770481-db87-4fed-ad1e-3c261452983f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:24:13.097560 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:13.086457 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a94b8d26-825b-49b0-b1f5-0b5879807918-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a94b8d26-825b-49b0-b1f5-0b5879807918-plugins-config-volume\") pod \"a94b8d26-825b-49b0-b1f5-0b5879807918\" (UID: \"a94b8d26-825b-49b0-b1f5-0b5879807918\") " Oct 30 18:24:13.097560 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:13.086765 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a94b8d26-825b-49b0-b1f5-0b5879807918-kube-api-access-s98jg\" (UniqueName: \"kubernetes.io/projected/a94b8d26-825b-49b0-b1f5-0b5879807918-kube-api-access-s98jg\") pod \"a94b8d26-825b-49b0-b1f5-0b5879807918\" (UID: \"a94b8d26-825b-49b0-b1f5-0b5879807918\") " Oct 30 18:24:13.097560 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:13.087307 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9770481-db87-4fed-ad1e-3c261452983f-metrics-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:13.097560 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:13.087328 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9770481-db87-4fed-ad1e-3c261452983f-maas-api-tls\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:13.097560 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:13.088018 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a94b8d26-825b-49b0-b1f5-0b5879807918-plugins-config-volume" pod "a94b8d26-825b-49b0-b1f5-0b5879807918" (UID: "a94b8d26-825b-49b0-b1f5-0b5879807918"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:13.097560 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:13.089055 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-10f1a3-6f6f779d4b-95h9b"] Oct 30 18:24:13.097560 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:13.089083 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-10f1a3-6f6f779d4b-95h9b"] Oct 30 18:24:13.097560 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:13.089394 4442 scope.go:122] "RemoveContainer" containerID="0fcc4387e5db6c987d7395945ec76ec193274173be432efe9d09d9293de9b5d5" Oct 30 18:24:13.097560 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:13.093736 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-10f1a3-d55fc9547-kj7nd"] Oct 30 18:24:13.097560 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:13.096816 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a94b8d26-825b-49b0-b1f5-0b5879807918-kube-api-access-s98jg" pod "a94b8d26-825b-49b0-b1f5-0b5879807918" (UID: "a94b8d26-825b-49b0-b1f5-0b5879807918"). InnerVolumeSpecName "kube-api-access-s98jg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:13.098642 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:13.098492 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-6976b597b7-2kmj8" Oct 30 18:24:13.100843 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:13.100814 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-10f1a3-d55fc9547-kj7nd"] Oct 30 18:24:13.108514 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:13.108495 4442 scope.go:122] "RemoveContainer" containerID="be603036fa2f3e7eff39cbe679a08fa3a90b973b86726dde62f2965763c666fe" Oct 30 18:24:13.126724 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:13.126394 4442 scope.go:122] "RemoveContainer" containerID="be603036fa2f3e7eff39cbe679a08fa3a90b973b86726dde62f2965763c666fe" Oct 30 18:24:13.127034 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:24:13.126957 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be603036fa2f3e7eff39cbe679a08fa3a90b973b86726dde62f2965763c666fe\": container with ID starting with be603036fa2f3e7eff39cbe679a08fa3a90b973b86726dde62f2965763c666fe not found: ID does not exist" containerID="be603036fa2f3e7eff39cbe679a08fa3a90b973b86726dde62f2965763c666fe" Oct 30 18:24:13.127034 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:13.126993 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be603036fa2f3e7eff39cbe679a08fa3a90b973b86726dde62f2965763c666fe"} err="failed to get container status \"be603036fa2f3e7eff39cbe679a08fa3a90b973b86726dde62f2965763c666fe\": rpc error: code = NotFound desc = could not find container \"be603036fa2f3e7eff39cbe679a08fa3a90b973b86726dde62f2965763c666fe\": container with ID starting with be603036fa2f3e7eff39cbe679a08fa3a90b973b86726dde62f2965763c666fe not found: ID does not exist" Oct 30 18:24:13.188655 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:13.188623 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72007b47-838f-4f2f-97d5-0d93d2d3e3f1-kube-api-access-4vvhj\" (UniqueName: \"kubernetes.io/projected/72007b47-838f-4f2f-97d5-0d93d2d3e3f1-kube-api-access-4vvhj\") pod \"72007b47-838f-4f2f-97d5-0d93d2d3e3f1\" (UID: \"72007b47-838f-4f2f-97d5-0d93d2d3e3f1\") " Oct 30 18:24:13.188815 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:13.188691 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72007b47-838f-4f2f-97d5-0d93d2d3e3f1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72007b47-838f-4f2f-97d5-0d93d2d3e3f1-plugins-config-volume\") pod \"72007b47-838f-4f2f-97d5-0d93d2d3e3f1\" (UID: \"72007b47-838f-4f2f-97d5-0d93d2d3e3f1\") " Oct 30 18:24:13.189161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:13.189126 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s98jg\" (UniqueName: \"kubernetes.io/projected/a94b8d26-825b-49b0-b1f5-0b5879807918-kube-api-access-s98jg\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:13.189380 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:13.189360 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a94b8d26-825b-49b0-b1f5-0b5879807918-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:13.189481 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:13.189198 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72007b47-838f-4f2f-97d5-0d93d2d3e3f1-plugins-config-volume" pod "72007b47-838f-4f2f-97d5-0d93d2d3e3f1" (UID: "72007b47-838f-4f2f-97d5-0d93d2d3e3f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:13.191107 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:13.191085 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72007b47-838f-4f2f-97d5-0d93d2d3e3f1-kube-api-access-4vvhj" pod "72007b47-838f-4f2f-97d5-0d93d2d3e3f1" (UID: "72007b47-838f-4f2f-97d5-0d93d2d3e3f1"). InnerVolumeSpecName "kube-api-access-4vvhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:13.212672 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:13.212645 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9096a-7946dfc647-x8bk4"] Oct 30 18:24:13.216526 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:13.216503 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9096a-7946dfc647-x8bk4"] Oct 30 18:24:13.290994 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:13.290903 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72007b47-838f-4f2f-97d5-0d93d2d3e3f1-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:13.290994 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:13.290928 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vvhj\" (UniqueName: \"kubernetes.io/projected/72007b47-838f-4f2f-97d5-0d93d2d3e3f1-kube-api-access-4vvhj\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:13.884294 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:13.884264 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9096a-5dbbbd6d66-zfdxn" Oct 30 18:24:13.884804 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:13.884301 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9096a-5dbbbd6d66-zfdxn" event={"ID":"a94b8d26-825b-49b0-b1f5-0b5879807918","Type":"ContainerDied","Data":"4c44faa65eaf7fed55969e39e8a6ea6d4fa0b6640e17b07a4a8dfa1fd2981a58"} Oct 30 18:24:13.884804 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:13.884369 4442 scope.go:122] "RemoveContainer" containerID="39a0af7aa0315475c5497bd64d9ef6a69f67c11cf189faa958faf5c7cd948030" Oct 30 18:24:13.888294 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:13.888273 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-6976b597b7-2kmj8" Oct 30 18:24:13.888422 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:13.888317 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-6976b597b7-2kmj8" event={"ID":"72007b47-838f-4f2f-97d5-0d93d2d3e3f1","Type":"ContainerDied","Data":"6a11f926f8e7ed07d9aa01e53aa0e889725631448a33494b599009019332a304"} Oct 30 18:24:13.897473 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:13.897297 4442 scope.go:122] "RemoveContainer" containerID="1cd4db0b0b5c0c9cb344ebae436c65f93f0680228ce0e2a550064dd1e1390faf" Oct 30 18:24:13.940269 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:13.940228 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-6976b597b7-2kmj8"] Oct 30 18:24:13.946401 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:13.946364 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e9096a-6976b597b7-2kmj8"] Oct 30 18:24:13.979273 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:13.979244 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e9096a-5dbbbd6d66-zfdxn"] Oct 30 18:24:13.985691 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:13.985645 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e9096a-5dbbbd6d66-zfdxn"] Oct 30 18:24:15.096975 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:15.096929 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="288df6a1-5215-4b1a-9013-4aa4beae9d23" path="/var/lib/kubelet/pods/288df6a1-5215-4b1a-9013-4aa4beae9d23/volumes" Oct 30 18:24:15.097632 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:15.097611 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bce19dc-a5e8-4487-b9c7-b67a03135319" path="/var/lib/kubelet/pods/4bce19dc-a5e8-4487-b9c7-b67a03135319/volumes" Oct 30 18:24:15.098318 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:15.098299 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72007b47-838f-4f2f-97d5-0d93d2d3e3f1" path="/var/lib/kubelet/pods/72007b47-838f-4f2f-97d5-0d93d2d3e3f1/volumes" Oct 30 18:24:15.098876 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:15.098858 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a94b8d26-825b-49b0-b1f5-0b5879807918" path="/var/lib/kubelet/pods/a94b8d26-825b-49b0-b1f5-0b5879807918/volumes" Oct 30 18:24:15.100109 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:15.100091 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9770481-db87-4fed-ad1e-3c261452983f" path="/var/lib/kubelet/pods/b9770481-db87-4fed-ad1e-3c261452983f/volumes" Oct 30 18:24:17.561990 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:24:17.561948 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:24:20.178238 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:20.178190 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-6bc646d6bc-7blzv"] Oct 30 18:24:20.180180 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:20.180156 4442 cpu_manager.go:414] "containerMap: removing container" podUID="4bce19dc-a5e8-4487-b9c7-b67a03135319" containerName="payload-processing" Oct 30 18:24:20.180479 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:20.180461 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bce19dc-a5e8-4487-b9c7-b67a03135319" containerName="payload-processing" Oct 30 18:24:20.180623 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:20.180611 4442 cpu_manager.go:414] "containerMap: removing container" podUID="cfb0bd5a-c194-4d65-b5ce-10989e213f54" containerName="payload-processing" Oct 30 18:24:20.180729 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:20.180718 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfb0bd5a-c194-4d65-b5ce-10989e213f54" containerName="payload-processing" Oct 30 18:24:20.180833 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:20.180823 4442 cpu_manager.go:414] "containerMap: removing container" podUID="b9770481-db87-4fed-ad1e-3c261452983f" containerName="maas-api" Oct 30 18:24:20.180930 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:20.180919 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9770481-db87-4fed-ad1e-3c261452983f" containerName="maas-api" Oct 30 18:24:20.184447 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:20.181114 4442 cpu_manager.go:414] "containerMap: removing container" podUID="72007b47-838f-4f2f-97d5-0d93d2d3e3f1" containerName="payload-pre-processing" Oct 30 18:24:20.184606 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:20.184588 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72007b47-838f-4f2f-97d5-0d93d2d3e3f1" containerName="payload-pre-processing" Oct 30 18:24:20.184721 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:20.184710 4442 cpu_manager.go:414] "containerMap: removing container" podUID="288df6a1-5215-4b1a-9013-4aa4beae9d23" containerName="payload-pre-processing" Oct 30 18:24:20.184814 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:20.184802 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="288df6a1-5215-4b1a-9013-4aa4beae9d23" containerName="payload-pre-processing" Oct 30 18:24:20.184921 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:20.184910 4442 cpu_manager.go:414] "containerMap: removing container" podUID="fe684747-79a4-4547-810e-591b9c1e6832" containerName="payload-pre-processing" Oct 30 18:24:20.185034 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:20.185020 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe684747-79a4-4547-810e-591b9c1e6832" containerName="payload-pre-processing" Oct 30 18:24:20.185185 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:20.185172 4442 cpu_manager.go:414] "containerMap: removing container" podUID="a94b8d26-825b-49b0-b1f5-0b5879807918" containerName="payload-processing" Oct 30 18:24:20.185290 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:20.185275 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a94b8d26-825b-49b0-b1f5-0b5879807918" containerName="payload-processing" Oct 30 18:24:20.198477 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:20.198441 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfb0bd5a-c194-4d65-b5ce-10989e213f54" containerName="payload-processing" Oct 30 18:24:20.198690 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:20.198677 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9770481-db87-4fed-ad1e-3c261452983f" containerName="maas-api" Oct 30 18:24:20.198807 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:20.198796 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="72007b47-838f-4f2f-97d5-0d93d2d3e3f1" containerName="payload-pre-processing" Oct 30 18:24:20.198913 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:20.198903 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="288df6a1-5215-4b1a-9013-4aa4beae9d23" containerName="payload-pre-processing" Oct 30 18:24:20.199092 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:20.199080 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe684747-79a4-4547-810e-591b9c1e6832" containerName="payload-pre-processing" Oct 30 18:24:20.199214 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:20.199204 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="a94b8d26-825b-49b0-b1f5-0b5879807918" containerName="payload-processing" Oct 30 18:24:20.199400 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:20.199386 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bce19dc-a5e8-4487-b9c7-b67a03135319" containerName="payload-processing" Oct 30 18:24:20.210365 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:20.210343 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-6bc646d6bc-7blzv" Oct 30 18:24:20.215061 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:20.215022 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9pfj\" (UniqueName: \"kubernetes.io/projected/f70fb3d5-b293-45df-9950-252593a8d081-kube-api-access-z9pfj\") pod \"payload-pre-processing-e2e-ipp-b-17df15-6bc646d6bc-7blzv\" (UID: \"f70fb3d5-b293-45df-9950-252593a8d081\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-6bc646d6bc-7blzv" Oct 30 18:24:20.215404 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:20.215380 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f70fb3d5-b293-45df-9950-252593a8d081-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-17df15-6bc646d6bc-7blzv\" (UID: \"f70fb3d5-b293-45df-9950-252593a8d081\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-6bc646d6bc-7blzv" Oct 30 18:24:20.234739 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:20.234705 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-17df15\"" Oct 30 18:24:20.299219 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:20.247730 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-6bc646d6bc-7blzv"] Oct 30 18:24:20.299469 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:20.247938 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-17df15-dockercfg-88d2b\"" Oct 30 18:24:20.329214 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:20.321627 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9pfj\" (UniqueName: \"kubernetes.io/projected/f70fb3d5-b293-45df-9950-252593a8d081-kube-api-access-z9pfj\") pod \"payload-pre-processing-e2e-ipp-b-17df15-6bc646d6bc-7blzv\" (UID: \"f70fb3d5-b293-45df-9950-252593a8d081\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-6bc646d6bc-7blzv" Oct 30 18:24:20.329214 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:20.321705 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f70fb3d5-b293-45df-9950-252593a8d081-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-17df15-6bc646d6bc-7blzv\" (UID: \"f70fb3d5-b293-45df-9950-252593a8d081\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-6bc646d6bc-7blzv" Oct 30 18:24:20.351203 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:20.351174 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-17df15-77f8475476-bmgzk"] Oct 30 18:24:20.404380 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:20.404352 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-17df15-77f8475476-bmgzk" Oct 30 18:24:20.408948 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:20.408876 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9pfj\" (UniqueName: \"kubernetes.io/projected/f70fb3d5-b293-45df-9950-252593a8d081-kube-api-access-z9pfj\") pod \"payload-pre-processing-e2e-ipp-b-17df15-6bc646d6bc-7blzv\" (UID: \"f70fb3d5-b293-45df-9950-252593a8d081\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-6bc646d6bc-7blzv" Oct 30 18:24:20.418198 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:20.418165 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-17df15-77f8475476-bmgzk"] Oct 30 18:24:20.422640 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:20.421895 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f70fb3d5-b293-45df-9950-252593a8d081-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-17df15-6bc646d6bc-7blzv\" (UID: \"f70fb3d5-b293-45df-9950-252593a8d081\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-6bc646d6bc-7blzv" Oct 30 18:24:20.533466 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:20.533426 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7bzd\" (UniqueName: \"kubernetes.io/projected/a54d2880-4e10-4dda-93da-006a6f24e9b3-kube-api-access-c7bzd\") pod \"payload-processing-e2e-ipp-b-17df15-77f8475476-bmgzk\" (UID: \"a54d2880-4e10-4dda-93da-006a6f24e9b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-17df15-77f8475476-bmgzk" Oct 30 18:24:20.533743 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:20.533717 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a54d2880-4e10-4dda-93da-006a6f24e9b3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-17df15-77f8475476-bmgzk\" (UID: \"a54d2880-4e10-4dda-93da-006a6f24e9b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-17df15-77f8475476-bmgzk" Oct 30 18:24:20.614009 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:20.613974 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74"] Oct 30 18:24:20.614355 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:20.614260 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" podUID="cbf948c4-274b-46fd-bc3c-1c2f0d9c3247" containerName="istio-proxy" containerID="cri-o://595fa4ed89e446d9f45d343940926444c7dfd92e0c8af8717839548b1907734b" gracePeriod=30 Oct 30 18:24:20.621068 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:20.621040 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-6bc646d6bc-7blzv" Oct 30 18:24:20.637268 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:20.635901 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a54d2880-4e10-4dda-93da-006a6f24e9b3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-17df15-77f8475476-bmgzk\" (UID: \"a54d2880-4e10-4dda-93da-006a6f24e9b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-17df15-77f8475476-bmgzk" Oct 30 18:24:20.637268 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:20.635983 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7bzd\" (UniqueName: \"kubernetes.io/projected/a54d2880-4e10-4dda-93da-006a6f24e9b3-kube-api-access-c7bzd\") pod \"payload-processing-e2e-ipp-b-17df15-77f8475476-bmgzk\" (UID: \"a54d2880-4e10-4dda-93da-006a6f24e9b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-17df15-77f8475476-bmgzk" Oct 30 18:24:20.637268 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:20.637218 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a54d2880-4e10-4dda-93da-006a6f24e9b3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-17df15-77f8475476-bmgzk\" (UID: \"a54d2880-4e10-4dda-93da-006a6f24e9b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-17df15-77f8475476-bmgzk" Oct 30 18:24:20.658159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:20.658096 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7bzd\" (UniqueName: \"kubernetes.io/projected/a54d2880-4e10-4dda-93da-006a6f24e9b3-kube-api-access-c7bzd\") pod \"payload-processing-e2e-ipp-b-17df15-77f8475476-bmgzk\" (UID: \"a54d2880-4e10-4dda-93da-006a6f24e9b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-17df15-77f8475476-bmgzk" Oct 30 18:24:20.792160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:20.781002 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-17df15-77f8475476-bmgzk" Oct 30 18:24:21.154899 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:21.154870 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-6bc646d6bc-7blzv"] Oct 30 18:24:21.155615 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:24:21.155592 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf70fb3d5_b293_45df_9950_252593a8d081.slice/crio-0b47eda4e4cf22a680e021918d2dde01ac07d89b3d14a3c982740c50c925dec9 WatchSource:0}: Error finding container 0b47eda4e4cf22a680e021918d2dde01ac07d89b3d14a3c982740c50c925dec9: Status 404 returned error can't find the container with id 0b47eda4e4cf22a680e021918d2dde01ac07d89b3d14a3c982740c50c925dec9 Oct 30 18:24:21.319501 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:21.319464 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-17df15-77f8475476-bmgzk"] Oct 30 18:24:21.321631 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:24:21.321595 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda54d2880_4e10_4dda_93da_006a6f24e9b3.slice/crio-a0fe5a7cb5131bb9a8ee2d76ab17f3ec7de1bd87a30e5ce26c0ed1cc7fc9e3b8 WatchSource:0}: Error finding container a0fe5a7cb5131bb9a8ee2d76ab17f3ec7de1bd87a30e5ce26c0ed1cc7fc9e3b8: Status 404 returned error can't find the container with id a0fe5a7cb5131bb9a8ee2d76ab17f3ec7de1bd87a30e5ce26c0ed1cc7fc9e3b8 Oct 30 18:24:22.067590 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:22.067556 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-6bc646d6bc-7blzv" event={"ID":"f70fb3d5-b293-45df-9950-252593a8d081","Type":"ContainerStarted","Data":"6b697fad9ee46ebddacc99a9679a6d6c30c95b15669891f1123a977aa15d6005"} Oct 30 18:24:22.067788 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:22.067608 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-6bc646d6bc-7blzv" event={"ID":"f70fb3d5-b293-45df-9950-252593a8d081","Type":"ContainerStarted","Data":"0b47eda4e4cf22a680e021918d2dde01ac07d89b3d14a3c982740c50c925dec9"} Oct 30 18:24:22.067788 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:22.067672 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-6bc646d6bc-7blzv" Oct 30 18:24:22.069560 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:22.069531 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-17df15-77f8475476-bmgzk" event={"ID":"a54d2880-4e10-4dda-93da-006a6f24e9b3","Type":"ContainerStarted","Data":"e449a4a37760b97a0b6b60614e14957a3ca3e90c2b4552dead970fea6865a320"} Oct 30 18:24:22.069702 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:22.069576 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-17df15-77f8475476-bmgzk" event={"ID":"a54d2880-4e10-4dda-93da-006a6f24e9b3","Type":"ContainerStarted","Data":"a0fe5a7cb5131bb9a8ee2d76ab17f3ec7de1bd87a30e5ce26c0ed1cc7fc9e3b8"} Oct 30 18:24:22.069761 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:22.069725 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-17df15-77f8475476-bmgzk" Oct 30 18:24:22.106562 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:22.106516 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-6bc646d6bc-7blzv" podStartSLOduration=2.106503756 podStartE2EDuration="2.106503756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:22.10605232 +0000 UTC m=+1559.196114671" watchObservedRunningTime="2026-10-30 18:24:22.106503756 +0000 UTC m=+1559.196566094" Oct 30 18:24:22.145327 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:22.145276 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-17df15-77f8475476-bmgzk" podStartSLOduration=2.145258096 podStartE2EDuration="2.145258096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:22.143004487 +0000 UTC m=+1559.233066829" watchObservedRunningTime="2026-10-30 18:24:22.145258096 +0000 UTC m=+1559.235320442" Oct 30 18:24:25.909151 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:25.909108 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" Oct 30 18:24:25.927489 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:25.927463 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-istio-podinfo\") pod \"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247\" (UID: \"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247\") " Oct 30 18:24:25.927643 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:25.927505 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-credential-socket\") pod \"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247\" (UID: \"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247\") " Oct 30 18:24:25.927643 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:25.927543 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-istio-data\") pod \"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247\" (UID: \"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247\") " Oct 30 18:24:25.927643 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:25.927569 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-workload-socket\") pod \"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247\" (UID: \"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247\") " Oct 30 18:24:25.927643 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:25.927598 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-istio-envoy\") pod \"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247\" (UID: \"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247\") " Oct 30 18:24:25.927863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:25.927716 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-kube-api-access-jl2xd\" (UniqueName: \"kubernetes.io/projected/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-kube-api-access-jl2xd\") pod \"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247\" (UID: \"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247\") " Oct 30 18:24:25.927863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:25.927803 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-workload-certs\") pod \"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247\" (UID: \"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247\") " Oct 30 18:24:25.927955 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:25.927822 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-credential-socket" pod "cbf948c4-274b-46fd-bc3c-1c2f0d9c3247" (UID: "cbf948c4-274b-46fd-bc3c-1c2f0d9c3247"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:25.927955 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:25.927864 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-istio-data" pod "cbf948c4-274b-46fd-bc3c-1c2f0d9c3247" (UID: "cbf948c4-274b-46fd-bc3c-1c2f0d9c3247"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:25.927955 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:25.927889 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-workload-socket" pod "cbf948c4-274b-46fd-bc3c-1c2f0d9c3247" (UID: "cbf948c4-274b-46fd-bc3c-1c2f0d9c3247"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:25.927955 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:25.927924 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-istio-token\" (UniqueName: \"kubernetes.io/projected/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-istio-token\") pod \"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247\" (UID: \"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247\") " Oct 30 18:24:25.928179 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:25.927961 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-istiod-ca-cert\") pod \"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247\" (UID: \"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247\") " Oct 30 18:24:25.928179 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:25.928122 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-workload-certs" pod "cbf948c4-274b-46fd-bc3c-1c2f0d9c3247" (UID: "cbf948c4-274b-46fd-bc3c-1c2f0d9c3247"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:25.928427 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:25.928403 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-istiod-ca-cert" pod "cbf948c4-274b-46fd-bc3c-1c2f0d9c3247" (UID: "cbf948c4-274b-46fd-bc3c-1c2f0d9c3247"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:25.928656 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:25.928632 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:25.928656 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:25.928660 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:25.929010 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:25.928673 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:25.929010 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:25.928687 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:25.929010 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:25.928700 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:25.930890 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:25.930865 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-istio-podinfo" pod "cbf948c4-274b-46fd-bc3c-1c2f0d9c3247" (UID: "cbf948c4-274b-46fd-bc3c-1c2f0d9c3247"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:24:25.931008 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:25.930899 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-istio-token" pod "cbf948c4-274b-46fd-bc3c-1c2f0d9c3247" (UID: "cbf948c4-274b-46fd-bc3c-1c2f0d9c3247"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:25.931008 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:25.930904 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-istio-envoy" pod "cbf948c4-274b-46fd-bc3c-1c2f0d9c3247" (UID: "cbf948c4-274b-46fd-bc3c-1c2f0d9c3247"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:25.931008 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:25.930958 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-kube-api-access-jl2xd" pod "cbf948c4-274b-46fd-bc3c-1c2f0d9c3247" (UID: "cbf948c4-274b-46fd-bc3c-1c2f0d9c3247"). InnerVolumeSpecName "kube-api-access-jl2xd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:26.030024 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:26.029984 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:26.030024 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:26.030021 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl2xd\" (UniqueName: \"kubernetes.io/projected/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-kube-api-access-jl2xd\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:26.030262 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:26.030035 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:26.030262 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:26.030048 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:26.111728 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:26.111692 4442 generic.go:356] "Generic (PLEG): container finished" podID="cbf948c4-274b-46fd-bc3c-1c2f0d9c3247" containerID="595fa4ed89e446d9f45d343940926444c7dfd92e0c8af8717839548b1907734b" exitCode=0 Oct 30 18:24:26.111899 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:26.111739 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" Oct 30 18:24:26.111899 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:26.111772 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" event={"ID":"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247","Type":"ContainerDied","Data":"595fa4ed89e446d9f45d343940926444c7dfd92e0c8af8717839548b1907734b"} Oct 30 18:24:26.111899 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:26.111834 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74" event={"ID":"cbf948c4-274b-46fd-bc3c-1c2f0d9c3247","Type":"ContainerDied","Data":"018218b52a72839174f29782938f08719af2641d4180fa3775bc5d83784babae"} Oct 30 18:24:26.111899 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:26.111865 4442 scope.go:122] "RemoveContainer" containerID="595fa4ed89e446d9f45d343940926444c7dfd92e0c8af8717839548b1907734b" Oct 30 18:24:26.124199 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:26.124176 4442 scope.go:122] "RemoveContainer" containerID="318b9c893169d6da3c506b4f3cddfb1f088595d443be070791da37e9149af3ce" Oct 30 18:24:26.137410 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:26.137390 4442 scope.go:122] "RemoveContainer" containerID="595fa4ed89e446d9f45d343940926444c7dfd92e0c8af8717839548b1907734b" Oct 30 18:24:26.137761 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:24:26.137737 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"595fa4ed89e446d9f45d343940926444c7dfd92e0c8af8717839548b1907734b\": container with ID starting with 595fa4ed89e446d9f45d343940926444c7dfd92e0c8af8717839548b1907734b not found: ID does not exist" containerID="595fa4ed89e446d9f45d343940926444c7dfd92e0c8af8717839548b1907734b" Oct 30 18:24:26.137827 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:26.137772 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"595fa4ed89e446d9f45d343940926444c7dfd92e0c8af8717839548b1907734b"} err="failed to get container status \"595fa4ed89e446d9f45d343940926444c7dfd92e0c8af8717839548b1907734b\": rpc error: code = NotFound desc = could not find container \"595fa4ed89e446d9f45d343940926444c7dfd92e0c8af8717839548b1907734b\": container with ID starting with 595fa4ed89e446d9f45d343940926444c7dfd92e0c8af8717839548b1907734b not found: ID does not exist" Oct 30 18:24:26.137827 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:26.137796 4442 scope.go:122] "RemoveContainer" containerID="318b9c893169d6da3c506b4f3cddfb1f088595d443be070791da37e9149af3ce" Oct 30 18:24:26.138084 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:24:26.138062 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"318b9c893169d6da3c506b4f3cddfb1f088595d443be070791da37e9149af3ce\": container with ID starting with 318b9c893169d6da3c506b4f3cddfb1f088595d443be070791da37e9149af3ce not found: ID does not exist" containerID="318b9c893169d6da3c506b4f3cddfb1f088595d443be070791da37e9149af3ce" Oct 30 18:24:26.138186 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:26.138095 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"318b9c893169d6da3c506b4f3cddfb1f088595d443be070791da37e9149af3ce"} err="failed to get container status \"318b9c893169d6da3c506b4f3cddfb1f088595d443be070791da37e9149af3ce\": rpc error: code = NotFound desc = could not find container \"318b9c893169d6da3c506b4f3cddfb1f088595d443be070791da37e9149af3ce\": container with ID starting with 318b9c893169d6da3c506b4f3cddfb1f088595d443be070791da37e9149af3ce not found: ID does not exist" Oct 30 18:24:26.165926 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:26.165896 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74"] Oct 30 18:24:26.172330 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:26.172305 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-e9096a-openshift-default-66f6d7f4dc-8lv74"] Oct 30 18:24:27.065632 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:27.065597 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbf948c4-274b-46fd-bc3c-1c2f0d9c3247" path="/var/lib/kubelet/pods/cbf948c4-274b-46fd-bc3c-1c2f0d9c3247/volumes" Oct 30 18:24:33.079510 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:33.079480 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-6bc646d6bc-7blzv" Oct 30 18:24:33.079980 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:33.079568 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-17df15-77f8475476-bmgzk" Oct 30 18:24:36.625479 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:36.625440 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-76135e-68d64b77cf-qtc44"] Oct 30 18:24:36.625944 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:36.625680 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76135e-68d64b77cf-qtc44" podUID="71aab62b-754d-4bae-8b80-d52633dc0183" containerName="maas-api" containerID="cri-o://68124cbc6cd8c12fcc810c82d8e98a28ac1d801d3e4b1ab18b0f36bcf51bddfc" gracePeriod=30 Oct 30 18:24:36.831189 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:36.822359 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-76135e-5d8b8684b6-g4jr8"] Oct 30 18:24:36.831189 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:36.822628 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-76135e-5d8b8684b6-g4jr8" podUID="6c344a53-a70f-416d-b9c0-0e6156517df9" containerName="payload-processing" containerID="cri-o://10452be3e51d807b3b06b98c99556728d92c41437f4ae2d4b2093b354a0839b7" gracePeriod=30 Oct 30 18:24:36.849146 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:36.848148 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-779fd9cbd7-4fqtg"] Oct 30 18:24:36.849146 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:36.848477 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-779fd9cbd7-4fqtg" podUID="4795553f-070d-48df-87c1-db573810c49f" containerName="payload-pre-processing" containerID="cri-o://b253673626709c557f82295e22ec21f30949edbf3374f8606ddda1135c941b05" gracePeriod=30 Oct 30 18:24:36.989499 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:36.989466 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76135e-68d64b77cf-qtc44" Oct 30 18:24:37.107336 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:37.107301 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71aab62b-754d-4bae-8b80-d52633dc0183-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71aab62b-754d-4bae-8b80-d52633dc0183-maas-api-tls\") pod \"71aab62b-754d-4bae-8b80-d52633dc0183\" (UID: \"71aab62b-754d-4bae-8b80-d52633dc0183\") " Oct 30 18:24:37.107490 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:37.107418 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71aab62b-754d-4bae-8b80-d52633dc0183-metrics-certs\" (UniqueName: \"kubernetes.io/secret/71aab62b-754d-4bae-8b80-d52633dc0183-metrics-certs\") pod \"71aab62b-754d-4bae-8b80-d52633dc0183\" (UID: \"71aab62b-754d-4bae-8b80-d52633dc0183\") " Oct 30 18:24:37.108185 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:37.107695 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71aab62b-754d-4bae-8b80-d52633dc0183-kube-api-access-pqfk6\" (UniqueName: \"kubernetes.io/projected/71aab62b-754d-4bae-8b80-d52633dc0183-kube-api-access-pqfk6\") pod \"71aab62b-754d-4bae-8b80-d52633dc0183\" (UID: \"71aab62b-754d-4bae-8b80-d52633dc0183\") " Oct 30 18:24:37.108185 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:37.107782 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71aab62b-754d-4bae-8b80-d52633dc0183-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71aab62b-754d-4bae-8b80-d52633dc0183-openshift-service-ca\") pod \"71aab62b-754d-4bae-8b80-d52633dc0183\" (UID: \"71aab62b-754d-4bae-8b80-d52633dc0183\") " Oct 30 18:24:37.109160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:37.109059 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71aab62b-754d-4bae-8b80-d52633dc0183-openshift-service-ca" pod "71aab62b-754d-4bae-8b80-d52633dc0183" (UID: "71aab62b-754d-4bae-8b80-d52633dc0183"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:37.113323 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:37.113281 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71aab62b-754d-4bae-8b80-d52633dc0183-metrics-certs" pod "71aab62b-754d-4bae-8b80-d52633dc0183" (UID: "71aab62b-754d-4bae-8b80-d52633dc0183"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:24:37.128154 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:37.124197 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71aab62b-754d-4bae-8b80-d52633dc0183-kube-api-access-pqfk6" pod "71aab62b-754d-4bae-8b80-d52633dc0183" (UID: "71aab62b-754d-4bae-8b80-d52633dc0183"). InnerVolumeSpecName "kube-api-access-pqfk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:37.128154 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:37.125022 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71aab62b-754d-4bae-8b80-d52633dc0183-maas-api-tls" pod "71aab62b-754d-4bae-8b80-d52633dc0183" (UID: "71aab62b-754d-4bae-8b80-d52633dc0183"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:24:37.345193 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:37.340061 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqfk6\" (UniqueName: \"kubernetes.io/projected/71aab62b-754d-4bae-8b80-d52633dc0183-kube-api-access-pqfk6\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:37.345193 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:37.340095 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71aab62b-754d-4bae-8b80-d52633dc0183-openshift-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:37.345193 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:37.340110 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71aab62b-754d-4bae-8b80-d52633dc0183-maas-api-tls\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:37.345193 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:37.340123 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71aab62b-754d-4bae-8b80-d52633dc0183-metrics-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:37.475154 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:37.473778 4442 generic.go:356] "Generic (PLEG): container finished" podID="4795553f-070d-48df-87c1-db573810c49f" containerID="b253673626709c557f82295e22ec21f30949edbf3374f8606ddda1135c941b05" exitCode=0 Oct 30 18:24:37.475154 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:37.473947 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-779fd9cbd7-4fqtg" event={"ID":"4795553f-070d-48df-87c1-db573810c49f","Type":"ContainerDied","Data":"b253673626709c557f82295e22ec21f30949edbf3374f8606ddda1135c941b05"} Oct 30 18:24:37.475154 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:37.473995 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-779fd9cbd7-4fqtg" event={"ID":"4795553f-070d-48df-87c1-db573810c49f","Type":"ContainerDied","Data":"9d2ef7eade479a90968cb9c68ff06f8ecf63881b680943e6414a1be1723e5e40"} Oct 30 18:24:37.475154 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:37.474016 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d2ef7eade479a90968cb9c68ff06f8ecf63881b680943e6414a1be1723e5e40" Oct 30 18:24:37.481278 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:37.481232 4442 generic.go:356] "Generic (PLEG): container finished" podID="71aab62b-754d-4bae-8b80-d52633dc0183" containerID="68124cbc6cd8c12fcc810c82d8e98a28ac1d801d3e4b1ab18b0f36bcf51bddfc" exitCode=0 Oct 30 18:24:37.481729 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:37.481376 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76135e-68d64b77cf-qtc44" Oct 30 18:24:37.482542 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:37.481395 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76135e-68d64b77cf-qtc44" event={"ID":"71aab62b-754d-4bae-8b80-d52633dc0183","Type":"ContainerDied","Data":"68124cbc6cd8c12fcc810c82d8e98a28ac1d801d3e4b1ab18b0f36bcf51bddfc"} Oct 30 18:24:37.483195 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:37.483048 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76135e-68d64b77cf-qtc44" event={"ID":"71aab62b-754d-4bae-8b80-d52633dc0183","Type":"ContainerDied","Data":"adaafe187455b2ec475768afb7a39b54d7c445b40f373fab0b7ad67ba1f5d346"} Oct 30 18:24:37.483377 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:37.483283 4442 scope.go:122] "RemoveContainer" containerID="68124cbc6cd8c12fcc810c82d8e98a28ac1d801d3e4b1ab18b0f36bcf51bddfc" Oct 30 18:24:37.487302 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:37.487280 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-76135e-5d8b8684b6-g4jr8" Oct 30 18:24:37.488382 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:37.488356 4442 generic.go:356] "Generic (PLEG): container finished" podID="6c344a53-a70f-416d-b9c0-0e6156517df9" containerID="10452be3e51d807b3b06b98c99556728d92c41437f4ae2d4b2093b354a0839b7" exitCode=0 Oct 30 18:24:37.488491 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:37.488425 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-76135e-5d8b8684b6-g4jr8" event={"ID":"6c344a53-a70f-416d-b9c0-0e6156517df9","Type":"ContainerDied","Data":"10452be3e51d807b3b06b98c99556728d92c41437f4ae2d4b2093b354a0839b7"} Oct 30 18:24:37.488491 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:37.488464 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-76135e-5d8b8684b6-g4jr8" event={"ID":"6c344a53-a70f-416d-b9c0-0e6156517df9","Type":"ContainerDied","Data":"5957c9ee2688f219664097e17bad31e6171dc93dd1d404bbef000e709ebd9282"} Oct 30 18:24:37.491809 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:37.491790 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-779fd9cbd7-4fqtg" Oct 30 18:24:37.501344 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:37.501330 4442 scope.go:122] "RemoveContainer" containerID="68124cbc6cd8c12fcc810c82d8e98a28ac1d801d3e4b1ab18b0f36bcf51bddfc" Oct 30 18:24:37.501609 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:24:37.501593 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68124cbc6cd8c12fcc810c82d8e98a28ac1d801d3e4b1ab18b0f36bcf51bddfc\": container with ID starting with 68124cbc6cd8c12fcc810c82d8e98a28ac1d801d3e4b1ab18b0f36bcf51bddfc not found: ID does not exist" containerID="68124cbc6cd8c12fcc810c82d8e98a28ac1d801d3e4b1ab18b0f36bcf51bddfc" Oct 30 18:24:37.501659 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:37.501616 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68124cbc6cd8c12fcc810c82d8e98a28ac1d801d3e4b1ab18b0f36bcf51bddfc"} err="failed to get container status \"68124cbc6cd8c12fcc810c82d8e98a28ac1d801d3e4b1ab18b0f36bcf51bddfc\": rpc error: code = NotFound desc = could not find container \"68124cbc6cd8c12fcc810c82d8e98a28ac1d801d3e4b1ab18b0f36bcf51bddfc\": container with ID starting with 68124cbc6cd8c12fcc810c82d8e98a28ac1d801d3e4b1ab18b0f36bcf51bddfc not found: ID does not exist" Oct 30 18:24:37.501659 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:37.501632 4442 scope.go:122] "RemoveContainer" containerID="10452be3e51d807b3b06b98c99556728d92c41437f4ae2d4b2093b354a0839b7" Oct 30 18:24:37.512053 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:37.512031 4442 scope.go:122] "RemoveContainer" containerID="10452be3e51d807b3b06b98c99556728d92c41437f4ae2d4b2093b354a0839b7" Oct 30 18:24:37.512536 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:24:37.512481 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10452be3e51d807b3b06b98c99556728d92c41437f4ae2d4b2093b354a0839b7\": container with ID starting with 10452be3e51d807b3b06b98c99556728d92c41437f4ae2d4b2093b354a0839b7 not found: ID does not exist" containerID="10452be3e51d807b3b06b98c99556728d92c41437f4ae2d4b2093b354a0839b7" Oct 30 18:24:37.512536 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:37.512514 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10452be3e51d807b3b06b98c99556728d92c41437f4ae2d4b2093b354a0839b7"} err="failed to get container status \"10452be3e51d807b3b06b98c99556728d92c41437f4ae2d4b2093b354a0839b7\": rpc error: code = NotFound desc = could not find container \"10452be3e51d807b3b06b98c99556728d92c41437f4ae2d4b2093b354a0839b7\": container with ID starting with 10452be3e51d807b3b06b98c99556728d92c41437f4ae2d4b2093b354a0839b7 not found: ID does not exist" Oct 30 18:24:37.571411 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:37.571316 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c344a53-a70f-416d-b9c0-0e6156517df9-kube-api-access-wskjc\" (UniqueName: \"kubernetes.io/projected/6c344a53-a70f-416d-b9c0-0e6156517df9-kube-api-access-wskjc\") pod \"6c344a53-a70f-416d-b9c0-0e6156517df9\" (UID: \"6c344a53-a70f-416d-b9c0-0e6156517df9\") " Oct 30 18:24:37.571592 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:37.571452 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4795553f-070d-48df-87c1-db573810c49f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4795553f-070d-48df-87c1-db573810c49f-plugins-config-volume\") pod \"4795553f-070d-48df-87c1-db573810c49f\" (UID: \"4795553f-070d-48df-87c1-db573810c49f\") " Oct 30 18:24:37.571592 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:37.571506 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4795553f-070d-48df-87c1-db573810c49f-kube-api-access-jqj25\" (UniqueName: \"kubernetes.io/projected/4795553f-070d-48df-87c1-db573810c49f-kube-api-access-jqj25\") pod \"4795553f-070d-48df-87c1-db573810c49f\" (UID: \"4795553f-070d-48df-87c1-db573810c49f\") " Oct 30 18:24:37.571719 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:37.571638 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c344a53-a70f-416d-b9c0-0e6156517df9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c344a53-a70f-416d-b9c0-0e6156517df9-plugins-config-volume\") pod \"6c344a53-a70f-416d-b9c0-0e6156517df9\" (UID: \"6c344a53-a70f-416d-b9c0-0e6156517df9\") " Oct 30 18:24:37.572769 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:37.572682 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c344a53-a70f-416d-b9c0-0e6156517df9-plugins-config-volume" pod "6c344a53-a70f-416d-b9c0-0e6156517df9" (UID: "6c344a53-a70f-416d-b9c0-0e6156517df9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:37.572769 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:37.572722 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4795553f-070d-48df-87c1-db573810c49f-plugins-config-volume" pod "4795553f-070d-48df-87c1-db573810c49f" (UID: "4795553f-070d-48df-87c1-db573810c49f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:37.575562 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:37.575510 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c344a53-a70f-416d-b9c0-0e6156517df9-kube-api-access-wskjc" pod "6c344a53-a70f-416d-b9c0-0e6156517df9" (UID: "6c344a53-a70f-416d-b9c0-0e6156517df9"). InnerVolumeSpecName "kube-api-access-wskjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:37.575706 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:37.575576 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4795553f-070d-48df-87c1-db573810c49f-kube-api-access-jqj25" pod "4795553f-070d-48df-87c1-db573810c49f" (UID: "4795553f-070d-48df-87c1-db573810c49f"). InnerVolumeSpecName "kube-api-access-jqj25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:37.616595 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:37.616563 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-76135e-68d64b77cf-qtc44"] Oct 30 18:24:37.624579 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:37.624481 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-76135e-68d64b77cf-qtc44"] Oct 30 18:24:37.673952 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:37.673922 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wskjc\" (UniqueName: \"kubernetes.io/projected/6c344a53-a70f-416d-b9c0-0e6156517df9-kube-api-access-wskjc\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:37.673952 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:37.673952 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4795553f-070d-48df-87c1-db573810c49f-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:37.674529 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:37.673964 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqj25\" (UniqueName: \"kubernetes.io/projected/4795553f-070d-48df-87c1-db573810c49f-kube-api-access-jqj25\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:37.674529 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:37.673974 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c344a53-a70f-416d-b9c0-0e6156517df9-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:38.498799 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:38.498741 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-76135e-5d8b8684b6-g4jr8" Oct 30 18:24:38.500301 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:38.500281 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-779fd9cbd7-4fqtg" Oct 30 18:24:38.576987 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:38.576952 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-76135e-5d8b8684b6-g4jr8"] Oct 30 18:24:38.580862 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:38.580840 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-76135e-5d8b8684b6-g4jr8"] Oct 30 18:24:38.588756 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:38.588730 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-200c58-7c6bfb8c8b-ppvqf"] Oct 30 18:24:38.589649 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:38.589630 4442 cpu_manager.go:414] "containerMap: removing container" podUID="cbf948c4-274b-46fd-bc3c-1c2f0d9c3247" containerName="istio-proxy" Oct 30 18:24:38.589649 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:38.589649 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbf948c4-274b-46fd-bc3c-1c2f0d9c3247" containerName="istio-proxy" Oct 30 18:24:38.589864 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:38.589678 4442 cpu_manager.go:414] "containerMap: removing container" podUID="4795553f-070d-48df-87c1-db573810c49f" containerName="payload-pre-processing" Oct 30 18:24:38.589864 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:38.589764 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4795553f-070d-48df-87c1-db573810c49f" containerName="payload-pre-processing" Oct 30 18:24:38.589864 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:38.589796 4442 cpu_manager.go:414] "containerMap: removing container" podUID="6c344a53-a70f-416d-b9c0-0e6156517df9" containerName="payload-processing" Oct 30 18:24:38.589864 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:38.589805 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c344a53-a70f-416d-b9c0-0e6156517df9" containerName="payload-processing" Oct 30 18:24:38.589864 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:38.589829 4442 cpu_manager.go:414] "containerMap: removing container" podUID="71aab62b-754d-4bae-8b80-d52633dc0183" containerName="maas-api" Oct 30 18:24:38.589864 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:38.589836 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71aab62b-754d-4bae-8b80-d52633dc0183" containerName="maas-api" Oct 30 18:24:38.590233 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:38.590018 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbf948c4-274b-46fd-bc3c-1c2f0d9c3247" containerName="istio-proxy" Oct 30 18:24:38.590233 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:38.590034 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbf948c4-274b-46fd-bc3c-1c2f0d9c3247" containerName="istio-proxy" Oct 30 18:24:38.590233 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:38.590049 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="4795553f-070d-48df-87c1-db573810c49f" containerName="payload-pre-processing" Oct 30 18:24:38.590233 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:38.590062 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c344a53-a70f-416d-b9c0-0e6156517df9" containerName="payload-processing" Oct 30 18:24:38.590233 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:38.590075 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="71aab62b-754d-4bae-8b80-d52633dc0183" containerName="maas-api" Oct 30 18:24:38.594059 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:38.594037 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-200c58-7c6bfb8c8b-ppvqf" Oct 30 18:24:38.615908 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:38.615882 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-200c58\"" Oct 30 18:24:38.621983 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:38.621960 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-779fd9cbd7-4fqtg"] Oct 30 18:24:38.632360 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:38.632304 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-200c58-7c6bfb8c8b-ppvqf"] Oct 30 18:24:38.645118 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:38.645086 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-76135e-779fd9cbd7-4fqtg"] Oct 30 18:24:38.674987 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:38.674958 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-200c58-7d66656f4c-8k98q"] Oct 30 18:24:38.675827 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:38.675800 4442 cpu_manager.go:414] "containerMap: removing container" podUID="cbf948c4-274b-46fd-bc3c-1c2f0d9c3247" containerName="istio-proxy" Oct 30 18:24:38.675827 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:38.675825 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbf948c4-274b-46fd-bc3c-1c2f0d9c3247" containerName="istio-proxy" Oct 30 18:24:38.680087 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:38.680058 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-200c58-7d66656f4c-8k98q" Oct 30 18:24:38.708332 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:38.708310 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-200c58-dockercfg-848cx\"" Oct 30 18:24:38.710303 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:38.710275 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f1c999a-d8e4-4ee1-8cca-f53f93eed216-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-200c58-7c6bfb8c8b-ppvqf\" (UID: \"0f1c999a-d8e4-4ee1-8cca-f53f93eed216\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-200c58-7c6bfb8c8b-ppvqf" Oct 30 18:24:38.710471 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:38.710442 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g4fc\" (UniqueName: \"kubernetes.io/projected/0f1c999a-d8e4-4ee1-8cca-f53f93eed216-kube-api-access-9g4fc\") pod \"payload-processing-e2e-praxis-b-200c58-7c6bfb8c8b-ppvqf\" (UID: \"0f1c999a-d8e4-4ee1-8cca-f53f93eed216\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-200c58-7c6bfb8c8b-ppvqf" Oct 30 18:24:38.715304 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:38.715273 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-200c58-7d66656f4c-8k98q"] Oct 30 18:24:38.859206 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:38.858492 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g4fc\" (UniqueName: \"kubernetes.io/projected/0f1c999a-d8e4-4ee1-8cca-f53f93eed216-kube-api-access-9g4fc\") pod \"payload-processing-e2e-praxis-b-200c58-7c6bfb8c8b-ppvqf\" (UID: \"0f1c999a-d8e4-4ee1-8cca-f53f93eed216\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-200c58-7c6bfb8c8b-ppvqf" Oct 30 18:24:38.859206 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:38.858728 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f1c999a-d8e4-4ee1-8cca-f53f93eed216-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-200c58-7c6bfb8c8b-ppvqf\" (UID: \"0f1c999a-d8e4-4ee1-8cca-f53f93eed216\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-200c58-7c6bfb8c8b-ppvqf" Oct 30 18:24:38.884158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:38.859723 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f1c999a-d8e4-4ee1-8cca-f53f93eed216-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-200c58-7c6bfb8c8b-ppvqf\" (UID: \"0f1c999a-d8e4-4ee1-8cca-f53f93eed216\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-200c58-7c6bfb8c8b-ppvqf" Oct 30 18:24:38.974157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:38.973656 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r9wz\" (UniqueName: \"kubernetes.io/projected/9e4d3b1c-e383-4db5-aeec-8a6a66d32808-kube-api-access-5r9wz\") pod \"payload-pre-processing-e2e-praxis-b-200c58-7d66656f4c-8k98q\" (UID: \"9e4d3b1c-e383-4db5-aeec-8a6a66d32808\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-200c58-7d66656f4c-8k98q" Oct 30 18:24:38.974157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:38.973757 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e4d3b1c-e383-4db5-aeec-8a6a66d32808-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-200c58-7d66656f4c-8k98q\" (UID: \"9e4d3b1c-e383-4db5-aeec-8a6a66d32808\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-200c58-7d66656f4c-8k98q" Oct 30 18:24:39.008042 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:39.008002 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g4fc\" (UniqueName: \"kubernetes.io/projected/0f1c999a-d8e4-4ee1-8cca-f53f93eed216-kube-api-access-9g4fc\") pod \"payload-processing-e2e-praxis-b-200c58-7c6bfb8c8b-ppvqf\" (UID: \"0f1c999a-d8e4-4ee1-8cca-f53f93eed216\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-200c58-7c6bfb8c8b-ppvqf" Oct 30 18:24:39.068645 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:39.068611 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4795553f-070d-48df-87c1-db573810c49f" path="/var/lib/kubelet/pods/4795553f-070d-48df-87c1-db573810c49f/volumes" Oct 30 18:24:39.069110 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:39.069093 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c344a53-a70f-416d-b9c0-0e6156517df9" path="/var/lib/kubelet/pods/6c344a53-a70f-416d-b9c0-0e6156517df9/volumes" Oct 30 18:24:39.069547 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:39.069531 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71aab62b-754d-4bae-8b80-d52633dc0183" path="/var/lib/kubelet/pods/71aab62b-754d-4bae-8b80-d52633dc0183/volumes" Oct 30 18:24:39.074616 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:39.074591 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r9wz\" (UniqueName: \"kubernetes.io/projected/9e4d3b1c-e383-4db5-aeec-8a6a66d32808-kube-api-access-5r9wz\") pod \"payload-pre-processing-e2e-praxis-b-200c58-7d66656f4c-8k98q\" (UID: \"9e4d3b1c-e383-4db5-aeec-8a6a66d32808\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-200c58-7d66656f4c-8k98q" Oct 30 18:24:39.074760 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:39.074646 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e4d3b1c-e383-4db5-aeec-8a6a66d32808-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-200c58-7d66656f4c-8k98q\" (UID: \"9e4d3b1c-e383-4db5-aeec-8a6a66d32808\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-200c58-7d66656f4c-8k98q" Oct 30 18:24:39.075366 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:39.075348 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e4d3b1c-e383-4db5-aeec-8a6a66d32808-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-200c58-7d66656f4c-8k98q\" (UID: \"9e4d3b1c-e383-4db5-aeec-8a6a66d32808\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-200c58-7d66656f4c-8k98q" Oct 30 18:24:39.093968 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:39.093938 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r9wz\" (UniqueName: \"kubernetes.io/projected/9e4d3b1c-e383-4db5-aeec-8a6a66d32808-kube-api-access-5r9wz\") pod \"payload-pre-processing-e2e-praxis-b-200c58-7d66656f4c-8k98q\" (UID: \"9e4d3b1c-e383-4db5-aeec-8a6a66d32808\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-200c58-7d66656f4c-8k98q" Oct 30 18:24:39.208312 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:39.208275 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-200c58-7c6bfb8c8b-ppvqf" Oct 30 18:24:39.300816 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:39.300725 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-200c58-7d66656f4c-8k98q" Oct 30 18:24:39.521058 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:39.521033 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-200c58-7c6bfb8c8b-ppvqf"] Oct 30 18:24:39.521432 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:24:39.521411 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f1c999a_d8e4_4ee1_8cca_f53f93eed216.slice/crio-ecd4fc5d689fb0a5193d5dc4aea5ed497d67ee350bf4dfc3304907b804838f89 WatchSource:0}: Error finding container ecd4fc5d689fb0a5193d5dc4aea5ed497d67ee350bf4dfc3304907b804838f89: Status 404 returned error can't find the container with id ecd4fc5d689fb0a5193d5dc4aea5ed497d67ee350bf4dfc3304907b804838f89 Oct 30 18:24:39.643412 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:24:39.643366 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e4d3b1c_e383_4db5_aeec_8a6a66d32808.slice/crio-337eddc122710c71ce758be9ebb79d943742cca8a88ab75d58cf50e3ada22863 WatchSource:0}: Error finding container 337eddc122710c71ce758be9ebb79d943742cca8a88ab75d58cf50e3ada22863: Status 404 returned error can't find the container with id 337eddc122710c71ce758be9ebb79d943742cca8a88ab75d58cf50e3ada22863 Oct 30 18:24:39.644244 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:39.644224 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-200c58-7d66656f4c-8k98q"] Oct 30 18:24:40.517249 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:40.517212 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-200c58-7d66656f4c-8k98q" event={"ID":"9e4d3b1c-e383-4db5-aeec-8a6a66d32808","Type":"ContainerStarted","Data":"0f31c7b74d23e79f7419b3c59e646d981db8b56cbd42ead3426ca08aac1dd4b6"} Oct 30 18:24:40.517739 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:40.517278 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-200c58-7d66656f4c-8k98q" event={"ID":"9e4d3b1c-e383-4db5-aeec-8a6a66d32808","Type":"ContainerStarted","Data":"337eddc122710c71ce758be9ebb79d943742cca8a88ab75d58cf50e3ada22863"} Oct 30 18:24:40.517739 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:40.517314 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-200c58-7d66656f4c-8k98q" Oct 30 18:24:40.518939 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:40.518917 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-200c58-7c6bfb8c8b-ppvqf" event={"ID":"0f1c999a-d8e4-4ee1-8cca-f53f93eed216","Type":"ContainerStarted","Data":"ede9700fe29c1e0fc3c58e85aa0652e83a5894d6bb337c65ff700f6cc78cb578"} Oct 30 18:24:40.519058 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:40.518962 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-200c58-7c6bfb8c8b-ppvqf" event={"ID":"0f1c999a-d8e4-4ee1-8cca-f53f93eed216","Type":"ContainerStarted","Data":"ecd4fc5d689fb0a5193d5dc4aea5ed497d67ee350bf4dfc3304907b804838f89"} Oct 30 18:24:40.519058 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:40.518989 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-200c58-7c6bfb8c8b-ppvqf" Oct 30 18:24:40.560007 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:40.559946 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-200c58-7d66656f4c-8k98q" podStartSLOduration=2.559928705 podStartE2EDuration="2.559928705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:40.555030982 +0000 UTC m=+1577.645093322" watchObservedRunningTime="2026-10-30 18:24:40.559928705 +0000 UTC m=+1577.649991046" Oct 30 18:24:40.597163 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:40.597070 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-200c58-7c6bfb8c8b-ppvqf" podStartSLOduration=2.597049641 podStartE2EDuration="2.597049641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:40.591740932 +0000 UTC m=+1577.681803270" watchObservedRunningTime="2026-10-30 18:24:40.597049641 +0000 UTC m=+1577.687111981" Oct 30 18:24:41.529608 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:41.529574 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-10f1a3-698744b678-gk8pv"] Oct 30 18:24:41.530073 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:41.529792 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-10f1a3-698744b678-gk8pv" podUID="de1d297f-6a7a-4466-ab3e-7582d6c63612" containerName="maas-api" containerID="cri-o://4a2cdb6dc6b4cd87986764150ef4eaa243a2c37fe39beae8177f86d1353e5761" gracePeriod=30 Oct 30 18:24:41.999167 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:41.997746 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-10f1a3-698744b678-gk8pv" Oct 30 18:24:42.196269 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:42.192256 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de1d297f-6a7a-4466-ab3e-7582d6c63612-metrics-certs\" (UniqueName: \"kubernetes.io/secret/de1d297f-6a7a-4466-ab3e-7582d6c63612-metrics-certs\") pod \"de1d297f-6a7a-4466-ab3e-7582d6c63612\" (UID: \"de1d297f-6a7a-4466-ab3e-7582d6c63612\") " Oct 30 18:24:42.202839 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:42.202296 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de1d297f-6a7a-4466-ab3e-7582d6c63612-metrics-certs" pod "de1d297f-6a7a-4466-ab3e-7582d6c63612" (UID: "de1d297f-6a7a-4466-ab3e-7582d6c63612"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:24:42.203423 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:42.203371 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de1d297f-6a7a-4466-ab3e-7582d6c63612-kube-api-access-ngxgp\" (UniqueName: \"kubernetes.io/projected/de1d297f-6a7a-4466-ab3e-7582d6c63612-kube-api-access-ngxgp\") pod \"de1d297f-6a7a-4466-ab3e-7582d6c63612\" (UID: \"de1d297f-6a7a-4466-ab3e-7582d6c63612\") " Oct 30 18:24:42.203553 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:42.203457 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de1d297f-6a7a-4466-ab3e-7582d6c63612-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de1d297f-6a7a-4466-ab3e-7582d6c63612-maas-api-tls\") pod \"de1d297f-6a7a-4466-ab3e-7582d6c63612\" (UID: \"de1d297f-6a7a-4466-ab3e-7582d6c63612\") " Oct 30 18:24:42.203553 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:42.203523 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de1d297f-6a7a-4466-ab3e-7582d6c63612-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de1d297f-6a7a-4466-ab3e-7582d6c63612-openshift-service-ca\") pod \"de1d297f-6a7a-4466-ab3e-7582d6c63612\" (UID: \"de1d297f-6a7a-4466-ab3e-7582d6c63612\") " Oct 30 18:24:42.204898 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:42.204867 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de1d297f-6a7a-4466-ab3e-7582d6c63612-openshift-service-ca" pod "de1d297f-6a7a-4466-ab3e-7582d6c63612" (UID: "de1d297f-6a7a-4466-ab3e-7582d6c63612"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:42.207052 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:42.207021 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de1d297f-6a7a-4466-ab3e-7582d6c63612-maas-api-tls" pod "de1d297f-6a7a-4466-ab3e-7582d6c63612" (UID: "de1d297f-6a7a-4466-ab3e-7582d6c63612"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:24:42.211247 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:42.211217 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de1d297f-6a7a-4466-ab3e-7582d6c63612-kube-api-access-ngxgp" pod "de1d297f-6a7a-4466-ab3e-7582d6c63612" (UID: "de1d297f-6a7a-4466-ab3e-7582d6c63612"). InnerVolumeSpecName "kube-api-access-ngxgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:42.304837 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:42.304807 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de1d297f-6a7a-4466-ab3e-7582d6c63612-metrics-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:42.304837 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:42.304835 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngxgp\" (UniqueName: \"kubernetes.io/projected/de1d297f-6a7a-4466-ab3e-7582d6c63612-kube-api-access-ngxgp\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:42.304837 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:42.304843 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de1d297f-6a7a-4466-ab3e-7582d6c63612-maas-api-tls\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:42.305072 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:42.304852 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de1d297f-6a7a-4466-ab3e-7582d6c63612-openshift-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:42.535942 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:42.535856 4442 generic.go:356] "Generic (PLEG): container finished" podID="de1d297f-6a7a-4466-ab3e-7582d6c63612" containerID="4a2cdb6dc6b4cd87986764150ef4eaa243a2c37fe39beae8177f86d1353e5761" exitCode=0 Oct 30 18:24:42.535942 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:42.535916 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-10f1a3-698744b678-gk8pv" Oct 30 18:24:42.536415 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:42.535937 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-10f1a3-698744b678-gk8pv" event={"ID":"de1d297f-6a7a-4466-ab3e-7582d6c63612","Type":"ContainerDied","Data":"4a2cdb6dc6b4cd87986764150ef4eaa243a2c37fe39beae8177f86d1353e5761"} Oct 30 18:24:42.536415 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:42.536013 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-10f1a3-698744b678-gk8pv" event={"ID":"de1d297f-6a7a-4466-ab3e-7582d6c63612","Type":"ContainerDied","Data":"f9faa8e85cc50a5a9db5d14aeda57ca0c328a0f930a5c82c0226eb85b182a756"} Oct 30 18:24:42.536415 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:42.536034 4442 scope.go:122] "RemoveContainer" containerID="4a2cdb6dc6b4cd87986764150ef4eaa243a2c37fe39beae8177f86d1353e5761" Oct 30 18:24:42.547356 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:42.547320 4442 scope.go:122] "RemoveContainer" containerID="4a2cdb6dc6b4cd87986764150ef4eaa243a2c37fe39beae8177f86d1353e5761" Oct 30 18:24:42.547681 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:24:42.547662 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a2cdb6dc6b4cd87986764150ef4eaa243a2c37fe39beae8177f86d1353e5761\": container with ID starting with 4a2cdb6dc6b4cd87986764150ef4eaa243a2c37fe39beae8177f86d1353e5761 not found: ID does not exist" containerID="4a2cdb6dc6b4cd87986764150ef4eaa243a2c37fe39beae8177f86d1353e5761" Oct 30 18:24:42.547746 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:42.547691 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a2cdb6dc6b4cd87986764150ef4eaa243a2c37fe39beae8177f86d1353e5761"} err="failed to get container status \"4a2cdb6dc6b4cd87986764150ef4eaa243a2c37fe39beae8177f86d1353e5761\": rpc error: code = NotFound desc = could not find container \"4a2cdb6dc6b4cd87986764150ef4eaa243a2c37fe39beae8177f86d1353e5761\": container with ID starting with 4a2cdb6dc6b4cd87986764150ef4eaa243a2c37fe39beae8177f86d1353e5761 not found: ID does not exist" Oct 30 18:24:42.588259 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:42.588228 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-10f1a3-698744b678-gk8pv"] Oct 30 18:24:42.599404 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:42.599380 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-10f1a3-698744b678-gk8pv"] Oct 30 18:24:43.068463 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:43.068426 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de1d297f-6a7a-4466-ab3e-7582d6c63612" path="/var/lib/kubelet/pods/de1d297f-6a7a-4466-ab3e-7582d6c63612/volumes" Oct 30 18:24:44.922872 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:44.922829 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt"] Oct 30 18:24:44.923417 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:44.923215 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" podUID="22d32885-e0c5-475e-b497-d10d09015d22" containerName="istio-proxy" containerID="cri-o://2101fe5edac5c2232d74426cce15c60f246e9922cca1f16dacf8c95827c658f6" gracePeriod=30 Oct 30 18:24:47.564815 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:24:47.564777 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:24:49.157031 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:49.156997 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx"] Oct 30 18:24:49.157562 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:49.157313 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" podUID="d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214" containerName="istio-proxy" containerID="cri-o://83874232eba21933c6e1384dcb9a6353a3c42b3a5e9328db265488473c7cb271" gracePeriod=30 Oct 30 18:24:50.234596 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:50.234564 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" Oct 30 18:24:50.291895 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:50.291849 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22d32885-e0c5-475e-b497-d10d09015d22-kube-api-access-gv5f4\" (UniqueName: \"kubernetes.io/projected/22d32885-e0c5-475e-b497-d10d09015d22-kube-api-access-gv5f4\") pod \"22d32885-e0c5-475e-b497-d10d09015d22\" (UID: \"22d32885-e0c5-475e-b497-d10d09015d22\") " Oct 30 18:24:50.292122 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:50.291946 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22d32885-e0c5-475e-b497-d10d09015d22-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22d32885-e0c5-475e-b497-d10d09015d22-istio-data\") pod \"22d32885-e0c5-475e-b497-d10d09015d22\" (UID: \"22d32885-e0c5-475e-b497-d10d09015d22\") " Oct 30 18:24:50.292122 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:50.292022 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22d32885-e0c5-475e-b497-d10d09015d22-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22d32885-e0c5-475e-b497-d10d09015d22-istiod-ca-cert\") pod \"22d32885-e0c5-475e-b497-d10d09015d22\" (UID: \"22d32885-e0c5-475e-b497-d10d09015d22\") " Oct 30 18:24:50.292259 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:50.292128 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22d32885-e0c5-475e-b497-d10d09015d22-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22d32885-e0c5-475e-b497-d10d09015d22-credential-socket\") pod \"22d32885-e0c5-475e-b497-d10d09015d22\" (UID: \"22d32885-e0c5-475e-b497-d10d09015d22\") " Oct 30 18:24:50.292259 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:50.292205 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22d32885-e0c5-475e-b497-d10d09015d22-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22d32885-e0c5-475e-b497-d10d09015d22-workload-socket\") pod \"22d32885-e0c5-475e-b497-d10d09015d22\" (UID: \"22d32885-e0c5-475e-b497-d10d09015d22\") " Oct 30 18:24:50.292259 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:50.292246 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22d32885-e0c5-475e-b497-d10d09015d22-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22d32885-e0c5-475e-b497-d10d09015d22-istio-podinfo\") pod \"22d32885-e0c5-475e-b497-d10d09015d22\" (UID: \"22d32885-e0c5-475e-b497-d10d09015d22\") " Oct 30 18:24:50.292375 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:50.292277 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22d32885-e0c5-475e-b497-d10d09015d22-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22d32885-e0c5-475e-b497-d10d09015d22-workload-certs\") pod \"22d32885-e0c5-475e-b497-d10d09015d22\" (UID: \"22d32885-e0c5-475e-b497-d10d09015d22\") " Oct 30 18:24:50.292375 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:50.292352 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22d32885-e0c5-475e-b497-d10d09015d22-istio-token\" (UniqueName: \"kubernetes.io/projected/22d32885-e0c5-475e-b497-d10d09015d22-istio-token\") pod \"22d32885-e0c5-475e-b497-d10d09015d22\" (UID: \"22d32885-e0c5-475e-b497-d10d09015d22\") " Oct 30 18:24:50.292478 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:50.292432 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22d32885-e0c5-475e-b497-d10d09015d22-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22d32885-e0c5-475e-b497-d10d09015d22-istio-envoy\") pod \"22d32885-e0c5-475e-b497-d10d09015d22\" (UID: \"22d32885-e0c5-475e-b497-d10d09015d22\") " Oct 30 18:24:50.292690 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:50.292660 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22d32885-e0c5-475e-b497-d10d09015d22-credential-socket" pod "22d32885-e0c5-475e-b497-d10d09015d22" (UID: "22d32885-e0c5-475e-b497-d10d09015d22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:50.292690 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:50.292670 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22d32885-e0c5-475e-b497-d10d09015d22-istiod-ca-cert" pod "22d32885-e0c5-475e-b497-d10d09015d22" (UID: "22d32885-e0c5-475e-b497-d10d09015d22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:50.292984 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:50.292959 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22d32885-e0c5-475e-b497-d10d09015d22-istio-data" pod "22d32885-e0c5-475e-b497-d10d09015d22" (UID: "22d32885-e0c5-475e-b497-d10d09015d22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:50.293068 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:50.293051 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22d32885-e0c5-475e-b497-d10d09015d22-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:50.293127 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:50.293076 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22d32885-e0c5-475e-b497-d10d09015d22-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:50.293236 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:50.293181 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22d32885-e0c5-475e-b497-d10d09015d22-workload-certs" pod "22d32885-e0c5-475e-b497-d10d09015d22" (UID: "22d32885-e0c5-475e-b497-d10d09015d22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:50.293838 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:50.293810 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22d32885-e0c5-475e-b497-d10d09015d22-workload-socket" pod "22d32885-e0c5-475e-b497-d10d09015d22" (UID: "22d32885-e0c5-475e-b497-d10d09015d22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:50.295298 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:50.295273 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22d32885-e0c5-475e-b497-d10d09015d22-kube-api-access-gv5f4" pod "22d32885-e0c5-475e-b497-d10d09015d22" (UID: "22d32885-e0c5-475e-b497-d10d09015d22"). InnerVolumeSpecName "kube-api-access-gv5f4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:50.295474 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:50.295457 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22d32885-e0c5-475e-b497-d10d09015d22-istio-token" pod "22d32885-e0c5-475e-b497-d10d09015d22" (UID: "22d32885-e0c5-475e-b497-d10d09015d22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:50.295571 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:50.295539 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22d32885-e0c5-475e-b497-d10d09015d22-istio-envoy" pod "22d32885-e0c5-475e-b497-d10d09015d22" (UID: "22d32885-e0c5-475e-b497-d10d09015d22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:50.296448 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:50.296425 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22d32885-e0c5-475e-b497-d10d09015d22-istio-podinfo" pod "22d32885-e0c5-475e-b497-d10d09015d22" (UID: "22d32885-e0c5-475e-b497-d10d09015d22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:24:50.394775 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:50.394741 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22d32885-e0c5-475e-b497-d10d09015d22-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:50.394775 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:50.394771 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv5f4\" (UniqueName: \"kubernetes.io/projected/22d32885-e0c5-475e-b497-d10d09015d22-kube-api-access-gv5f4\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:50.394775 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:50.394784 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22d32885-e0c5-475e-b497-d10d09015d22-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:50.395043 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:50.394793 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22d32885-e0c5-475e-b497-d10d09015d22-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:50.395043 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:50.394801 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22d32885-e0c5-475e-b497-d10d09015d22-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:50.395043 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:50.394810 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22d32885-e0c5-475e-b497-d10d09015d22-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:50.395043 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:50.394817 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22d32885-e0c5-475e-b497-d10d09015d22-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:50.600441 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:50.600363 4442 generic.go:356] "Generic (PLEG): container finished" podID="22d32885-e0c5-475e-b497-d10d09015d22" containerID="2101fe5edac5c2232d74426cce15c60f246e9922cca1f16dacf8c95827c658f6" exitCode=0 Oct 30 18:24:50.600441 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:50.600412 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" Oct 30 18:24:50.600620 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:50.600413 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" event={"ID":"22d32885-e0c5-475e-b497-d10d09015d22","Type":"ContainerDied","Data":"2101fe5edac5c2232d74426cce15c60f246e9922cca1f16dacf8c95827c658f6"} Oct 30 18:24:50.600620 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:50.600548 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt" event={"ID":"22d32885-e0c5-475e-b497-d10d09015d22","Type":"ContainerDied","Data":"990fdd4289c572c7cddf1d24338e5bb599e99fd5aea8abc5bd95997f33ea194c"} Oct 30 18:24:50.600620 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:50.600585 4442 scope.go:122] "RemoveContainer" containerID="2101fe5edac5c2232d74426cce15c60f246e9922cca1f16dacf8c95827c658f6" Oct 30 18:24:50.612068 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:50.612049 4442 scope.go:122] "RemoveContainer" containerID="77ad92f64f90785547d2304549fc0b52597b278094c6e0174453e4fab2ed6f21" Oct 30 18:24:50.623114 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:50.623093 4442 scope.go:122] "RemoveContainer" containerID="2101fe5edac5c2232d74426cce15c60f246e9922cca1f16dacf8c95827c658f6" Oct 30 18:24:50.623471 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:24:50.623443 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2101fe5edac5c2232d74426cce15c60f246e9922cca1f16dacf8c95827c658f6\": container with ID starting with 2101fe5edac5c2232d74426cce15c60f246e9922cca1f16dacf8c95827c658f6 not found: ID does not exist" containerID="2101fe5edac5c2232d74426cce15c60f246e9922cca1f16dacf8c95827c658f6" Oct 30 18:24:50.623559 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:50.623482 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2101fe5edac5c2232d74426cce15c60f246e9922cca1f16dacf8c95827c658f6"} err="failed to get container status \"2101fe5edac5c2232d74426cce15c60f246e9922cca1f16dacf8c95827c658f6\": rpc error: code = NotFound desc = could not find container \"2101fe5edac5c2232d74426cce15c60f246e9922cca1f16dacf8c95827c658f6\": container with ID starting with 2101fe5edac5c2232d74426cce15c60f246e9922cca1f16dacf8c95827c658f6 not found: ID does not exist" Oct 30 18:24:50.623559 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:50.623506 4442 scope.go:122] "RemoveContainer" containerID="77ad92f64f90785547d2304549fc0b52597b278094c6e0174453e4fab2ed6f21" Oct 30 18:24:50.623770 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:24:50.623753 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77ad92f64f90785547d2304549fc0b52597b278094c6e0174453e4fab2ed6f21\": container with ID starting with 77ad92f64f90785547d2304549fc0b52597b278094c6e0174453e4fab2ed6f21 not found: ID does not exist" containerID="77ad92f64f90785547d2304549fc0b52597b278094c6e0174453e4fab2ed6f21" Oct 30 18:24:50.623816 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:50.623778 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77ad92f64f90785547d2304549fc0b52597b278094c6e0174453e4fab2ed6f21"} err="failed to get container status \"77ad92f64f90785547d2304549fc0b52597b278094c6e0174453e4fab2ed6f21\": rpc error: code = NotFound desc = could not find container \"77ad92f64f90785547d2304549fc0b52597b278094c6e0174453e4fab2ed6f21\": container with ID starting with 77ad92f64f90785547d2304549fc0b52597b278094c6e0174453e4fab2ed6f21 not found: ID does not exist" Oct 30 18:24:50.654268 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:50.654232 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt"] Oct 30 18:24:50.658200 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:50.658171 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-76135e-openshift-default-fd74fffc5-fd2vt"] Oct 30 18:24:51.080273 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:51.080239 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22d32885-e0c5-475e-b497-d10d09015d22" path="/var/lib/kubelet/pods/22d32885-e0c5-475e-b497-d10d09015d22/volumes" Oct 30 18:24:51.530477 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:51.530450 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-200c58-7d66656f4c-8k98q" Oct 30 18:24:51.530886 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:51.530524 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-200c58-7c6bfb8c8b-ppvqf" Oct 30 18:24:52.260805 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:52.260766 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-50b278-99494fc99-wv8sj"] Oct 30 18:24:52.261870 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:52.261849 4442 cpu_manager.go:414] "containerMap: removing container" podUID="22d32885-e0c5-475e-b497-d10d09015d22" containerName="istio-proxy" Oct 30 18:24:52.261947 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:52.261874 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22d32885-e0c5-475e-b497-d10d09015d22" containerName="istio-proxy" Oct 30 18:24:52.261947 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:52.261908 4442 cpu_manager.go:414] "containerMap: removing container" podUID="de1d297f-6a7a-4466-ab3e-7582d6c63612" containerName="maas-api" Oct 30 18:24:52.261947 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:52.261916 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de1d297f-6a7a-4466-ab3e-7582d6c63612" containerName="maas-api" Oct 30 18:24:52.261947 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:52.261930 4442 cpu_manager.go:414] "containerMap: removing container" podUID="22d32885-e0c5-475e-b497-d10d09015d22" containerName="istio-proxy" Oct 30 18:24:52.261947 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:52.261940 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22d32885-e0c5-475e-b497-d10d09015d22" containerName="istio-proxy" Oct 30 18:24:52.262127 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:52.262115 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="22d32885-e0c5-475e-b497-d10d09015d22" containerName="istio-proxy" Oct 30 18:24:52.262179 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:52.262152 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="de1d297f-6a7a-4466-ab3e-7582d6c63612" containerName="maas-api" Oct 30 18:24:52.262179 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:52.262166 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="22d32885-e0c5-475e-b497-d10d09015d22" containerName="istio-proxy" Oct 30 18:24:52.268010 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:52.267988 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-50b278-99494fc99-wv8sj" Oct 30 18:24:52.281824 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:52.281795 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-50b278-99494fc99-wv8sj"] Oct 30 18:24:52.282309 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:52.282277 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-50b278\"" Oct 30 18:24:52.289075 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:52.289048 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-50b278-dockercfg-9vvnv\"" Oct 30 18:24:52.366156 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:52.359155 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzl57\" (UniqueName: \"kubernetes.io/projected/1f2c8508-03f8-4615-9c58-a9e4a4c0093b-kube-api-access-rzl57\") pod \"payload-pre-processing-e2e-praxis-a-50b278-99494fc99-wv8sj\" (UID: \"1f2c8508-03f8-4615-9c58-a9e4a4c0093b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-50b278-99494fc99-wv8sj" Oct 30 18:24:52.392507 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:52.392466 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f2c8508-03f8-4615-9c58-a9e4a4c0093b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-50b278-99494fc99-wv8sj\" (UID: \"1f2c8508-03f8-4615-9c58-a9e4a4c0093b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-50b278-99494fc99-wv8sj" Oct 30 18:24:52.513446 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:52.513360 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzl57\" (UniqueName: \"kubernetes.io/projected/1f2c8508-03f8-4615-9c58-a9e4a4c0093b-kube-api-access-rzl57\") pod \"payload-pre-processing-e2e-praxis-a-50b278-99494fc99-wv8sj\" (UID: \"1f2c8508-03f8-4615-9c58-a9e4a4c0093b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-50b278-99494fc99-wv8sj" Oct 30 18:24:52.516094 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:52.513786 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f2c8508-03f8-4615-9c58-a9e4a4c0093b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-50b278-99494fc99-wv8sj\" (UID: \"1f2c8508-03f8-4615-9c58-a9e4a4c0093b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-50b278-99494fc99-wv8sj" Oct 30 18:24:52.517455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:52.517431 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f2c8508-03f8-4615-9c58-a9e4a4c0093b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-50b278-99494fc99-wv8sj\" (UID: \"1f2c8508-03f8-4615-9c58-a9e4a4c0093b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-50b278-99494fc99-wv8sj" Oct 30 18:24:52.529236 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:52.529193 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-50b278-fd5695c7b-mdddw"] Oct 30 18:24:52.536089 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:52.534613 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-50b278-fd5695c7b-mdddw"] Oct 30 18:24:52.536089 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:52.534747 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-50b278-fd5695c7b-mdddw" Oct 30 18:24:52.538182 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:52.538157 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzl57\" (UniqueName: \"kubernetes.io/projected/1f2c8508-03f8-4615-9c58-a9e4a4c0093b-kube-api-access-rzl57\") pod \"payload-pre-processing-e2e-praxis-a-50b278-99494fc99-wv8sj\" (UID: \"1f2c8508-03f8-4615-9c58-a9e4a4c0093b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-50b278-99494fc99-wv8sj" Oct 30 18:24:52.584592 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:52.584545 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-50b278-99494fc99-wv8sj" Oct 30 18:24:52.718981 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:52.718942 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjjpm\" (UniqueName: \"kubernetes.io/projected/6567cc2e-3472-44f3-a360-5c08b825abf8-kube-api-access-hjjpm\") pod \"payload-processing-e2e-praxis-a-50b278-fd5695c7b-mdddw\" (UID: \"6567cc2e-3472-44f3-a360-5c08b825abf8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-50b278-fd5695c7b-mdddw" Oct 30 18:24:52.719210 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:52.719010 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6567cc2e-3472-44f3-a360-5c08b825abf8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-50b278-fd5695c7b-mdddw\" (UID: \"6567cc2e-3472-44f3-a360-5c08b825abf8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-50b278-fd5695c7b-mdddw" Oct 30 18:24:52.820056 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:52.819972 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjjpm\" (UniqueName: \"kubernetes.io/projected/6567cc2e-3472-44f3-a360-5c08b825abf8-kube-api-access-hjjpm\") pod \"payload-processing-e2e-praxis-a-50b278-fd5695c7b-mdddw\" (UID: \"6567cc2e-3472-44f3-a360-5c08b825abf8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-50b278-fd5695c7b-mdddw" Oct 30 18:24:52.820056 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:52.820017 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6567cc2e-3472-44f3-a360-5c08b825abf8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-50b278-fd5695c7b-mdddw\" (UID: \"6567cc2e-3472-44f3-a360-5c08b825abf8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-50b278-fd5695c7b-mdddw" Oct 30 18:24:52.820739 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:52.820716 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6567cc2e-3472-44f3-a360-5c08b825abf8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-50b278-fd5695c7b-mdddw\" (UID: \"6567cc2e-3472-44f3-a360-5c08b825abf8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-50b278-fd5695c7b-mdddw" Oct 30 18:24:52.839361 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:52.839325 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjjpm\" (UniqueName: \"kubernetes.io/projected/6567cc2e-3472-44f3-a360-5c08b825abf8-kube-api-access-hjjpm\") pod \"payload-processing-e2e-praxis-a-50b278-fd5695c7b-mdddw\" (UID: \"6567cc2e-3472-44f3-a360-5c08b825abf8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-50b278-fd5695c7b-mdddw" Oct 30 18:24:52.865168 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:52.865119 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-50b278-fd5695c7b-mdddw" Oct 30 18:24:52.900367 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:52.900334 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-50b278-99494fc99-wv8sj"] Oct 30 18:24:52.900828 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:24:52.900804 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f2c8508_03f8_4615_9c58_a9e4a4c0093b.slice/crio-014c97110c0db1645cfac001ae4b605a42c5d5c069be55b0fcf70e7b26d756df WatchSource:0}: Error finding container 014c97110c0db1645cfac001ae4b605a42c5d5c069be55b0fcf70e7b26d756df: Status 404 returned error can't find the container with id 014c97110c0db1645cfac001ae4b605a42c5d5c069be55b0fcf70e7b26d756df Oct 30 18:24:53.177040 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:53.177003 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-50b278-fd5695c7b-mdddw"] Oct 30 18:24:53.187617 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:24:53.187588 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6567cc2e_3472_44f3_a360_5c08b825abf8.slice/crio-a6ca581e6a773083db9b65ffa24c61100c4232e34260c295ecbf3dd68ed9d429 WatchSource:0}: Error finding container a6ca581e6a773083db9b65ffa24c61100c4232e34260c295ecbf3dd68ed9d429: Status 404 returned error can't find the container with id a6ca581e6a773083db9b65ffa24c61100c4232e34260c295ecbf3dd68ed9d429 Oct 30 18:24:53.631949 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:53.631917 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-50b278-fd5695c7b-mdddw" event={"ID":"6567cc2e-3472-44f3-a360-5c08b825abf8","Type":"ContainerStarted","Data":"2e74e652582d54d764d2f470cfb05249d7d4801e062d954add9534ecad969a24"} Oct 30 18:24:53.632433 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:53.631986 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-50b278-fd5695c7b-mdddw" Oct 30 18:24:53.632433 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:53.632010 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-50b278-fd5695c7b-mdddw" event={"ID":"6567cc2e-3472-44f3-a360-5c08b825abf8","Type":"ContainerStarted","Data":"a6ca581e6a773083db9b65ffa24c61100c4232e34260c295ecbf3dd68ed9d429"} Oct 30 18:24:53.634548 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:53.634522 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-50b278-99494fc99-wv8sj" event={"ID":"1f2c8508-03f8-4615-9c58-a9e4a4c0093b","Type":"ContainerStarted","Data":"124bcfbf713d2ef4ae5a6bb105d060fb2d437cd7c6ff9c79dea92f68c6ba0d65"} Oct 30 18:24:53.634677 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:53.634570 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-50b278-99494fc99-wv8sj" event={"ID":"1f2c8508-03f8-4615-9c58-a9e4a4c0093b","Type":"ContainerStarted","Data":"014c97110c0db1645cfac001ae4b605a42c5d5c069be55b0fcf70e7b26d756df"} Oct 30 18:24:53.634677 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:53.634612 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-50b278-99494fc99-wv8sj" Oct 30 18:24:53.672901 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:53.672853 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-50b278-fd5695c7b-mdddw" podStartSLOduration=1.672837544 podStartE2EDuration="1.672837544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:53.668823104 +0000 UTC m=+1590.758885444" watchObservedRunningTime="2026-10-30 18:24:53.672837544 +0000 UTC m=+1590.762899883" Oct 30 18:24:53.710780 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:53.710716 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-50b278-99494fc99-wv8sj" podStartSLOduration=1.710691063 podStartE2EDuration="1.710691063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:53.704592476 +0000 UTC m=+1590.794654815" watchObservedRunningTime="2026-10-30 18:24:53.710691063 +0000 UTC m=+1590.800753403" Oct 30 18:24:54.456337 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:54.456306 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" Oct 30 18:24:54.566479 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:54.566439 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-workload-socket\") pod \"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214\" (UID: \"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214\") " Oct 30 18:24:54.566716 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:54.566528 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-workload-certs\") pod \"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214\" (UID: \"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214\") " Oct 30 18:24:54.566716 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:54.566585 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-istio-token\" (UniqueName: \"kubernetes.io/projected/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-istio-token\") pod \"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214\" (UID: \"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214\") " Oct 30 18:24:54.566716 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:54.566683 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-credential-socket\") pod \"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214\" (UID: \"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214\") " Oct 30 18:24:54.566911 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:54.566724 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-istio-podinfo\") pod \"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214\" (UID: \"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214\") " Oct 30 18:24:54.566911 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:54.566763 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-istio-envoy\") pod \"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214\" (UID: \"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214\") " Oct 30 18:24:54.566911 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:54.566864 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-kube-api-access-9t7sj\" (UniqueName: \"kubernetes.io/projected/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-kube-api-access-9t7sj\") pod \"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214\" (UID: \"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214\") " Oct 30 18:24:54.567067 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:54.566916 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-istiod-ca-cert\") pod \"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214\" (UID: \"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214\") " Oct 30 18:24:54.567067 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:54.566972 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-istio-data\") pod \"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214\" (UID: \"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214\") " Oct 30 18:24:54.568183 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:54.568118 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-workload-certs" pod "d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214" (UID: "d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:54.568335 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:54.568301 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-workload-socket" pod "d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214" (UID: "d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:54.569749 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:54.569721 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-istiod-ca-cert" pod "d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214" (UID: "d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:54.570299 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:54.570238 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-credential-socket" pod "d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214" (UID: "d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:54.570909 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:54.570875 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-istio-data" pod "d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214" (UID: "d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:54.572824 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:54.572798 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-istio-token" pod "d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214" (UID: "d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:54.574401 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:54.574382 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-istio-envoy" pod "d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214" (UID: "d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:54.578183 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:54.576291 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-kube-api-access-9t7sj" pod "d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214" (UID: "d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214"). InnerVolumeSpecName "kube-api-access-9t7sj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:54.578183 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:54.576506 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-istio-podinfo" pod "d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214" (UID: "d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:24:54.645535 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:54.645502 4442 generic.go:356] "Generic (PLEG): container finished" podID="d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214" containerID="83874232eba21933c6e1384dcb9a6353a3c42b3a5e9328db265488473c7cb271" exitCode=0 Oct 30 18:24:54.646019 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:54.645565 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" Oct 30 18:24:54.646019 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:54.645589 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" event={"ID":"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214","Type":"ContainerDied","Data":"83874232eba21933c6e1384dcb9a6353a3c42b3a5e9328db265488473c7cb271"} Oct 30 18:24:54.646019 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:54.645689 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx" event={"ID":"d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214","Type":"ContainerDied","Data":"37beac749322b5d49a607bfe9c0095d4ebf13dbb436838b83ca71e5a0fa39161"} Oct 30 18:24:54.646019 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:54.645719 4442 scope.go:122] "RemoveContainer" containerID="83874232eba21933c6e1384dcb9a6353a3c42b3a5e9328db265488473c7cb271" Oct 30 18:24:54.658923 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:54.658900 4442 scope.go:122] "RemoveContainer" containerID="fe437537e5a1ea841df80e36da3e51f00839aa35a49ad76b6cf07e0236736778" Oct 30 18:24:54.669252 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:54.669222 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:54.669252 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:54.669255 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:54.669464 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:54.669268 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:54.669464 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:54.669277 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t7sj\" (UniqueName: \"kubernetes.io/projected/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-kube-api-access-9t7sj\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:54.669464 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:54.669287 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:54.669464 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:54.669294 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:54.669464 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:54.669301 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:54.669464 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:54.669309 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:54.669464 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:54.669316 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:54.673282 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:54.673263 4442 scope.go:122] "RemoveContainer" containerID="83874232eba21933c6e1384dcb9a6353a3c42b3a5e9328db265488473c7cb271" Oct 30 18:24:54.673672 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:24:54.673643 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83874232eba21933c6e1384dcb9a6353a3c42b3a5e9328db265488473c7cb271\": container with ID starting with 83874232eba21933c6e1384dcb9a6353a3c42b3a5e9328db265488473c7cb271 not found: ID does not exist" containerID="83874232eba21933c6e1384dcb9a6353a3c42b3a5e9328db265488473c7cb271" Oct 30 18:24:54.673783 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:54.673683 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83874232eba21933c6e1384dcb9a6353a3c42b3a5e9328db265488473c7cb271"} err="failed to get container status \"83874232eba21933c6e1384dcb9a6353a3c42b3a5e9328db265488473c7cb271\": rpc error: code = NotFound desc = could not find container \"83874232eba21933c6e1384dcb9a6353a3c42b3a5e9328db265488473c7cb271\": container with ID starting with 83874232eba21933c6e1384dcb9a6353a3c42b3a5e9328db265488473c7cb271 not found: ID does not exist" Oct 30 18:24:54.673783 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:54.673711 4442 scope.go:122] "RemoveContainer" containerID="fe437537e5a1ea841df80e36da3e51f00839aa35a49ad76b6cf07e0236736778" Oct 30 18:24:54.674112 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:24:54.674084 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe437537e5a1ea841df80e36da3e51f00839aa35a49ad76b6cf07e0236736778\": container with ID starting with fe437537e5a1ea841df80e36da3e51f00839aa35a49ad76b6cf07e0236736778 not found: ID does not exist" containerID="fe437537e5a1ea841df80e36da3e51f00839aa35a49ad76b6cf07e0236736778" Oct 30 18:24:54.674349 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:54.674121 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe437537e5a1ea841df80e36da3e51f00839aa35a49ad76b6cf07e0236736778"} err="failed to get container status \"fe437537e5a1ea841df80e36da3e51f00839aa35a49ad76b6cf07e0236736778\": rpc error: code = NotFound desc = could not find container \"fe437537e5a1ea841df80e36da3e51f00839aa35a49ad76b6cf07e0236736778\": container with ID starting with fe437537e5a1ea841df80e36da3e51f00839aa35a49ad76b6cf07e0236736778 not found: ID does not exist" Oct 30 18:24:54.706106 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:54.706076 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx"] Oct 30 18:24:54.709792 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:54.709770 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-10f1a3-openshift-default-7ddf867857-rspfx"] Oct 30 18:24:55.068906 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:55.068825 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214" path="/var/lib/kubelet/pods/d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214/volumes" Oct 30 18:24:56.657568 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:56.657361 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-011d6f-8458db466f-xvrj5"] Oct 30 18:24:56.658042 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:56.657611 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-011d6f-8458db466f-xvrj5" podUID="5d86cecf-4477-44f5-82b7-328fa9a81d50" containerName="maas-api" containerID="cri-o://f9dcf823aa442002b866842159c8a460da783a08683dc074dffffe942ad42cce" gracePeriod=30 Oct 30 18:24:56.683996 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:56.683957 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-011d6f-8458db466f-xvrj5" podUID="5d86cecf-4477-44f5-82b7-328fa9a81d50" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.128:8443/health\": dial tcp 10.134.0.128:8443: connect: connection refused" Oct 30 18:24:56.952394 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:56.952361 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-011d6f-8458db466f-xvrj5" Oct 30 18:24:56.999924 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:56.999613 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d86cecf-4477-44f5-82b7-328fa9a81d50-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d86cecf-4477-44f5-82b7-328fa9a81d50-metrics-certs\") pod \"5d86cecf-4477-44f5-82b7-328fa9a81d50\" (UID: \"5d86cecf-4477-44f5-82b7-328fa9a81d50\") " Oct 30 18:24:56.999924 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:56.999867 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d86cecf-4477-44f5-82b7-328fa9a81d50-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d86cecf-4477-44f5-82b7-328fa9a81d50-maas-api-tls\") pod \"5d86cecf-4477-44f5-82b7-328fa9a81d50\" (UID: \"5d86cecf-4477-44f5-82b7-328fa9a81d50\") " Oct 30 18:24:57.000230 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:56.999970 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d86cecf-4477-44f5-82b7-328fa9a81d50-kube-api-access-pdcp5\" (UniqueName: \"kubernetes.io/projected/5d86cecf-4477-44f5-82b7-328fa9a81d50-kube-api-access-pdcp5\") pod \"5d86cecf-4477-44f5-82b7-328fa9a81d50\" (UID: \"5d86cecf-4477-44f5-82b7-328fa9a81d50\") " Oct 30 18:24:57.000230 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:57.000051 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d86cecf-4477-44f5-82b7-328fa9a81d50-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d86cecf-4477-44f5-82b7-328fa9a81d50-openshift-service-ca\") pod \"5d86cecf-4477-44f5-82b7-328fa9a81d50\" (UID: \"5d86cecf-4477-44f5-82b7-328fa9a81d50\") " Oct 30 18:24:57.000668 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:57.000641 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d86cecf-4477-44f5-82b7-328fa9a81d50-openshift-service-ca" pod "5d86cecf-4477-44f5-82b7-328fa9a81d50" (UID: "5d86cecf-4477-44f5-82b7-328fa9a81d50"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:57.000863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:57.000779 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d86cecf-4477-44f5-82b7-328fa9a81d50-openshift-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:57.012292 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:57.012262 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d86cecf-4477-44f5-82b7-328fa9a81d50-maas-api-tls" pod "5d86cecf-4477-44f5-82b7-328fa9a81d50" (UID: "5d86cecf-4477-44f5-82b7-328fa9a81d50"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:24:57.012448 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:57.012335 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d86cecf-4477-44f5-82b7-328fa9a81d50-metrics-certs" pod "5d86cecf-4477-44f5-82b7-328fa9a81d50" (UID: "5d86cecf-4477-44f5-82b7-328fa9a81d50"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:24:57.012448 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:57.012424 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d86cecf-4477-44f5-82b7-328fa9a81d50-kube-api-access-pdcp5" pod "5d86cecf-4477-44f5-82b7-328fa9a81d50" (UID: "5d86cecf-4477-44f5-82b7-328fa9a81d50"). InnerVolumeSpecName "kube-api-access-pdcp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:57.101687 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:57.101658 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdcp5\" (UniqueName: \"kubernetes.io/projected/5d86cecf-4477-44f5-82b7-328fa9a81d50-kube-api-access-pdcp5\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:57.101687 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:57.101684 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d86cecf-4477-44f5-82b7-328fa9a81d50-metrics-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:57.101866 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:57.101694 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d86cecf-4477-44f5-82b7-328fa9a81d50-maas-api-tls\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:24:57.679445 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:57.679414 4442 generic.go:356] "Generic (PLEG): container finished" podID="5d86cecf-4477-44f5-82b7-328fa9a81d50" containerID="f9dcf823aa442002b866842159c8a460da783a08683dc074dffffe942ad42cce" exitCode=0 Oct 30 18:24:57.679860 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:57.679481 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-011d6f-8458db466f-xvrj5" Oct 30 18:24:57.679860 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:57.679500 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-011d6f-8458db466f-xvrj5" event={"ID":"5d86cecf-4477-44f5-82b7-328fa9a81d50","Type":"ContainerDied","Data":"f9dcf823aa442002b866842159c8a460da783a08683dc074dffffe942ad42cce"} Oct 30 18:24:57.679860 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:57.679555 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-011d6f-8458db466f-xvrj5" event={"ID":"5d86cecf-4477-44f5-82b7-328fa9a81d50","Type":"ContainerDied","Data":"0bee74b7940ee88171160e0a7b6d3cac6803ff0d4a56b7c6c4f61b771c48dfa2"} Oct 30 18:24:57.679860 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:57.679582 4442 scope.go:122] "RemoveContainer" containerID="f9dcf823aa442002b866842159c8a460da783a08683dc074dffffe942ad42cce" Oct 30 18:24:57.692424 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:57.692405 4442 scope.go:122] "RemoveContainer" containerID="f9dcf823aa442002b866842159c8a460da783a08683dc074dffffe942ad42cce" Oct 30 18:24:57.692782 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:24:57.692762 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9dcf823aa442002b866842159c8a460da783a08683dc074dffffe942ad42cce\": container with ID starting with f9dcf823aa442002b866842159c8a460da783a08683dc074dffffe942ad42cce not found: ID does not exist" containerID="f9dcf823aa442002b866842159c8a460da783a08683dc074dffffe942ad42cce" Oct 30 18:24:57.692846 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:57.692790 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9dcf823aa442002b866842159c8a460da783a08683dc074dffffe942ad42cce"} err="failed to get container status \"f9dcf823aa442002b866842159c8a460da783a08683dc074dffffe942ad42cce\": rpc error: code = NotFound desc = could not find container \"f9dcf823aa442002b866842159c8a460da783a08683dc074dffffe942ad42cce\": container with ID starting with f9dcf823aa442002b866842159c8a460da783a08683dc074dffffe942ad42cce not found: ID does not exist" Oct 30 18:24:57.756559 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:57.756192 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-011d6f-8458db466f-xvrj5"] Oct 30 18:24:57.806167 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:57.805842 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-011d6f-8458db466f-xvrj5"] Oct 30 18:24:59.066623 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:24:59.066588 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d86cecf-4477-44f5-82b7-328fa9a81d50" path="/var/lib/kubelet/pods/5d86cecf-4477-44f5-82b7-328fa9a81d50/volumes" Oct 30 18:25:00.195461 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:00.195427 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0ac6bb-86c4fbf5cb-x5t46"] Oct 30 18:25:00.233980 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:00.233951 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0ac6bb-67fc4d5447-5n5xd"] Oct 30 18:25:01.869733 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:01.869694 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-200c58-844699bd54-xg4s6"] Oct 30 18:25:01.870633 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:01.870612 4442 cpu_manager.go:414] "containerMap: removing container" podUID="d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214" containerName="istio-proxy" Oct 30 18:25:01.870633 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:01.870633 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214" containerName="istio-proxy" Oct 30 18:25:01.870815 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:01.870652 4442 cpu_manager.go:414] "containerMap: removing container" podUID="d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214" containerName="istio-proxy" Oct 30 18:25:01.870815 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:01.870661 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214" containerName="istio-proxy" Oct 30 18:25:01.870815 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:01.870679 4442 cpu_manager.go:414] "containerMap: removing container" podUID="5d86cecf-4477-44f5-82b7-328fa9a81d50" containerName="maas-api" Oct 30 18:25:01.870815 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:01.870687 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d86cecf-4477-44f5-82b7-328fa9a81d50" containerName="maas-api" Oct 30 18:25:01.871002 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:01.870869 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214" containerName="istio-proxy" Oct 30 18:25:01.871002 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:01.870885 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d86cecf-4477-44f5-82b7-328fa9a81d50" containerName="maas-api" Oct 30 18:25:01.876629 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:01.876606 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-200c58-844699bd54-xg4s6" Oct 30 18:25:01.892965 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:01.892936 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-200c58-844699bd54-xg4s6"] Oct 30 18:25:01.899692 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:01.899668 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-200c58-serving-cert\"" Oct 30 18:25:01.903180 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:01.903118 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddc07105-e8f9-446d-b932-ab76d5471d87-metrics-certs\") pod \"maas-api-e2e-praxis-b-200c58-844699bd54-xg4s6\" (UID: \"ddc07105-e8f9-446d-b932-ab76d5471d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-200c58-844699bd54-xg4s6" Oct 30 18:25:01.903348 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:01.903310 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddc07105-e8f9-446d-b932-ab76d5471d87-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-200c58-844699bd54-xg4s6\" (UID: \"ddc07105-e8f9-446d-b932-ab76d5471d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-200c58-844699bd54-xg4s6" Oct 30 18:25:01.903422 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:01.903353 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddc07105-e8f9-446d-b932-ab76d5471d87-maas-api-tls\") pod \"maas-api-e2e-praxis-b-200c58-844699bd54-xg4s6\" (UID: \"ddc07105-e8f9-446d-b932-ab76d5471d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-200c58-844699bd54-xg4s6" Oct 30 18:25:01.903422 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:01.903386 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtdkr\" (UniqueName: \"kubernetes.io/projected/ddc07105-e8f9-446d-b932-ab76d5471d87-kube-api-access-jtdkr\") pod \"maas-api-e2e-praxis-b-200c58-844699bd54-xg4s6\" (UID: \"ddc07105-e8f9-446d-b932-ab76d5471d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-200c58-844699bd54-xg4s6" Oct 30 18:25:02.004755 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:02.004714 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddc07105-e8f9-446d-b932-ab76d5471d87-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-200c58-844699bd54-xg4s6\" (UID: \"ddc07105-e8f9-446d-b932-ab76d5471d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-200c58-844699bd54-xg4s6" Oct 30 18:25:02.004755 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:02.004759 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddc07105-e8f9-446d-b932-ab76d5471d87-maas-api-tls\") pod \"maas-api-e2e-praxis-b-200c58-844699bd54-xg4s6\" (UID: \"ddc07105-e8f9-446d-b932-ab76d5471d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-200c58-844699bd54-xg4s6" Oct 30 18:25:02.005012 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:02.004781 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtdkr\" (UniqueName: \"kubernetes.io/projected/ddc07105-e8f9-446d-b932-ab76d5471d87-kube-api-access-jtdkr\") pod \"maas-api-e2e-praxis-b-200c58-844699bd54-xg4s6\" (UID: \"ddc07105-e8f9-446d-b932-ab76d5471d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-200c58-844699bd54-xg4s6" Oct 30 18:25:02.005012 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:02.004870 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddc07105-e8f9-446d-b932-ab76d5471d87-metrics-certs\") pod \"maas-api-e2e-praxis-b-200c58-844699bd54-xg4s6\" (UID: \"ddc07105-e8f9-446d-b932-ab76d5471d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-200c58-844699bd54-xg4s6" Oct 30 18:25:02.005419 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:02.005396 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddc07105-e8f9-446d-b932-ab76d5471d87-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-200c58-844699bd54-xg4s6\" (UID: \"ddc07105-e8f9-446d-b932-ab76d5471d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-200c58-844699bd54-xg4s6" Oct 30 18:25:02.007530 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:02.007501 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddc07105-e8f9-446d-b932-ab76d5471d87-metrics-certs\") pod \"maas-api-e2e-praxis-b-200c58-844699bd54-xg4s6\" (UID: \"ddc07105-e8f9-446d-b932-ab76d5471d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-200c58-844699bd54-xg4s6" Oct 30 18:25:02.007632 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:02.007541 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddc07105-e8f9-446d-b932-ab76d5471d87-maas-api-tls\") pod \"maas-api-e2e-praxis-b-200c58-844699bd54-xg4s6\" (UID: \"ddc07105-e8f9-446d-b932-ab76d5471d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-200c58-844699bd54-xg4s6" Oct 30 18:25:02.023793 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:02.023757 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtdkr\" (UniqueName: \"kubernetes.io/projected/ddc07105-e8f9-446d-b932-ab76d5471d87-kube-api-access-jtdkr\") pod \"maas-api-e2e-praxis-b-200c58-844699bd54-xg4s6\" (UID: \"ddc07105-e8f9-446d-b932-ab76d5471d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-200c58-844699bd54-xg4s6" Oct 30 18:25:02.196096 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:02.195970 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-200c58-844699bd54-xg4s6" Oct 30 18:25:02.542383 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:02.542349 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-200c58-844699bd54-xg4s6"] Oct 30 18:25:02.543637 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:25:02.543600 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddc07105_e8f9_446d_b932_ab76d5471d87.slice/crio-3057fa0eaa2ceb9d261723336829dc85d804a4c8d33a87b6848eaa52aabd9885 WatchSource:0}: Error finding container 3057fa0eaa2ceb9d261723336829dc85d804a4c8d33a87b6848eaa52aabd9885: Status 404 returned error can't find the container with id 3057fa0eaa2ceb9d261723336829dc85d804a4c8d33a87b6848eaa52aabd9885 Oct 30 18:25:02.729993 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:02.729950 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-200c58-844699bd54-xg4s6" event={"ID":"ddc07105-e8f9-446d-b932-ab76d5471d87","Type":"ContainerStarted","Data":"3057fa0eaa2ceb9d261723336829dc85d804a4c8d33a87b6848eaa52aabd9885"} Oct 30 18:25:03.784162 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:03.783258 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-200c58-844699bd54-xg4s6" event={"ID":"ddc07105-e8f9-446d-b932-ab76d5471d87","Type":"ContainerStarted","Data":"c6641aaaffbfcca844d2de0a231a8c04b8cb94797afbba026a44645cc4dc1dae"} Oct 30 18:25:03.784647 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:03.784366 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-200c58-844699bd54-xg4s6" Oct 30 18:25:03.848557 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:03.848489 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-200c58-844699bd54-xg4s6" podStartSLOduration=2.388947238 podStartE2EDuration="2.848468605s" totalImagesPullingTime="459.521367ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:25:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:25:03.845844931 +0000 UTC m=+1600.935907274" watchObservedRunningTime="2026-10-30 18:25:03.848468605 +0000 UTC m=+1600.938530945" Oct 30 18:25:04.650326 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:04.650298 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-50b278-99494fc99-wv8sj" Oct 30 18:25:04.650750 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:04.650729 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-50b278-fd5695c7b-mdddw" Oct 30 18:25:05.195940 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:05.195890 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-0ac6bb-86c4fbf5cb-x5t46" podUID="141a5be6-7d7b-4199-b58f-9e6831a1ac7b" containerName="payload-processing" containerID="cri-o://bea746ed9299c87c0ef6a391cd6664b36646177047d1c3f89c0bf08f893c709a" gracePeriod=25 Oct 30 18:25:05.505928 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:05.505895 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0ac6bb-86c4fbf5cb-x5t46" Oct 30 18:25:05.702071 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:05.702035 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/141a5be6-7d7b-4199-b58f-9e6831a1ac7b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/141a5be6-7d7b-4199-b58f-9e6831a1ac7b-plugins-config-volume\") pod \"141a5be6-7d7b-4199-b58f-9e6831a1ac7b\" (UID: \"141a5be6-7d7b-4199-b58f-9e6831a1ac7b\") " Oct 30 18:25:05.702305 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:05.702114 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/141a5be6-7d7b-4199-b58f-9e6831a1ac7b-kube-api-access-tppvd\" (UniqueName: \"kubernetes.io/projected/141a5be6-7d7b-4199-b58f-9e6831a1ac7b-kube-api-access-tppvd\") pod \"141a5be6-7d7b-4199-b58f-9e6831a1ac7b\" (UID: \"141a5be6-7d7b-4199-b58f-9e6831a1ac7b\") " Oct 30 18:25:05.702513 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:05.702486 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/141a5be6-7d7b-4199-b58f-9e6831a1ac7b-plugins-config-volume" pod "141a5be6-7d7b-4199-b58f-9e6831a1ac7b" (UID: "141a5be6-7d7b-4199-b58f-9e6831a1ac7b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:05.702813 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:05.702794 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/141a5be6-7d7b-4199-b58f-9e6831a1ac7b-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:25:05.704536 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:05.704489 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0ac6bb-67fc4d5447-5n5xd" podUID="35427437-feb0-490d-966e-8978f4cfe11e" containerName="payload-pre-processing" containerID="cri-o://88fbf6a636da6a7cf13c3195268205ca2eff17d7c6b2a1b6d24ef1ed455c0ac5" gracePeriod=25 Oct 30 18:25:05.704633 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:05.704556 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/141a5be6-7d7b-4199-b58f-9e6831a1ac7b-kube-api-access-tppvd" pod "141a5be6-7d7b-4199-b58f-9e6831a1ac7b" (UID: "141a5be6-7d7b-4199-b58f-9e6831a1ac7b"). InnerVolumeSpecName "kube-api-access-tppvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:05.801929 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:05.801894 4442 generic.go:356] "Generic (PLEG): container finished" podID="141a5be6-7d7b-4199-b58f-9e6831a1ac7b" containerID="bea746ed9299c87c0ef6a391cd6664b36646177047d1c3f89c0bf08f893c709a" exitCode=0 Oct 30 18:25:05.802115 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:05.801938 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0ac6bb-86c4fbf5cb-x5t46" Oct 30 18:25:05.802115 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:05.801975 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0ac6bb-86c4fbf5cb-x5t46" event={"ID":"141a5be6-7d7b-4199-b58f-9e6831a1ac7b","Type":"ContainerDied","Data":"bea746ed9299c87c0ef6a391cd6664b36646177047d1c3f89c0bf08f893c709a"} Oct 30 18:25:05.802115 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:05.802032 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0ac6bb-86c4fbf5cb-x5t46" event={"ID":"141a5be6-7d7b-4199-b58f-9e6831a1ac7b","Type":"ContainerDied","Data":"6053dca6b85dd59af437ee1e8619428f831c9cc16d05a821d47b235b308e6a6c"} Oct 30 18:25:05.802115 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:05.802062 4442 scope.go:122] "RemoveContainer" containerID="bea746ed9299c87c0ef6a391cd6664b36646177047d1c3f89c0bf08f893c709a" Oct 30 18:25:05.803534 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:05.803516 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tppvd\" (UniqueName: \"kubernetes.io/projected/141a5be6-7d7b-4199-b58f-9e6831a1ac7b-kube-api-access-tppvd\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:25:05.820896 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:05.820877 4442 scope.go:122] "RemoveContainer" containerID="bea746ed9299c87c0ef6a391cd6664b36646177047d1c3f89c0bf08f893c709a" Oct 30 18:25:05.821287 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:25:05.821267 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bea746ed9299c87c0ef6a391cd6664b36646177047d1c3f89c0bf08f893c709a\": container with ID starting with bea746ed9299c87c0ef6a391cd6664b36646177047d1c3f89c0bf08f893c709a not found: ID does not exist" containerID="bea746ed9299c87c0ef6a391cd6664b36646177047d1c3f89c0bf08f893c709a" Oct 30 18:25:05.821344 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:05.821296 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bea746ed9299c87c0ef6a391cd6664b36646177047d1c3f89c0bf08f893c709a"} err="failed to get container status \"bea746ed9299c87c0ef6a391cd6664b36646177047d1c3f89c0bf08f893c709a\": rpc error: code = NotFound desc = could not find container \"bea746ed9299c87c0ef6a391cd6664b36646177047d1c3f89c0bf08f893c709a\": container with ID starting with bea746ed9299c87c0ef6a391cd6664b36646177047d1c3f89c0bf08f893c709a not found: ID does not exist" Oct 30 18:25:05.854755 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:05.854725 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0ac6bb-86c4fbf5cb-x5t46"] Oct 30 18:25:05.859093 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:05.859064 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0ac6bb-86c4fbf5cb-x5t46"] Oct 30 18:25:06.067622 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:06.067584 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0ac6bb-67fc4d5447-5n5xd" Oct 30 18:25:06.233701 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:06.233668 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35427437-feb0-490d-966e-8978f4cfe11e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35427437-feb0-490d-966e-8978f4cfe11e-plugins-config-volume\") pod \"35427437-feb0-490d-966e-8978f4cfe11e\" (UID: \"35427437-feb0-490d-966e-8978f4cfe11e\") " Oct 30 18:25:06.234062 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:06.233833 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35427437-feb0-490d-966e-8978f4cfe11e-kube-api-access-5mlf5\" (UniqueName: \"kubernetes.io/projected/35427437-feb0-490d-966e-8978f4cfe11e-kube-api-access-5mlf5\") pod \"35427437-feb0-490d-966e-8978f4cfe11e\" (UID: \"35427437-feb0-490d-966e-8978f4cfe11e\") " Oct 30 18:25:06.234118 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:06.234066 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35427437-feb0-490d-966e-8978f4cfe11e-plugins-config-volume" pod "35427437-feb0-490d-966e-8978f4cfe11e" (UID: "35427437-feb0-490d-966e-8978f4cfe11e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:06.234399 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:06.234374 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35427437-feb0-490d-966e-8978f4cfe11e-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:25:06.236028 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:06.236007 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35427437-feb0-490d-966e-8978f4cfe11e-kube-api-access-5mlf5" pod "35427437-feb0-490d-966e-8978f4cfe11e" (UID: "35427437-feb0-490d-966e-8978f4cfe11e"). InnerVolumeSpecName "kube-api-access-5mlf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:06.336247 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:06.336158 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mlf5\" (UniqueName: \"kubernetes.io/projected/35427437-feb0-490d-966e-8978f4cfe11e-kube-api-access-5mlf5\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:25:06.812805 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:06.812769 4442 generic.go:356] "Generic (PLEG): container finished" podID="35427437-feb0-490d-966e-8978f4cfe11e" containerID="88fbf6a636da6a7cf13c3195268205ca2eff17d7c6b2a1b6d24ef1ed455c0ac5" exitCode=0 Oct 30 18:25:06.812994 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:06.812843 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0ac6bb-67fc4d5447-5n5xd" Oct 30 18:25:06.812994 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:06.812848 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0ac6bb-67fc4d5447-5n5xd" event={"ID":"35427437-feb0-490d-966e-8978f4cfe11e","Type":"ContainerDied","Data":"88fbf6a636da6a7cf13c3195268205ca2eff17d7c6b2a1b6d24ef1ed455c0ac5"} Oct 30 18:25:06.812994 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:06.812942 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0ac6bb-67fc4d5447-5n5xd" event={"ID":"35427437-feb0-490d-966e-8978f4cfe11e","Type":"ContainerDied","Data":"b86deff7d5499e86eb0a577c6f88eb5430052ce9f1d51bf2e184fd323a403fd6"} Oct 30 18:25:06.812994 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:06.812968 4442 scope.go:122] "RemoveContainer" containerID="88fbf6a636da6a7cf13c3195268205ca2eff17d7c6b2a1b6d24ef1ed455c0ac5" Oct 30 18:25:06.825902 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:06.825879 4442 scope.go:122] "RemoveContainer" containerID="88fbf6a636da6a7cf13c3195268205ca2eff17d7c6b2a1b6d24ef1ed455c0ac5" Oct 30 18:25:06.826230 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:25:06.826206 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88fbf6a636da6a7cf13c3195268205ca2eff17d7c6b2a1b6d24ef1ed455c0ac5\": container with ID starting with 88fbf6a636da6a7cf13c3195268205ca2eff17d7c6b2a1b6d24ef1ed455c0ac5 not found: ID does not exist" containerID="88fbf6a636da6a7cf13c3195268205ca2eff17d7c6b2a1b6d24ef1ed455c0ac5" Oct 30 18:25:06.826304 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:06.826240 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88fbf6a636da6a7cf13c3195268205ca2eff17d7c6b2a1b6d24ef1ed455c0ac5"} err="failed to get container status \"88fbf6a636da6a7cf13c3195268205ca2eff17d7c6b2a1b6d24ef1ed455c0ac5\": rpc error: code = NotFound desc = could not find container \"88fbf6a636da6a7cf13c3195268205ca2eff17d7c6b2a1b6d24ef1ed455c0ac5\": container with ID starting with 88fbf6a636da6a7cf13c3195268205ca2eff17d7c6b2a1b6d24ef1ed455c0ac5 not found: ID does not exist" Oct 30 18:25:06.863588 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:06.863547 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0ac6bb-67fc4d5447-5n5xd"] Oct 30 18:25:06.875503 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:06.875473 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0ac6bb-67fc4d5447-5n5xd"] Oct 30 18:25:07.067810 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:07.067733 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="141a5be6-7d7b-4199-b58f-9e6831a1ac7b" path="/var/lib/kubelet/pods/141a5be6-7d7b-4199-b58f-9e6831a1ac7b/volumes" Oct 30 18:25:07.068311 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:07.068294 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35427437-feb0-490d-966e-8978f4cfe11e" path="/var/lib/kubelet/pods/35427437-feb0-490d-966e-8978f4cfe11e/volumes" Oct 30 18:25:10.808086 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:10.808061 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-200c58-844699bd54-xg4s6" Oct 30 18:25:13.626007 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:13.625972 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-50b278-5b75c7cb45-mmhbl"] Oct 30 18:25:13.627099 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:13.627074 4442 cpu_manager.go:414] "containerMap: removing container" podUID="141a5be6-7d7b-4199-b58f-9e6831a1ac7b" containerName="payload-processing" Oct 30 18:25:13.627245 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:13.627100 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="141a5be6-7d7b-4199-b58f-9e6831a1ac7b" containerName="payload-processing" Oct 30 18:25:13.627245 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:13.627149 4442 cpu_manager.go:414] "containerMap: removing container" podUID="35427437-feb0-490d-966e-8978f4cfe11e" containerName="payload-pre-processing" Oct 30 18:25:13.627245 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:13.627157 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35427437-feb0-490d-966e-8978f4cfe11e" containerName="payload-pre-processing" Oct 30 18:25:13.627406 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:13.627303 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="141a5be6-7d7b-4199-b58f-9e6831a1ac7b" containerName="payload-processing" Oct 30 18:25:13.627406 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:13.627318 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="35427437-feb0-490d-966e-8978f4cfe11e" containerName="payload-pre-processing" Oct 30 18:25:13.627406 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:13.627334 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9e1bc7b-1615-4d9d-9a73-ee8a58cf4214" containerName="istio-proxy" Oct 30 18:25:13.631718 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:13.631690 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-50b278-5b75c7cb45-mmhbl" Oct 30 18:25:13.648057 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:13.647986 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-50b278-5b75c7cb45-mmhbl"] Oct 30 18:25:13.654431 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:13.654403 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-50b278-serving-cert\"" Oct 30 18:25:13.743000 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:13.742966 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce947c77-f4b8-4178-9834-cb8e330b9070-maas-api-tls\") pod \"maas-api-e2e-praxis-a-50b278-5b75c7cb45-mmhbl\" (UID: \"ce947c77-f4b8-4178-9834-cb8e330b9070\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-50b278-5b75c7cb45-mmhbl" Oct 30 18:25:13.743000 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:13.743012 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pj65\" (UniqueName: \"kubernetes.io/projected/ce947c77-f4b8-4178-9834-cb8e330b9070-kube-api-access-6pj65\") pod \"maas-api-e2e-praxis-a-50b278-5b75c7cb45-mmhbl\" (UID: \"ce947c77-f4b8-4178-9834-cb8e330b9070\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-50b278-5b75c7cb45-mmhbl" Oct 30 18:25:13.743280 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:13.743252 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce947c77-f4b8-4178-9834-cb8e330b9070-metrics-certs\") pod \"maas-api-e2e-praxis-a-50b278-5b75c7cb45-mmhbl\" (UID: \"ce947c77-f4b8-4178-9834-cb8e330b9070\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-50b278-5b75c7cb45-mmhbl" Oct 30 18:25:13.743434 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:13.743413 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce947c77-f4b8-4178-9834-cb8e330b9070-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-50b278-5b75c7cb45-mmhbl\" (UID: \"ce947c77-f4b8-4178-9834-cb8e330b9070\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-50b278-5b75c7cb45-mmhbl" Oct 30 18:25:13.844844 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:13.844800 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce947c77-f4b8-4178-9834-cb8e330b9070-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-50b278-5b75c7cb45-mmhbl\" (UID: \"ce947c77-f4b8-4178-9834-cb8e330b9070\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-50b278-5b75c7cb45-mmhbl" Oct 30 18:25:13.845050 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:13.844906 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce947c77-f4b8-4178-9834-cb8e330b9070-maas-api-tls\") pod \"maas-api-e2e-praxis-a-50b278-5b75c7cb45-mmhbl\" (UID: \"ce947c77-f4b8-4178-9834-cb8e330b9070\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-50b278-5b75c7cb45-mmhbl" Oct 30 18:25:13.845050 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:13.844939 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pj65\" (UniqueName: \"kubernetes.io/projected/ce947c77-f4b8-4178-9834-cb8e330b9070-kube-api-access-6pj65\") pod \"maas-api-e2e-praxis-a-50b278-5b75c7cb45-mmhbl\" (UID: \"ce947c77-f4b8-4178-9834-cb8e330b9070\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-50b278-5b75c7cb45-mmhbl" Oct 30 18:25:13.845212 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:13.845055 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce947c77-f4b8-4178-9834-cb8e330b9070-metrics-certs\") pod \"maas-api-e2e-praxis-a-50b278-5b75c7cb45-mmhbl\" (UID: \"ce947c77-f4b8-4178-9834-cb8e330b9070\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-50b278-5b75c7cb45-mmhbl" Oct 30 18:25:13.845713 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:13.845679 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce947c77-f4b8-4178-9834-cb8e330b9070-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-50b278-5b75c7cb45-mmhbl\" (UID: \"ce947c77-f4b8-4178-9834-cb8e330b9070\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-50b278-5b75c7cb45-mmhbl" Oct 30 18:25:13.847742 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:13.847721 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce947c77-f4b8-4178-9834-cb8e330b9070-metrics-certs\") pod \"maas-api-e2e-praxis-a-50b278-5b75c7cb45-mmhbl\" (UID: \"ce947c77-f4b8-4178-9834-cb8e330b9070\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-50b278-5b75c7cb45-mmhbl" Oct 30 18:25:13.847843 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:13.847782 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce947c77-f4b8-4178-9834-cb8e330b9070-maas-api-tls\") pod \"maas-api-e2e-praxis-a-50b278-5b75c7cb45-mmhbl\" (UID: \"ce947c77-f4b8-4178-9834-cb8e330b9070\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-50b278-5b75c7cb45-mmhbl" Oct 30 18:25:13.917520 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:13.917411 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pj65\" (UniqueName: \"kubernetes.io/projected/ce947c77-f4b8-4178-9834-cb8e330b9070-kube-api-access-6pj65\") pod \"maas-api-e2e-praxis-a-50b278-5b75c7cb45-mmhbl\" (UID: \"ce947c77-f4b8-4178-9834-cb8e330b9070\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-50b278-5b75c7cb45-mmhbl" Oct 30 18:25:13.952900 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:13.951611 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-50b278-5b75c7cb45-mmhbl" Oct 30 18:25:14.291959 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:14.291931 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-50b278-5b75c7cb45-mmhbl"] Oct 30 18:25:14.292953 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:25:14.292928 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce947c77_f4b8_4178_9834_cb8e330b9070.slice/crio-f25932dc64a70fb6ab573776dacfb1348468ba512ade00799c102e8a88f024be WatchSource:0}: Error finding container f25932dc64a70fb6ab573776dacfb1348468ba512ade00799c102e8a88f024be: Status 404 returned error can't find the container with id f25932dc64a70fb6ab573776dacfb1348468ba512ade00799c102e8a88f024be Oct 30 18:25:15.091308 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:15.091273 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-50b278-5b75c7cb45-mmhbl" event={"ID":"ce947c77-f4b8-4178-9834-cb8e330b9070","Type":"ContainerStarted","Data":"f25932dc64a70fb6ab573776dacfb1348468ba512ade00799c102e8a88f024be"} Oct 30 18:25:16.092400 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:16.092367 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-50b278-5b75c7cb45-mmhbl" event={"ID":"ce947c77-f4b8-4178-9834-cb8e330b9070","Type":"ContainerStarted","Data":"cb03064fb2da85273fa791a13b8d583042d9856e8ef43e17aca1e526eddeb37a"} Oct 30 18:25:16.092831 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:16.092462 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-50b278-5b75c7cb45-mmhbl" Oct 30 18:25:16.133919 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:16.133860 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-50b278-5b75c7cb45-mmhbl" podStartSLOduration=2.7503452790000003 podStartE2EDuration="3.133842725s" totalImagesPullingTime="383.497446ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:25:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:25:16.129438072 +0000 UTC m=+1613.219500413" watchObservedRunningTime="2026-10-30 18:25:16.133842725 +0000 UTC m=+1613.223905064" Oct 30 18:25:17.565465 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:25:17.565113 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:25:22.105368 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:22.105340 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-50b278-5b75c7cb45-mmhbl" Oct 30 18:25:32.340617 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:32.340567 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-17df15-79665f588d-dxcz6"] Oct 30 18:25:32.341850 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:32.341807 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-17df15-79665f588d-dxcz6" podUID="1fb3df6c-8dde-409c-985e-0bc963a96a11" containerName="maas-api" containerID="cri-o://38f69f783658c677dd5fb88a854a701447835ccff1a6b50b9a46491f29e14036" gracePeriod=30 Oct 30 18:25:32.520935 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:32.520884 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-17df15-77f8475476-bmgzk"] Oct 30 18:25:32.521748 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:32.521413 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-17df15-77f8475476-bmgzk" podUID="a54d2880-4e10-4dda-93da-006a6f24e9b3" containerName="payload-processing" containerID="cri-o://e449a4a37760b97a0b6b60614e14957a3ca3e90c2b4552dead970fea6865a320" gracePeriod=30 Oct 30 18:25:32.535300 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:32.535262 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-6bc646d6bc-7blzv"] Oct 30 18:25:32.535517 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:32.535486 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-6bc646d6bc-7blzv" podUID="f70fb3d5-b293-45df-9950-252593a8d081" containerName="payload-pre-processing" containerID="cri-o://6b697fad9ee46ebddacc99a9679a6d6c30c95b15669891f1123a977aa15d6005" gracePeriod=30 Oct 30 18:25:32.748960 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:32.746647 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-17df15-79665f588d-dxcz6" Oct 30 18:25:32.856561 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:32.856200 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-17df15-77f8475476-bmgzk" Oct 30 18:25:32.897251 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:32.897203 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-6bc646d6bc-7blzv" Oct 30 18:25:32.905527 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:32.905500 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a54d2880-4e10-4dda-93da-006a6f24e9b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a54d2880-4e10-4dda-93da-006a6f24e9b3-plugins-config-volume\") pod \"a54d2880-4e10-4dda-93da-006a6f24e9b3\" (UID: \"a54d2880-4e10-4dda-93da-006a6f24e9b3\") " Oct 30 18:25:32.905696 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:32.905548 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fb3df6c-8dde-409c-985e-0bc963a96a11-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fb3df6c-8dde-409c-985e-0bc963a96a11-openshift-service-ca\") pod \"1fb3df6c-8dde-409c-985e-0bc963a96a11\" (UID: \"1fb3df6c-8dde-409c-985e-0bc963a96a11\") " Oct 30 18:25:32.905696 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:32.905610 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f70fb3d5-b293-45df-9950-252593a8d081-kube-api-access-z9pfj\" (UniqueName: \"kubernetes.io/projected/f70fb3d5-b293-45df-9950-252593a8d081-kube-api-access-z9pfj\") pod \"f70fb3d5-b293-45df-9950-252593a8d081\" (UID: \"f70fb3d5-b293-45df-9950-252593a8d081\") " Oct 30 18:25:32.905696 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:32.905655 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f70fb3d5-b293-45df-9950-252593a8d081-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f70fb3d5-b293-45df-9950-252593a8d081-plugins-config-volume\") pod \"f70fb3d5-b293-45df-9950-252593a8d081\" (UID: \"f70fb3d5-b293-45df-9950-252593a8d081\") " Oct 30 18:25:32.905870 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:32.905703 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fb3df6c-8dde-409c-985e-0bc963a96a11-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fb3df6c-8dde-409c-985e-0bc963a96a11-metrics-certs\") pod \"1fb3df6c-8dde-409c-985e-0bc963a96a11\" (UID: \"1fb3df6c-8dde-409c-985e-0bc963a96a11\") " Oct 30 18:25:32.905870 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:32.905778 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fb3df6c-8dde-409c-985e-0bc963a96a11-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fb3df6c-8dde-409c-985e-0bc963a96a11-maas-api-tls\") pod \"1fb3df6c-8dde-409c-985e-0bc963a96a11\" (UID: \"1fb3df6c-8dde-409c-985e-0bc963a96a11\") " Oct 30 18:25:32.905976 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:32.905877 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a54d2880-4e10-4dda-93da-006a6f24e9b3-kube-api-access-c7bzd\" (UniqueName: \"kubernetes.io/projected/a54d2880-4e10-4dda-93da-006a6f24e9b3-kube-api-access-c7bzd\") pod \"a54d2880-4e10-4dda-93da-006a6f24e9b3\" (UID: \"a54d2880-4e10-4dda-93da-006a6f24e9b3\") " Oct 30 18:25:32.905976 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:32.905961 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fb3df6c-8dde-409c-985e-0bc963a96a11-kube-api-access-87wzs\" (UniqueName: \"kubernetes.io/projected/1fb3df6c-8dde-409c-985e-0bc963a96a11-kube-api-access-87wzs\") pod \"1fb3df6c-8dde-409c-985e-0bc963a96a11\" (UID: \"1fb3df6c-8dde-409c-985e-0bc963a96a11\") " Oct 30 18:25:32.908187 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:32.907158 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f70fb3d5-b293-45df-9950-252593a8d081-plugins-config-volume" pod "f70fb3d5-b293-45df-9950-252593a8d081" (UID: "f70fb3d5-b293-45df-9950-252593a8d081"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:32.908187 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:32.907606 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a54d2880-4e10-4dda-93da-006a6f24e9b3-plugins-config-volume" pod "a54d2880-4e10-4dda-93da-006a6f24e9b3" (UID: "a54d2880-4e10-4dda-93da-006a6f24e9b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:32.908771 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:32.908738 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fb3df6c-8dde-409c-985e-0bc963a96a11-openshift-service-ca" pod "1fb3df6c-8dde-409c-985e-0bc963a96a11" (UID: "1fb3df6c-8dde-409c-985e-0bc963a96a11"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:32.977653 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:32.973753 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f70fb3d5-b293-45df-9950-252593a8d081-kube-api-access-z9pfj" pod "f70fb3d5-b293-45df-9950-252593a8d081" (UID: "f70fb3d5-b293-45df-9950-252593a8d081"). InnerVolumeSpecName "kube-api-access-z9pfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:32.977653 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:32.973870 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a54d2880-4e10-4dda-93da-006a6f24e9b3-kube-api-access-c7bzd" pod "a54d2880-4e10-4dda-93da-006a6f24e9b3" (UID: "a54d2880-4e10-4dda-93da-006a6f24e9b3"). InnerVolumeSpecName "kube-api-access-c7bzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:33.042563 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:33.035030 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7bzd\" (UniqueName: \"kubernetes.io/projected/a54d2880-4e10-4dda-93da-006a6f24e9b3-kube-api-access-c7bzd\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:25:33.042563 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:33.035103 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a54d2880-4e10-4dda-93da-006a6f24e9b3-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:25:33.042563 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:33.035117 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fb3df6c-8dde-409c-985e-0bc963a96a11-openshift-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:25:33.042563 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:33.035150 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9pfj\" (UniqueName: \"kubernetes.io/projected/f70fb3d5-b293-45df-9950-252593a8d081-kube-api-access-z9pfj\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:25:33.042563 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:33.035165 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f70fb3d5-b293-45df-9950-252593a8d081-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:25:33.239025 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:33.238941 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fb3df6c-8dde-409c-985e-0bc963a96a11-metrics-certs" pod "1fb3df6c-8dde-409c-985e-0bc963a96a11" (UID: "1fb3df6c-8dde-409c-985e-0bc963a96a11"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:25:33.259212 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:33.258649 4442 generic.go:356] "Generic (PLEG): container finished" podID="a54d2880-4e10-4dda-93da-006a6f24e9b3" containerID="e449a4a37760b97a0b6b60614e14957a3ca3e90c2b4552dead970fea6865a320" exitCode=0 Oct 30 18:25:33.259212 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:33.258875 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-17df15-77f8475476-bmgzk" Oct 30 18:25:33.293164 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:33.265000 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fb3df6c-8dde-409c-985e-0bc963a96a11-metrics-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:25:33.293164 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:33.280558 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fb3df6c-8dde-409c-985e-0bc963a96a11-kube-api-access-87wzs" pod "1fb3df6c-8dde-409c-985e-0bc963a96a11" (UID: "1fb3df6c-8dde-409c-985e-0bc963a96a11"). InnerVolumeSpecName "kube-api-access-87wzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:33.293164 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:33.282890 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-17df15-77f8475476-bmgzk" event={"ID":"a54d2880-4e10-4dda-93da-006a6f24e9b3","Type":"ContainerDied","Data":"e449a4a37760b97a0b6b60614e14957a3ca3e90c2b4552dead970fea6865a320"} Oct 30 18:25:33.293164 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:33.282946 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-17df15-77f8475476-bmgzk" event={"ID":"a54d2880-4e10-4dda-93da-006a6f24e9b3","Type":"ContainerDied","Data":"a0fe5a7cb5131bb9a8ee2d76ab17f3ec7de1bd87a30e5ce26c0ed1cc7fc9e3b8"} Oct 30 18:25:33.293164 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:33.282978 4442 scope.go:122] "RemoveContainer" containerID="e449a4a37760b97a0b6b60614e14957a3ca3e90c2b4552dead970fea6865a320" Oct 30 18:25:33.310851 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:33.310440 4442 generic.go:356] "Generic (PLEG): container finished" podID="f70fb3d5-b293-45df-9950-252593a8d081" containerID="6b697fad9ee46ebddacc99a9679a6d6c30c95b15669891f1123a977aa15d6005" exitCode=0 Oct 30 18:25:33.310851 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:33.310593 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-6bc646d6bc-7blzv" Oct 30 18:25:33.313308 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:33.311110 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-6bc646d6bc-7blzv" event={"ID":"f70fb3d5-b293-45df-9950-252593a8d081","Type":"ContainerDied","Data":"6b697fad9ee46ebddacc99a9679a6d6c30c95b15669891f1123a977aa15d6005"} Oct 30 18:25:33.313308 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:33.311226 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-6bc646d6bc-7blzv" event={"ID":"f70fb3d5-b293-45df-9950-252593a8d081","Type":"ContainerDied","Data":"0b47eda4e4cf22a680e021918d2dde01ac07d89b3d14a3c982740c50c925dec9"} Oct 30 18:25:33.313308 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:33.312569 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fb3df6c-8dde-409c-985e-0bc963a96a11-maas-api-tls" pod "1fb3df6c-8dde-409c-985e-0bc963a96a11" (UID: "1fb3df6c-8dde-409c-985e-0bc963a96a11"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:25:33.326777 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:33.317077 4442 generic.go:356] "Generic (PLEG): container finished" podID="1fb3df6c-8dde-409c-985e-0bc963a96a11" containerID="38f69f783658c677dd5fb88a854a701447835ccff1a6b50b9a46491f29e14036" exitCode=0 Oct 30 18:25:33.326777 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:33.317260 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-17df15-79665f588d-dxcz6" event={"ID":"1fb3df6c-8dde-409c-985e-0bc963a96a11","Type":"ContainerDied","Data":"38f69f783658c677dd5fb88a854a701447835ccff1a6b50b9a46491f29e14036"} Oct 30 18:25:33.326777 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:33.317304 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-17df15-79665f588d-dxcz6" event={"ID":"1fb3df6c-8dde-409c-985e-0bc963a96a11","Type":"ContainerDied","Data":"a893a8808dbed08a3b5e1eca9a6e49f03b8c1244a1b8b945bab4c410ae4d5e14"} Oct 30 18:25:33.326777 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:33.317383 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-17df15-79665f588d-dxcz6" Oct 30 18:25:33.349202 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:33.336276 4442 scope.go:122] "RemoveContainer" containerID="e449a4a37760b97a0b6b60614e14957a3ca3e90c2b4552dead970fea6865a320" Oct 30 18:25:33.349202 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:25:33.342214 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e449a4a37760b97a0b6b60614e14957a3ca3e90c2b4552dead970fea6865a320\": container with ID starting with e449a4a37760b97a0b6b60614e14957a3ca3e90c2b4552dead970fea6865a320 not found: ID does not exist" containerID="e449a4a37760b97a0b6b60614e14957a3ca3e90c2b4552dead970fea6865a320" Oct 30 18:25:33.349202 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:33.342289 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e449a4a37760b97a0b6b60614e14957a3ca3e90c2b4552dead970fea6865a320"} err="failed to get container status \"e449a4a37760b97a0b6b60614e14957a3ca3e90c2b4552dead970fea6865a320\": rpc error: code = NotFound desc = could not find container \"e449a4a37760b97a0b6b60614e14957a3ca3e90c2b4552dead970fea6865a320\": container with ID starting with e449a4a37760b97a0b6b60614e14957a3ca3e90c2b4552dead970fea6865a320 not found: ID does not exist" Oct 30 18:25:33.349202 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:33.342460 4442 scope.go:122] "RemoveContainer" containerID="6b697fad9ee46ebddacc99a9679a6d6c30c95b15669891f1123a977aa15d6005" Oct 30 18:25:33.366706 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:33.366633 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fb3df6c-8dde-409c-985e-0bc963a96a11-maas-api-tls\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:25:33.366706 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:33.366670 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87wzs\" (UniqueName: \"kubernetes.io/projected/1fb3df6c-8dde-409c-985e-0bc963a96a11-kube-api-access-87wzs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:25:33.409927 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:33.400309 4442 scope.go:122] "RemoveContainer" containerID="6b697fad9ee46ebddacc99a9679a6d6c30c95b15669891f1123a977aa15d6005" Oct 30 18:25:33.411763 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:25:33.411533 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b697fad9ee46ebddacc99a9679a6d6c30c95b15669891f1123a977aa15d6005\": container with ID starting with 6b697fad9ee46ebddacc99a9679a6d6c30c95b15669891f1123a977aa15d6005 not found: ID does not exist" containerID="6b697fad9ee46ebddacc99a9679a6d6c30c95b15669891f1123a977aa15d6005" Oct 30 18:25:33.411763 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:33.411581 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b697fad9ee46ebddacc99a9679a6d6c30c95b15669891f1123a977aa15d6005"} err="failed to get container status \"6b697fad9ee46ebddacc99a9679a6d6c30c95b15669891f1123a977aa15d6005\": rpc error: code = NotFound desc = could not find container \"6b697fad9ee46ebddacc99a9679a6d6c30c95b15669891f1123a977aa15d6005\": container with ID starting with 6b697fad9ee46ebddacc99a9679a6d6c30c95b15669891f1123a977aa15d6005 not found: ID does not exist" Oct 30 18:25:33.411763 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:33.411604 4442 scope.go:122] "RemoveContainer" containerID="38f69f783658c677dd5fb88a854a701447835ccff1a6b50b9a46491f29e14036" Oct 30 18:25:33.430556 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:33.429485 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-6bc646d6bc-7blzv"] Oct 30 18:25:33.439820 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:33.439778 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-17df15-6bc646d6bc-7blzv"] Oct 30 18:25:33.459551 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:33.459492 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-17df15-79665f588d-dxcz6"] Oct 30 18:25:33.466170 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:33.466123 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-17df15-79665f588d-dxcz6"] Oct 30 18:25:33.467551 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:33.467538 4442 scope.go:122] "RemoveContainer" containerID="dd529ba1c6afd0841c42f5fd2c856f09ecd533cd8be828141f3e449b51a762e5" Oct 30 18:25:33.479441 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:33.479417 4442 scope.go:122] "RemoveContainer" containerID="38f69f783658c677dd5fb88a854a701447835ccff1a6b50b9a46491f29e14036" Oct 30 18:25:33.479818 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:25:33.479795 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38f69f783658c677dd5fb88a854a701447835ccff1a6b50b9a46491f29e14036\": container with ID starting with 38f69f783658c677dd5fb88a854a701447835ccff1a6b50b9a46491f29e14036 not found: ID does not exist" containerID="38f69f783658c677dd5fb88a854a701447835ccff1a6b50b9a46491f29e14036" Oct 30 18:25:33.479910 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:33.479869 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38f69f783658c677dd5fb88a854a701447835ccff1a6b50b9a46491f29e14036"} err="failed to get container status \"38f69f783658c677dd5fb88a854a701447835ccff1a6b50b9a46491f29e14036\": rpc error: code = NotFound desc = could not find container \"38f69f783658c677dd5fb88a854a701447835ccff1a6b50b9a46491f29e14036\": container with ID starting with 38f69f783658c677dd5fb88a854a701447835ccff1a6b50b9a46491f29e14036 not found: ID does not exist" Oct 30 18:25:33.479910 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:33.479891 4442 scope.go:122] "RemoveContainer" containerID="dd529ba1c6afd0841c42f5fd2c856f09ecd533cd8be828141f3e449b51a762e5" Oct 30 18:25:33.480226 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:25:33.480209 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd529ba1c6afd0841c42f5fd2c856f09ecd533cd8be828141f3e449b51a762e5\": container with ID starting with dd529ba1c6afd0841c42f5fd2c856f09ecd533cd8be828141f3e449b51a762e5 not found: ID does not exist" containerID="dd529ba1c6afd0841c42f5fd2c856f09ecd533cd8be828141f3e449b51a762e5" Oct 30 18:25:33.480304 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:33.480236 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd529ba1c6afd0841c42f5fd2c856f09ecd533cd8be828141f3e449b51a762e5"} err="failed to get container status \"dd529ba1c6afd0841c42f5fd2c856f09ecd533cd8be828141f3e449b51a762e5\": rpc error: code = NotFound desc = could not find container \"dd529ba1c6afd0841c42f5fd2c856f09ecd533cd8be828141f3e449b51a762e5\": container with ID starting with dd529ba1c6afd0841c42f5fd2c856f09ecd533cd8be828141f3e449b51a762e5 not found: ID does not exist" Oct 30 18:25:33.495283 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:33.495224 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-17df15-77f8475476-bmgzk"] Oct 30 18:25:33.499448 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:33.499425 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-17df15-77f8475476-bmgzk"] Oct 30 18:25:35.066574 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:35.066542 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fb3df6c-8dde-409c-985e-0bc963a96a11" path="/var/lib/kubelet/pods/1fb3df6c-8dde-409c-985e-0bc963a96a11/volumes" Oct 30 18:25:35.067021 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:35.067005 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a54d2880-4e10-4dda-93da-006a6f24e9b3" path="/var/lib/kubelet/pods/a54d2880-4e10-4dda-93da-006a6f24e9b3/volumes" Oct 30 18:25:35.067405 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:35.067391 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f70fb3d5-b293-45df-9950-252593a8d081" path="/var/lib/kubelet/pods/f70fb3d5-b293-45df-9950-252593a8d081/volumes" Oct 30 18:25:38.182501 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:38.182462 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq"] Oct 30 18:25:38.183000 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:38.182665 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" podUID="01ac5b0b-929b-4a9c-9b91-e884edd40abb" containerName="istio-proxy" containerID="cri-o://e0c4510e20171ee0cc4126f77079dd46ed8e6dc312ab03809a574f514f527aeb" gracePeriod=30 Oct 30 18:25:38.225552 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:38.225520 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9"] Oct 30 18:25:38.225750 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:38.225725 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" podUID="18a1649e-7af6-4854-bb2f-0c44af4b0276" containerName="istio-proxy" containerID="cri-o://68eaa3648c083d54d9a33805cbc64cae442acb31f8c60e8591eb28c578b05d4b" gracePeriod=30 Oct 30 18:25:42.691098 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:42.691061 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw"] Oct 30 18:25:42.691904 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:42.691880 4442 cpu_manager.go:414] "containerMap: removing container" podUID="f70fb3d5-b293-45df-9950-252593a8d081" containerName="payload-pre-processing" Oct 30 18:25:42.691992 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:42.691905 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f70fb3d5-b293-45df-9950-252593a8d081" containerName="payload-pre-processing" Oct 30 18:25:42.691992 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:42.691931 4442 cpu_manager.go:414] "containerMap: removing container" podUID="1fb3df6c-8dde-409c-985e-0bc963a96a11" containerName="maas-api" Oct 30 18:25:42.691992 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:42.691936 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fb3df6c-8dde-409c-985e-0bc963a96a11" containerName="maas-api" Oct 30 18:25:42.691992 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:42.691947 4442 cpu_manager.go:414] "containerMap: removing container" podUID="1fb3df6c-8dde-409c-985e-0bc963a96a11" containerName="maas-api" Oct 30 18:25:42.691992 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:42.691952 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fb3df6c-8dde-409c-985e-0bc963a96a11" containerName="maas-api" Oct 30 18:25:42.691992 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:42.691961 4442 cpu_manager.go:414] "containerMap: removing container" podUID="a54d2880-4e10-4dda-93da-006a6f24e9b3" containerName="payload-processing" Oct 30 18:25:42.691992 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:42.691966 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a54d2880-4e10-4dda-93da-006a6f24e9b3" containerName="payload-processing" Oct 30 18:25:42.692369 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:42.692054 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fb3df6c-8dde-409c-985e-0bc963a96a11" containerName="maas-api" Oct 30 18:25:42.692369 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:42.692062 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="a54d2880-4e10-4dda-93da-006a6f24e9b3" containerName="payload-processing" Oct 30 18:25:42.692369 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:42.692076 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="f70fb3d5-b293-45df-9950-252593a8d081" containerName="payload-pre-processing" Oct 30 18:25:42.698216 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:42.698193 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" Oct 30 18:25:42.715724 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:42.712554 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-afc614-openshift-default-dockercfg-mgkkm\"" Oct 30 18:25:42.733554 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:42.733513 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw"] Oct 30 18:25:42.915110 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:42.914307 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29c8e32e-e676-4b79-9643-25c88f1a6f1a-istiod-ca-cert\") pod \"e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw\" (UID: \"29c8e32e-e676-4b79-9643-25c88f1a6f1a\") " pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" Oct 30 18:25:42.915110 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:42.914366 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29c8e32e-e676-4b79-9643-25c88f1a6f1a-istio-podinfo\") pod \"e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw\" (UID: \"29c8e32e-e676-4b79-9643-25c88f1a6f1a\") " pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" Oct 30 18:25:42.915110 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:42.914430 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9s5m\" (UniqueName: \"kubernetes.io/projected/29c8e32e-e676-4b79-9643-25c88f1a6f1a-kube-api-access-n9s5m\") pod \"e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw\" (UID: \"29c8e32e-e676-4b79-9643-25c88f1a6f1a\") " pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" Oct 30 18:25:42.915110 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:42.914504 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29c8e32e-e676-4b79-9643-25c88f1a6f1a-istio-envoy\") pod \"e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw\" (UID: \"29c8e32e-e676-4b79-9643-25c88f1a6f1a\") " pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" Oct 30 18:25:42.915110 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:42.914533 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c8e32e-e676-4b79-9643-25c88f1a6f1a-credential-socket\") pod \"e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw\" (UID: \"29c8e32e-e676-4b79-9643-25c88f1a6f1a\") " pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" Oct 30 18:25:42.915110 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:42.914580 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c8e32e-e676-4b79-9643-25c88f1a6f1a-workload-socket\") pod \"e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw\" (UID: \"29c8e32e-e676-4b79-9643-25c88f1a6f1a\") " pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" Oct 30 18:25:42.915110 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:42.914607 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29c8e32e-e676-4b79-9643-25c88f1a6f1a-workload-certs\") pod \"e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw\" (UID: \"29c8e32e-e676-4b79-9643-25c88f1a6f1a\") " pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" Oct 30 18:25:42.915110 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:42.914692 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29c8e32e-e676-4b79-9643-25c88f1a6f1a-istio-data\") pod \"e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw\" (UID: \"29c8e32e-e676-4b79-9643-25c88f1a6f1a\") " pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" Oct 30 18:25:42.915110 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:42.914822 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29c8e32e-e676-4b79-9643-25c88f1a6f1a-istio-token\") pod \"e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw\" (UID: \"29c8e32e-e676-4b79-9643-25c88f1a6f1a\") " pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" Oct 30 18:25:43.017639 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.017553 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c8e32e-e676-4b79-9643-25c88f1a6f1a-workload-socket\") pod \"e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw\" (UID: \"29c8e32e-e676-4b79-9643-25c88f1a6f1a\") " pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" Oct 30 18:25:43.027168 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.021786 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29c8e32e-e676-4b79-9643-25c88f1a6f1a-workload-certs\") pod \"e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw\" (UID: \"29c8e32e-e676-4b79-9643-25c88f1a6f1a\") " pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" Oct 30 18:25:43.027168 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.021905 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29c8e32e-e676-4b79-9643-25c88f1a6f1a-istio-data\") pod \"e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw\" (UID: \"29c8e32e-e676-4b79-9643-25c88f1a6f1a\") " pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" Oct 30 18:25:43.027168 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.021967 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29c8e32e-e676-4b79-9643-25c88f1a6f1a-istio-token\") pod \"e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw\" (UID: \"29c8e32e-e676-4b79-9643-25c88f1a6f1a\") " pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" Oct 30 18:25:43.027168 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.022061 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29c8e32e-e676-4b79-9643-25c88f1a6f1a-istiod-ca-cert\") pod \"e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw\" (UID: \"29c8e32e-e676-4b79-9643-25c88f1a6f1a\") " pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" Oct 30 18:25:43.027168 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.022095 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29c8e32e-e676-4b79-9643-25c88f1a6f1a-istio-podinfo\") pod \"e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw\" (UID: \"29c8e32e-e676-4b79-9643-25c88f1a6f1a\") " pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" Oct 30 18:25:43.027168 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.022211 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9s5m\" (UniqueName: \"kubernetes.io/projected/29c8e32e-e676-4b79-9643-25c88f1a6f1a-kube-api-access-n9s5m\") pod \"e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw\" (UID: \"29c8e32e-e676-4b79-9643-25c88f1a6f1a\") " pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" Oct 30 18:25:43.027168 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.022305 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29c8e32e-e676-4b79-9643-25c88f1a6f1a-istio-envoy\") pod \"e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw\" (UID: \"29c8e32e-e676-4b79-9643-25c88f1a6f1a\") " pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" Oct 30 18:25:43.027168 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.022335 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c8e32e-e676-4b79-9643-25c88f1a6f1a-credential-socket\") pod \"e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw\" (UID: \"29c8e32e-e676-4b79-9643-25c88f1a6f1a\") " pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" Oct 30 18:25:43.027168 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.022768 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c8e32e-e676-4b79-9643-25c88f1a6f1a-credential-socket\") pod \"e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw\" (UID: \"29c8e32e-e676-4b79-9643-25c88f1a6f1a\") " pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" Oct 30 18:25:43.027168 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.018198 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c8e32e-e676-4b79-9643-25c88f1a6f1a-workload-socket\") pod \"e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw\" (UID: \"29c8e32e-e676-4b79-9643-25c88f1a6f1a\") " pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" Oct 30 18:25:43.027168 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.023031 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29c8e32e-e676-4b79-9643-25c88f1a6f1a-workload-certs\") pod \"e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw\" (UID: \"29c8e32e-e676-4b79-9643-25c88f1a6f1a\") " pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" Oct 30 18:25:43.027168 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.023308 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29c8e32e-e676-4b79-9643-25c88f1a6f1a-istio-data\") pod \"e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw\" (UID: \"29c8e32e-e676-4b79-9643-25c88f1a6f1a\") " pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" Oct 30 18:25:43.037213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.034035 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29c8e32e-e676-4b79-9643-25c88f1a6f1a-istiod-ca-cert\") pod \"e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw\" (UID: \"29c8e32e-e676-4b79-9643-25c88f1a6f1a\") " pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" Oct 30 18:25:43.072324 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.056842 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29c8e32e-e676-4b79-9643-25c88f1a6f1a-istio-token\") pod \"e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw\" (UID: \"29c8e32e-e676-4b79-9643-25c88f1a6f1a\") " pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" Oct 30 18:25:43.072324 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.064288 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29c8e32e-e676-4b79-9643-25c88f1a6f1a-istio-envoy\") pod \"e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw\" (UID: \"29c8e32e-e676-4b79-9643-25c88f1a6f1a\") " pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" Oct 30 18:25:43.076603 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.076562 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29c8e32e-e676-4b79-9643-25c88f1a6f1a-istio-podinfo\") pod \"e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw\" (UID: \"29c8e32e-e676-4b79-9643-25c88f1a6f1a\") " pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" Oct 30 18:25:43.178842 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.178802 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9s5m\" (UniqueName: \"kubernetes.io/projected/29c8e32e-e676-4b79-9643-25c88f1a6f1a-kube-api-access-n9s5m\") pod \"e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw\" (UID: \"29c8e32e-e676-4b79-9643-25c88f1a6f1a\") " pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" Oct 30 18:25:43.284705 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.284676 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r"] Oct 30 18:25:43.285648 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.285625 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fb3df6c-8dde-409c-985e-0bc963a96a11" containerName="maas-api" Oct 30 18:25:43.290440 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.289941 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" Oct 30 18:25:43.316416 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.316391 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" Oct 30 18:25:43.388265 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.387115 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-istio-envoy\") pod \"e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r\" (UID: \"54fe8523-0ed9-477e-a23d-8c7d624ba2d4\") " pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" Oct 30 18:25:43.388265 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.387186 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-istiod-ca-cert\") pod \"e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r\" (UID: \"54fe8523-0ed9-477e-a23d-8c7d624ba2d4\") " pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" Oct 30 18:25:43.388265 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.387299 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-credential-socket\") pod \"e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r\" (UID: \"54fe8523-0ed9-477e-a23d-8c7d624ba2d4\") " pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" Oct 30 18:25:43.388265 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.387327 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-istio-token\") pod \"e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r\" (UID: \"54fe8523-0ed9-477e-a23d-8c7d624ba2d4\") " pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" Oct 30 18:25:43.388265 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.387375 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-istio-podinfo\") pod \"e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r\" (UID: \"54fe8523-0ed9-477e-a23d-8c7d624ba2d4\") " pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" Oct 30 18:25:43.388265 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.387479 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-workload-certs\") pod \"e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r\" (UID: \"54fe8523-0ed9-477e-a23d-8c7d624ba2d4\") " pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" Oct 30 18:25:43.388265 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.387563 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9dj6\" (UniqueName: \"kubernetes.io/projected/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-kube-api-access-s9dj6\") pod \"e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r\" (UID: \"54fe8523-0ed9-477e-a23d-8c7d624ba2d4\") " pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" Oct 30 18:25:43.388265 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.387604 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-istio-data\") pod \"e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r\" (UID: \"54fe8523-0ed9-477e-a23d-8c7d624ba2d4\") " pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" Oct 30 18:25:43.388265 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.387646 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-workload-socket\") pod \"e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r\" (UID: \"54fe8523-0ed9-477e-a23d-8c7d624ba2d4\") " pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" Oct 30 18:25:43.422904 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.422820 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-be7f9e-openshift-default-dockercfg-n49xw\"" Oct 30 18:25:43.535212 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.533624 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-workload-certs\") pod \"e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r\" (UID: \"54fe8523-0ed9-477e-a23d-8c7d624ba2d4\") " pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" Oct 30 18:25:43.535212 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.533725 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9dj6\" (UniqueName: \"kubernetes.io/projected/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-kube-api-access-s9dj6\") pod \"e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r\" (UID: \"54fe8523-0ed9-477e-a23d-8c7d624ba2d4\") " pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" Oct 30 18:25:43.535212 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.533767 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-istio-data\") pod \"e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r\" (UID: \"54fe8523-0ed9-477e-a23d-8c7d624ba2d4\") " pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" Oct 30 18:25:43.535212 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.533807 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-workload-socket\") pod \"e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r\" (UID: \"54fe8523-0ed9-477e-a23d-8c7d624ba2d4\") " pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" Oct 30 18:25:43.535212 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.533873 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-istio-envoy\") pod \"e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r\" (UID: \"54fe8523-0ed9-477e-a23d-8c7d624ba2d4\") " pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" Oct 30 18:25:43.535212 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.533905 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-istiod-ca-cert\") pod \"e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r\" (UID: \"54fe8523-0ed9-477e-a23d-8c7d624ba2d4\") " pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" Oct 30 18:25:43.535212 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.534012 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-credential-socket\") pod \"e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r\" (UID: \"54fe8523-0ed9-477e-a23d-8c7d624ba2d4\") " pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" Oct 30 18:25:43.535212 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.534034 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-istio-token\") pod \"e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r\" (UID: \"54fe8523-0ed9-477e-a23d-8c7d624ba2d4\") " pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" Oct 30 18:25:43.535212 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.534082 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-istio-podinfo\") pod \"e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r\" (UID: \"54fe8523-0ed9-477e-a23d-8c7d624ba2d4\") " pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" Oct 30 18:25:43.563190 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.536221 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-workload-socket\") pod \"e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r\" (UID: \"54fe8523-0ed9-477e-a23d-8c7d624ba2d4\") " pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" Oct 30 18:25:43.563190 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.536700 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-workload-certs\") pod \"e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r\" (UID: \"54fe8523-0ed9-477e-a23d-8c7d624ba2d4\") " pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" Oct 30 18:25:43.563190 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.538583 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-istiod-ca-cert\") pod \"e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r\" (UID: \"54fe8523-0ed9-477e-a23d-8c7d624ba2d4\") " pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" Oct 30 18:25:43.563190 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.539190 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-credential-socket\") pod \"e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r\" (UID: \"54fe8523-0ed9-477e-a23d-8c7d624ba2d4\") " pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" Oct 30 18:25:43.563190 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.539889 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-istio-data\") pod \"e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r\" (UID: \"54fe8523-0ed9-477e-a23d-8c7d624ba2d4\") " pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" Oct 30 18:25:43.563190 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.546457 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-istio-podinfo\") pod \"e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r\" (UID: \"54fe8523-0ed9-477e-a23d-8c7d624ba2d4\") " pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" Oct 30 18:25:43.563190 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.546936 4442 generic.go:356] "Generic (PLEG): container finished" podID="18a1649e-7af6-4854-bb2f-0c44af4b0276" containerID="68eaa3648c083d54d9a33805cbc64cae442acb31f8c60e8591eb28c578b05d4b" exitCode=0 Oct 30 18:25:43.563190 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.547108 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" event={"ID":"18a1649e-7af6-4854-bb2f-0c44af4b0276","Type":"ContainerDied","Data":"68eaa3648c083d54d9a33805cbc64cae442acb31f8c60e8591eb28c578b05d4b"} Oct 30 18:25:43.563190 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.554022 4442 generic.go:356] "Generic (PLEG): container finished" podID="01ac5b0b-929b-4a9c-9b91-e884edd40abb" containerID="e0c4510e20171ee0cc4126f77079dd46ed8e6dc312ab03809a574f514f527aeb" exitCode=0 Oct 30 18:25:43.563190 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.554186 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" event={"ID":"01ac5b0b-929b-4a9c-9b91-e884edd40abb","Type":"ContainerDied","Data":"e0c4510e20171ee0cc4126f77079dd46ed8e6dc312ab03809a574f514f527aeb"} Oct 30 18:25:43.585365 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.585231 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9dj6\" (UniqueName: \"kubernetes.io/projected/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-kube-api-access-s9dj6\") pod \"e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r\" (UID: \"54fe8523-0ed9-477e-a23d-8c7d624ba2d4\") " pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" Oct 30 18:25:43.585882 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.585739 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-istio-token\") pod \"e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r\" (UID: \"54fe8523-0ed9-477e-a23d-8c7d624ba2d4\") " pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" Oct 30 18:25:43.590471 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.590369 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r"] Oct 30 18:25:43.597411 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.597348 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-istio-envoy\") pod \"e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r\" (UID: \"54fe8523-0ed9-477e-a23d-8c7d624ba2d4\") " pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" Oct 30 18:25:43.655775 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.654655 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" Oct 30 18:25:43.784952 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.784685 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" Oct 30 18:25:43.860732 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.859745 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01ac5b0b-929b-4a9c-9b91-e884edd40abb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01ac5b0b-929b-4a9c-9b91-e884edd40abb-istio-envoy\") pod \"01ac5b0b-929b-4a9c-9b91-e884edd40abb\" (UID: \"01ac5b0b-929b-4a9c-9b91-e884edd40abb\") " Oct 30 18:25:43.860732 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.859838 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01ac5b0b-929b-4a9c-9b91-e884edd40abb-istio-token\" (UniqueName: \"kubernetes.io/projected/01ac5b0b-929b-4a9c-9b91-e884edd40abb-istio-token\") pod \"01ac5b0b-929b-4a9c-9b91-e884edd40abb\" (UID: \"01ac5b0b-929b-4a9c-9b91-e884edd40abb\") " Oct 30 18:25:43.860732 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.860561 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01ac5b0b-929b-4a9c-9b91-e884edd40abb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01ac5b0b-929b-4a9c-9b91-e884edd40abb-workload-certs\") pod \"01ac5b0b-929b-4a9c-9b91-e884edd40abb\" (UID: \"01ac5b0b-929b-4a9c-9b91-e884edd40abb\") " Oct 30 18:25:43.861043 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.860904 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01ac5b0b-929b-4a9c-9b91-e884edd40abb-workload-certs" pod "01ac5b0b-929b-4a9c-9b91-e884edd40abb" (UID: "01ac5b0b-929b-4a9c-9b91-e884edd40abb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:43.861104 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.861052 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01ac5b0b-929b-4a9c-9b91-e884edd40abb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01ac5b0b-929b-4a9c-9b91-e884edd40abb-istio-data\") pod \"01ac5b0b-929b-4a9c-9b91-e884edd40abb\" (UID: \"01ac5b0b-929b-4a9c-9b91-e884edd40abb\") " Oct 30 18:25:43.861382 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.861325 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01ac5b0b-929b-4a9c-9b91-e884edd40abb-istio-data" pod "01ac5b0b-929b-4a9c-9b91-e884edd40abb" (UID: "01ac5b0b-929b-4a9c-9b91-e884edd40abb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:43.861489 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.861469 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01ac5b0b-929b-4a9c-9b91-e884edd40abb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ac5b0b-929b-4a9c-9b91-e884edd40abb-workload-socket\") pod \"01ac5b0b-929b-4a9c-9b91-e884edd40abb\" (UID: \"01ac5b0b-929b-4a9c-9b91-e884edd40abb\") " Oct 30 18:25:43.861731 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.861711 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01ac5b0b-929b-4a9c-9b91-e884edd40abb-workload-socket" pod "01ac5b0b-929b-4a9c-9b91-e884edd40abb" (UID: "01ac5b0b-929b-4a9c-9b91-e884edd40abb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:43.861794 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.861779 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01ac5b0b-929b-4a9c-9b91-e884edd40abb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ac5b0b-929b-4a9c-9b91-e884edd40abb-credential-socket\") pod \"01ac5b0b-929b-4a9c-9b91-e884edd40abb\" (UID: \"01ac5b0b-929b-4a9c-9b91-e884edd40abb\") " Oct 30 18:25:43.862059 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.862037 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01ac5b0b-929b-4a9c-9b91-e884edd40abb-credential-socket" pod "01ac5b0b-929b-4a9c-9b91-e884edd40abb" (UID: "01ac5b0b-929b-4a9c-9b91-e884edd40abb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:43.862173 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.862106 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01ac5b0b-929b-4a9c-9b91-e884edd40abb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01ac5b0b-929b-4a9c-9b91-e884edd40abb-istiod-ca-cert\") pod \"01ac5b0b-929b-4a9c-9b91-e884edd40abb\" (UID: \"01ac5b0b-929b-4a9c-9b91-e884edd40abb\") " Oct 30 18:25:43.862481 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.862438 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01ac5b0b-929b-4a9c-9b91-e884edd40abb-istiod-ca-cert" pod "01ac5b0b-929b-4a9c-9b91-e884edd40abb" (UID: "01ac5b0b-929b-4a9c-9b91-e884edd40abb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:43.862564 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.862512 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01ac5b0b-929b-4a9c-9b91-e884edd40abb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01ac5b0b-929b-4a9c-9b91-e884edd40abb-istio-podinfo\") pod \"01ac5b0b-929b-4a9c-9b91-e884edd40abb\" (UID: \"01ac5b0b-929b-4a9c-9b91-e884edd40abb\") " Oct 30 18:25:43.862970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.862946 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01ac5b0b-929b-4a9c-9b91-e884edd40abb-kube-api-access-4zjsl\" (UniqueName: \"kubernetes.io/projected/01ac5b0b-929b-4a9c-9b91-e884edd40abb-kube-api-access-4zjsl\") pod \"01ac5b0b-929b-4a9c-9b91-e884edd40abb\" (UID: \"01ac5b0b-929b-4a9c-9b91-e884edd40abb\") " Oct 30 18:25:43.864054 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.864028 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ac5b0b-929b-4a9c-9b91-e884edd40abb-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:25:43.864193 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.864070 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ac5b0b-929b-4a9c-9b91-e884edd40abb-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:25:43.864193 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.864083 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01ac5b0b-929b-4a9c-9b91-e884edd40abb-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:25:43.864193 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.864096 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01ac5b0b-929b-4a9c-9b91-e884edd40abb-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:25:43.864193 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.864109 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01ac5b0b-929b-4a9c-9b91-e884edd40abb-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:25:43.867926 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.867890 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01ac5b0b-929b-4a9c-9b91-e884edd40abb-istio-podinfo" pod "01ac5b0b-929b-4a9c-9b91-e884edd40abb" (UID: "01ac5b0b-929b-4a9c-9b91-e884edd40abb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:25:43.868252 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.868214 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01ac5b0b-929b-4a9c-9b91-e884edd40abb-istio-token" pod "01ac5b0b-929b-4a9c-9b91-e884edd40abb" (UID: "01ac5b0b-929b-4a9c-9b91-e884edd40abb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:43.868765 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.868730 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01ac5b0b-929b-4a9c-9b91-e884edd40abb-istio-envoy" pod "01ac5b0b-929b-4a9c-9b91-e884edd40abb" (UID: "01ac5b0b-929b-4a9c-9b91-e884edd40abb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:43.871603 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.871578 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01ac5b0b-929b-4a9c-9b91-e884edd40abb-kube-api-access-4zjsl" pod "01ac5b0b-929b-4a9c-9b91-e884edd40abb" (UID: "01ac5b0b-929b-4a9c-9b91-e884edd40abb"). InnerVolumeSpecName "kube-api-access-4zjsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:43.884470 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:25:43.884430 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c8e32e_e676_4b79_9643_25c88f1a6f1a.slice/crio-bab7deec04d160b291a4ffd3e0d60525dc03e2a2c8181fd64ff7df16740424fc WatchSource:0}: Error finding container bab7deec04d160b291a4ffd3e0d60525dc03e2a2c8181fd64ff7df16740424fc: Status 404 returned error can't find the container with id bab7deec04d160b291a4ffd3e0d60525dc03e2a2c8181fd64ff7df16740424fc Oct 30 18:25:43.884907 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.884828 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw"] Oct 30 18:25:43.887628 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.887594 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:25:43.887728 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.887673 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:25:43.887728 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.887717 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:25:43.921804 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.921760 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" Oct 30 18:25:43.964825 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.964798 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18a1649e-7af6-4854-bb2f-0c44af4b0276-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18a1649e-7af6-4854-bb2f-0c44af4b0276-istiod-ca-cert\") pod \"18a1649e-7af6-4854-bb2f-0c44af4b0276\" (UID: \"18a1649e-7af6-4854-bb2f-0c44af4b0276\") " Oct 30 18:25:43.965426 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.965371 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18a1649e-7af6-4854-bb2f-0c44af4b0276-credential-socket" pod "18a1649e-7af6-4854-bb2f-0c44af4b0276" (UID: "18a1649e-7af6-4854-bb2f-0c44af4b0276"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:43.965582 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.965070 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18a1649e-7af6-4854-bb2f-0c44af4b0276-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a1649e-7af6-4854-bb2f-0c44af4b0276-credential-socket\") pod \"18a1649e-7af6-4854-bb2f-0c44af4b0276\" (UID: \"18a1649e-7af6-4854-bb2f-0c44af4b0276\") " Oct 30 18:25:43.965823 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.965800 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18a1649e-7af6-4854-bb2f-0c44af4b0276-istio-token\" (UniqueName: \"kubernetes.io/projected/18a1649e-7af6-4854-bb2f-0c44af4b0276-istio-token\") pod \"18a1649e-7af6-4854-bb2f-0c44af4b0276\" (UID: \"18a1649e-7af6-4854-bb2f-0c44af4b0276\") " Oct 30 18:25:43.967091 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.966290 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18a1649e-7af6-4854-bb2f-0c44af4b0276-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18a1649e-7af6-4854-bb2f-0c44af4b0276-istio-data\") pod \"18a1649e-7af6-4854-bb2f-0c44af4b0276\" (UID: \"18a1649e-7af6-4854-bb2f-0c44af4b0276\") " Oct 30 18:25:43.967351 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.966530 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18a1649e-7af6-4854-bb2f-0c44af4b0276-istio-data" pod "18a1649e-7af6-4854-bb2f-0c44af4b0276" (UID: "18a1649e-7af6-4854-bb2f-0c44af4b0276"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:43.971266 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.967897 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18a1649e-7af6-4854-bb2f-0c44af4b0276-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18a1649e-7af6-4854-bb2f-0c44af4b0276-workload-certs\") pod \"18a1649e-7af6-4854-bb2f-0c44af4b0276\" (UID: \"18a1649e-7af6-4854-bb2f-0c44af4b0276\") " Oct 30 18:25:43.971342 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.971262 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18a1649e-7af6-4854-bb2f-0c44af4b0276-kube-api-access-rtj9z\" (UniqueName: \"kubernetes.io/projected/18a1649e-7af6-4854-bb2f-0c44af4b0276-kube-api-access-rtj9z\") pod \"18a1649e-7af6-4854-bb2f-0c44af4b0276\" (UID: \"18a1649e-7af6-4854-bb2f-0c44af4b0276\") " Oct 30 18:25:43.971342 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.971320 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18a1649e-7af6-4854-bb2f-0c44af4b0276-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18a1649e-7af6-4854-bb2f-0c44af4b0276-istio-podinfo\") pod \"18a1649e-7af6-4854-bb2f-0c44af4b0276\" (UID: \"18a1649e-7af6-4854-bb2f-0c44af4b0276\") " Oct 30 18:25:43.971455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.971371 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18a1649e-7af6-4854-bb2f-0c44af4b0276-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18a1649e-7af6-4854-bb2f-0c44af4b0276-istio-envoy\") pod \"18a1649e-7af6-4854-bb2f-0c44af4b0276\" (UID: \"18a1649e-7af6-4854-bb2f-0c44af4b0276\") " Oct 30 18:25:43.971455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.971421 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18a1649e-7af6-4854-bb2f-0c44af4b0276-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a1649e-7af6-4854-bb2f-0c44af4b0276-workload-socket\") pod \"18a1649e-7af6-4854-bb2f-0c44af4b0276\" (UID: \"18a1649e-7af6-4854-bb2f-0c44af4b0276\") " Oct 30 18:25:43.972742 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.972200 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01ac5b0b-929b-4a9c-9b91-e884edd40abb-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:25:43.972742 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.972228 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zjsl\" (UniqueName: \"kubernetes.io/projected/01ac5b0b-929b-4a9c-9b91-e884edd40abb-kube-api-access-4zjsl\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:25:43.972742 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.972244 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a1649e-7af6-4854-bb2f-0c44af4b0276-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:25:43.972742 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.972268 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01ac5b0b-929b-4a9c-9b91-e884edd40abb-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:25:43.972742 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.972281 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01ac5b0b-929b-4a9c-9b91-e884edd40abb-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:25:43.972742 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.972294 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18a1649e-7af6-4854-bb2f-0c44af4b0276-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:25:43.972742 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.967850 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18a1649e-7af6-4854-bb2f-0c44af4b0276-workload-certs" pod "18a1649e-7af6-4854-bb2f-0c44af4b0276" (UID: "18a1649e-7af6-4854-bb2f-0c44af4b0276"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:43.972742 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.969972 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18a1649e-7af6-4854-bb2f-0c44af4b0276-istio-token" pod "18a1649e-7af6-4854-bb2f-0c44af4b0276" (UID: "18a1649e-7af6-4854-bb2f-0c44af4b0276"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:43.972742 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.972339 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18a1649e-7af6-4854-bb2f-0c44af4b0276-istiod-ca-cert" pod "18a1649e-7af6-4854-bb2f-0c44af4b0276" (UID: "18a1649e-7af6-4854-bb2f-0c44af4b0276"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:43.972742 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.972675 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18a1649e-7af6-4854-bb2f-0c44af4b0276-workload-socket" pod "18a1649e-7af6-4854-bb2f-0c44af4b0276" (UID: "18a1649e-7af6-4854-bb2f-0c44af4b0276"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:43.978207 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.978096 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18a1649e-7af6-4854-bb2f-0c44af4b0276-istio-envoy" pod "18a1649e-7af6-4854-bb2f-0c44af4b0276" (UID: "18a1649e-7af6-4854-bb2f-0c44af4b0276"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:43.981334 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.978385 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18a1649e-7af6-4854-bb2f-0c44af4b0276-kube-api-access-rtj9z" pod "18a1649e-7af6-4854-bb2f-0c44af4b0276" (UID: "18a1649e-7af6-4854-bb2f-0c44af4b0276"). InnerVolumeSpecName "kube-api-access-rtj9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:43.981334 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:43.981315 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18a1649e-7af6-4854-bb2f-0c44af4b0276-istio-podinfo" pod "18a1649e-7af6-4854-bb2f-0c44af4b0276" (UID: "18a1649e-7af6-4854-bb2f-0c44af4b0276"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:25:44.055471 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:44.054518 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r"] Oct 30 18:25:44.056115 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:25:44.055698 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54fe8523_0ed9_477e_a23d_8c7d624ba2d4.slice/crio-a4d9dd87156c4cd21798944c948ce8816bb1f8e59954fdc32a9cbc976b6f6e30 WatchSource:0}: Error finding container a4d9dd87156c4cd21798944c948ce8816bb1f8e59954fdc32a9cbc976b6f6e30: Status 404 returned error can't find the container with id a4d9dd87156c4cd21798944c948ce8816bb1f8e59954fdc32a9cbc976b6f6e30 Oct 30 18:25:44.061452 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:44.060729 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:25:44.061452 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:44.060811 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:25:44.061452 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:44.060855 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:25:44.073548 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:44.073425 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18a1649e-7af6-4854-bb2f-0c44af4b0276-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:25:44.073548 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:44.073454 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18a1649e-7af6-4854-bb2f-0c44af4b0276-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:25:44.073548 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:44.073468 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18a1649e-7af6-4854-bb2f-0c44af4b0276-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:25:44.073548 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:44.073483 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtj9z\" (UniqueName: \"kubernetes.io/projected/18a1649e-7af6-4854-bb2f-0c44af4b0276-kube-api-access-rtj9z\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:25:44.073548 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:44.073499 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18a1649e-7af6-4854-bb2f-0c44af4b0276-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:25:44.073548 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:44.073512 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18a1649e-7af6-4854-bb2f-0c44af4b0276-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:25:44.073548 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:44.073526 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a1649e-7af6-4854-bb2f-0c44af4b0276-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:25:44.565319 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:44.565282 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" event={"ID":"54fe8523-0ed9-477e-a23d-8c7d624ba2d4","Type":"ContainerStarted","Data":"be3cc516aab817d15f7d3dfd489b7545d6353c939bff37985ee155f6f00919df"} Oct 30 18:25:44.565584 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:44.565562 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" event={"ID":"54fe8523-0ed9-477e-a23d-8c7d624ba2d4","Type":"ContainerStarted","Data":"a4d9dd87156c4cd21798944c948ce8816bb1f8e59954fdc32a9cbc976b6f6e30"} Oct 30 18:25:44.567082 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:44.567042 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" Oct 30 18:25:44.567237 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:44.567068 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq" event={"ID":"01ac5b0b-929b-4a9c-9b91-e884edd40abb","Type":"ContainerDied","Data":"3c8669f7e094f9bb98bb8018a80656bfc2139654a7aa3783ff19a94b6f30451c"} Oct 30 18:25:44.567237 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:44.567173 4442 scope.go:122] "RemoveContainer" containerID="e0c4510e20171ee0cc4126f77079dd46ed8e6dc312ab03809a574f514f527aeb" Oct 30 18:25:44.573306 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:44.573286 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" Oct 30 18:25:44.573427 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:44.573317 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9" event={"ID":"18a1649e-7af6-4854-bb2f-0c44af4b0276","Type":"ContainerDied","Data":"fe204c87278aafe243953793e710ddb687d3b5ff6341567ecca09bba394621d4"} Oct 30 18:25:44.575395 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:44.575368 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" event={"ID":"29c8e32e-e676-4b79-9643-25c88f1a6f1a","Type":"ContainerStarted","Data":"190401a7ae3d22201b3afd9adfbff94e809554cf2cd17a81774e38683963bb1b"} Oct 30 18:25:44.575546 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:44.575528 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" event={"ID":"29c8e32e-e676-4b79-9643-25c88f1a6f1a","Type":"ContainerStarted","Data":"bab7deec04d160b291a4ffd3e0d60525dc03e2a2c8181fd64ff7df16740424fc"} Oct 30 18:25:44.584620 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:44.584603 4442 scope.go:122] "RemoveContainer" containerID="68eaa3648c083d54d9a33805cbc64cae442acb31f8c60e8591eb28c578b05d4b" Oct 30 18:25:44.609000 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:44.608959 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" podStartSLOduration=1.6089460519999998 podStartE2EDuration="1.608946052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:25:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:25:44.607317277 +0000 UTC m=+1641.697379616" watchObservedRunningTime="2026-10-30 18:25:44.608946052 +0000 UTC m=+1641.699008392" Oct 30 18:25:44.652230 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:44.649816 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" podStartSLOduration=2.649801618 podStartE2EDuration="2.649801618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:25:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:25:44.648206007 +0000 UTC m=+1641.738268347" watchObservedRunningTime="2026-10-30 18:25:44.649801618 +0000 UTC m=+1641.739863956" Oct 30 18:25:44.656463 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:44.656440 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" Oct 30 18:25:44.685778 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:44.685751 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq"] Oct 30 18:25:44.690018 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:44.689993 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-vhqlq"] Oct 30 18:25:44.729701 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:44.729659 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9"] Oct 30 18:25:44.735518 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:44.735491 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-9q9g9"] Oct 30 18:25:45.068261 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:45.068229 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01ac5b0b-929b-4a9c-9b91-e884edd40abb" path="/var/lib/kubelet/pods/01ac5b0b-929b-4a9c-9b91-e884edd40abb/volumes" Oct 30 18:25:45.068752 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:45.068737 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18a1649e-7af6-4854-bb2f-0c44af4b0276" path="/var/lib/kubelet/pods/18a1649e-7af6-4854-bb2f-0c44af4b0276/volumes" Oct 30 18:25:45.316694 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:45.316663 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" Oct 30 18:25:45.323285 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:45.323209 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" Oct 30 18:25:45.585771 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:45.585689 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" Oct 30 18:25:45.586765 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:45.586746 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" Oct 30 18:25:45.690959 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:45.690915 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" Oct 30 18:25:46.593620 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:46.593585 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" Oct 30 18:25:46.594674 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:46.594654 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" Oct 30 18:25:47.562523 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:25:47.562482 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:25:48.890328 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:48.890291 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0ac6bb-7f54cddc4c-2vs6c"] Oct 30 18:25:48.890873 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:48.890544 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0ac6bb-7f54cddc4c-2vs6c" podUID="1ebd21aa-152c-4087-b33b-461678c392a6" containerName="maas-api" containerID="cri-o://777966f70e8350f1fb48e6c73b097809769a7dc72380a155102be2429459c5dd" gracePeriod=30 Oct 30 18:25:48.958152 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:48.957839 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-afc614-c4d5cff79-8rgjs"] Oct 30 18:25:48.958898 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:48.958866 4442 cpu_manager.go:414] "containerMap: removing container" podUID="01ac5b0b-929b-4a9c-9b91-e884edd40abb" containerName="istio-proxy" Oct 30 18:25:48.958898 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:48.958890 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01ac5b0b-929b-4a9c-9b91-e884edd40abb" containerName="istio-proxy" Oct 30 18:25:48.959047 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:48.958923 4442 cpu_manager.go:414] "containerMap: removing container" podUID="18a1649e-7af6-4854-bb2f-0c44af4b0276" containerName="istio-proxy" Oct 30 18:25:48.959047 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:48.958932 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18a1649e-7af6-4854-bb2f-0c44af4b0276" containerName="istio-proxy" Oct 30 18:25:48.959166 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:48.959119 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="18a1649e-7af6-4854-bb2f-0c44af4b0276" containerName="istio-proxy" Oct 30 18:25:48.959166 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:48.959160 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="01ac5b0b-929b-4a9c-9b91-e884edd40abb" containerName="istio-proxy" Oct 30 18:25:48.962645 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:48.962621 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-afc614-c4d5cff79-8rgjs" Oct 30 18:25:48.981789 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:48.981245 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-afc614\"" Oct 30 18:25:48.986796 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:48.981264 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-afc614-dockercfg-2rh2n\"" Oct 30 18:25:48.986796 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:48.986744 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-afc614-c4d5cff79-8rgjs"] Oct 30 18:25:49.069927 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:49.069892 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2ee7fe9-5c24-422e-9140-5d1ec2564efc-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-afc614-c4d5cff79-8rgjs\" (UID: \"c2ee7fe9-5c24-422e-9140-5d1ec2564efc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-afc614-c4d5cff79-8rgjs" Oct 30 18:25:49.070126 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:49.070050 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klgj9\" (UniqueName: \"kubernetes.io/projected/c2ee7fe9-5c24-422e-9140-5d1ec2564efc-kube-api-access-klgj9\") pod \"payload-pre-processing-e2e-praxis-b-afc614-c4d5cff79-8rgjs\" (UID: \"c2ee7fe9-5c24-422e-9140-5d1ec2564efc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-afc614-c4d5cff79-8rgjs" Oct 30 18:25:49.073275 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:49.073252 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-afc614-78f958bbd8-dgrmm"] Oct 30 18:25:49.077910 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:49.077882 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-afc614-78f958bbd8-dgrmm" Oct 30 18:25:49.091764 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:49.091737 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-afc614-78f958bbd8-dgrmm"] Oct 30 18:25:49.203208 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:49.177677 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w25xd\" (UniqueName: \"kubernetes.io/projected/7c46dd87-b6ad-4929-bf8c-ab10fd137e5b-kube-api-access-w25xd\") pod \"payload-processing-e2e-praxis-b-afc614-78f958bbd8-dgrmm\" (UID: \"7c46dd87-b6ad-4929-bf8c-ab10fd137e5b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-afc614-78f958bbd8-dgrmm" Oct 30 18:25:49.203208 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:49.177852 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2ee7fe9-5c24-422e-9140-5d1ec2564efc-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-afc614-c4d5cff79-8rgjs\" (UID: \"c2ee7fe9-5c24-422e-9140-5d1ec2564efc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-afc614-c4d5cff79-8rgjs" Oct 30 18:25:49.203208 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:49.177902 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c46dd87-b6ad-4929-bf8c-ab10fd137e5b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-afc614-78f958bbd8-dgrmm\" (UID: \"7c46dd87-b6ad-4929-bf8c-ab10fd137e5b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-afc614-78f958bbd8-dgrmm" Oct 30 18:25:49.203208 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:49.178023 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klgj9\" (UniqueName: \"kubernetes.io/projected/c2ee7fe9-5c24-422e-9140-5d1ec2564efc-kube-api-access-klgj9\") pod \"payload-pre-processing-e2e-praxis-b-afc614-c4d5cff79-8rgjs\" (UID: \"c2ee7fe9-5c24-422e-9140-5d1ec2564efc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-afc614-c4d5cff79-8rgjs" Oct 30 18:25:49.203208 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:49.199587 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2ee7fe9-5c24-422e-9140-5d1ec2564efc-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-afc614-c4d5cff79-8rgjs\" (UID: \"c2ee7fe9-5c24-422e-9140-5d1ec2564efc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-afc614-c4d5cff79-8rgjs" Oct 30 18:25:49.248496 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:49.247266 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klgj9\" (UniqueName: \"kubernetes.io/projected/c2ee7fe9-5c24-422e-9140-5d1ec2564efc-kube-api-access-klgj9\") pod \"payload-pre-processing-e2e-praxis-b-afc614-c4d5cff79-8rgjs\" (UID: \"c2ee7fe9-5c24-422e-9140-5d1ec2564efc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-afc614-c4d5cff79-8rgjs" Oct 30 18:25:49.323169 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:49.316043 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c46dd87-b6ad-4929-bf8c-ab10fd137e5b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-afc614-78f958bbd8-dgrmm\" (UID: \"7c46dd87-b6ad-4929-bf8c-ab10fd137e5b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-afc614-78f958bbd8-dgrmm" Oct 30 18:25:49.325421 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:49.316668 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-afc614-c4d5cff79-8rgjs" Oct 30 18:25:49.326443 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:49.326405 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w25xd\" (UniqueName: \"kubernetes.io/projected/7c46dd87-b6ad-4929-bf8c-ab10fd137e5b-kube-api-access-w25xd\") pod \"payload-processing-e2e-praxis-b-afc614-78f958bbd8-dgrmm\" (UID: \"7c46dd87-b6ad-4929-bf8c-ab10fd137e5b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-afc614-78f958bbd8-dgrmm" Oct 30 18:25:49.331161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:49.317682 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c46dd87-b6ad-4929-bf8c-ab10fd137e5b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-afc614-78f958bbd8-dgrmm\" (UID: \"7c46dd87-b6ad-4929-bf8c-ab10fd137e5b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-afc614-78f958bbd8-dgrmm" Oct 30 18:25:49.469986 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:49.468810 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w25xd\" (UniqueName: \"kubernetes.io/projected/7c46dd87-b6ad-4929-bf8c-ab10fd137e5b-kube-api-access-w25xd\") pod \"payload-processing-e2e-praxis-b-afc614-78f958bbd8-dgrmm\" (UID: \"7c46dd87-b6ad-4929-bf8c-ab10fd137e5b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-afc614-78f958bbd8-dgrmm" Oct 30 18:25:49.546590 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:49.546539 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0ac6bb-7f54cddc4c-2vs6c" Oct 30 18:25:49.656226 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:49.655623 4442 generic.go:356] "Generic (PLEG): container finished" podID="1ebd21aa-152c-4087-b33b-461678c392a6" containerID="777966f70e8350f1fb48e6c73b097809769a7dc72380a155102be2429459c5dd" exitCode=0 Oct 30 18:25:49.656226 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:49.655705 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0ac6bb-7f54cddc4c-2vs6c" event={"ID":"1ebd21aa-152c-4087-b33b-461678c392a6","Type":"ContainerDied","Data":"777966f70e8350f1fb48e6c73b097809769a7dc72380a155102be2429459c5dd"} Oct 30 18:25:49.656226 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:49.655748 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0ac6bb-7f54cddc4c-2vs6c" event={"ID":"1ebd21aa-152c-4087-b33b-461678c392a6","Type":"ContainerDied","Data":"d0ded2b44fd4a3e35e9b83a445d9ac5076bdfdeaeaadb5a0dce7c048fe95f414"} Oct 30 18:25:49.656226 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:49.655769 4442 scope.go:122] "RemoveContainer" containerID="777966f70e8350f1fb48e6c73b097809769a7dc72380a155102be2429459c5dd" Oct 30 18:25:49.656226 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:49.655886 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0ac6bb-7f54cddc4c-2vs6c" Oct 30 18:25:49.671180 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:49.671161 4442 scope.go:122] "RemoveContainer" containerID="777966f70e8350f1fb48e6c73b097809769a7dc72380a155102be2429459c5dd" Oct 30 18:25:49.671540 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:25:49.671517 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"777966f70e8350f1fb48e6c73b097809769a7dc72380a155102be2429459c5dd\": container with ID starting with 777966f70e8350f1fb48e6c73b097809769a7dc72380a155102be2429459c5dd not found: ID does not exist" containerID="777966f70e8350f1fb48e6c73b097809769a7dc72380a155102be2429459c5dd" Oct 30 18:25:49.671609 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:49.671554 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"777966f70e8350f1fb48e6c73b097809769a7dc72380a155102be2429459c5dd"} err="failed to get container status \"777966f70e8350f1fb48e6c73b097809769a7dc72380a155102be2429459c5dd\": rpc error: code = NotFound desc = could not find container \"777966f70e8350f1fb48e6c73b097809769a7dc72380a155102be2429459c5dd\": container with ID starting with 777966f70e8350f1fb48e6c73b097809769a7dc72380a155102be2429459c5dd not found: ID does not exist" Oct 30 18:25:49.683258 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:49.683235 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ebd21aa-152c-4087-b33b-461678c392a6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ebd21aa-152c-4087-b33b-461678c392a6-maas-api-tls\") pod \"1ebd21aa-152c-4087-b33b-461678c392a6\" (UID: \"1ebd21aa-152c-4087-b33b-461678c392a6\") " Oct 30 18:25:49.683355 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:49.683291 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ebd21aa-152c-4087-b33b-461678c392a6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ebd21aa-152c-4087-b33b-461678c392a6-metrics-certs\") pod \"1ebd21aa-152c-4087-b33b-461678c392a6\" (UID: \"1ebd21aa-152c-4087-b33b-461678c392a6\") " Oct 30 18:25:49.683469 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:49.683450 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ebd21aa-152c-4087-b33b-461678c392a6-kube-api-access-4qptp\" (UniqueName: \"kubernetes.io/projected/1ebd21aa-152c-4087-b33b-461678c392a6-kube-api-access-4qptp\") pod \"1ebd21aa-152c-4087-b33b-461678c392a6\" (UID: \"1ebd21aa-152c-4087-b33b-461678c392a6\") " Oct 30 18:25:49.683623 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:49.683602 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ebd21aa-152c-4087-b33b-461678c392a6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ebd21aa-152c-4087-b33b-461678c392a6-openshift-service-ca\") pod \"1ebd21aa-152c-4087-b33b-461678c392a6\" (UID: \"1ebd21aa-152c-4087-b33b-461678c392a6\") " Oct 30 18:25:49.683999 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:49.683966 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ebd21aa-152c-4087-b33b-461678c392a6-openshift-service-ca" pod "1ebd21aa-152c-4087-b33b-461678c392a6" (UID: "1ebd21aa-152c-4087-b33b-461678c392a6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:49.684333 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:49.684311 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ebd21aa-152c-4087-b33b-461678c392a6-openshift-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.685961 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:49.685934 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ebd21aa-152c-4087-b33b-461678c392a6-metrics-certs" pod "1ebd21aa-152c-4087-b33b-461678c392a6" (UID: "1ebd21aa-152c-4087-b33b-461678c392a6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:25:49.686175 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:49.686120 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ebd21aa-152c-4087-b33b-461678c392a6-maas-api-tls" pod "1ebd21aa-152c-4087-b33b-461678c392a6" (UID: "1ebd21aa-152c-4087-b33b-461678c392a6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:25:49.686266 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:49.686229 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ebd21aa-152c-4087-b33b-461678c392a6-kube-api-access-4qptp" pod "1ebd21aa-152c-4087-b33b-461678c392a6" (UID: "1ebd21aa-152c-4087-b33b-461678c392a6"). InnerVolumeSpecName "kube-api-access-4qptp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:49.697427 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:49.697105 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-afc614-78f958bbd8-dgrmm" Oct 30 18:25:49.785267 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:49.785237 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ebd21aa-152c-4087-b33b-461678c392a6-maas-api-tls\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.785267 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:49.785266 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ebd21aa-152c-4087-b33b-461678c392a6-metrics-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.785421 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:49.785278 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qptp\" (UniqueName: \"kubernetes.io/projected/1ebd21aa-152c-4087-b33b-461678c392a6-kube-api-access-4qptp\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.865183 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:49.865151 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-afc614-c4d5cff79-8rgjs"] Oct 30 18:25:50.019386 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:50.019348 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-afc614-78f958bbd8-dgrmm"] Oct 30 18:25:50.022084 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:25:50.022060 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c46dd87_b6ad_4929_bf8c_ab10fd137e5b.slice/crio-ab2ea51dd84ec5b037f30b6feaa9dede2757a884c5d291d93b28ba85426f1182 WatchSource:0}: Error finding container ab2ea51dd84ec5b037f30b6feaa9dede2757a884c5d291d93b28ba85426f1182: Status 404 returned error can't find the container with id ab2ea51dd84ec5b037f30b6feaa9dede2757a884c5d291d93b28ba85426f1182 Oct 30 18:25:50.114022 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:50.113947 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0ac6bb-7f54cddc4c-2vs6c"] Oct 30 18:25:50.123987 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:50.123951 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0ac6bb-7f54cddc4c-2vs6c"] Oct 30 18:25:50.668330 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:50.668292 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-afc614-c4d5cff79-8rgjs" event={"ID":"c2ee7fe9-5c24-422e-9140-5d1ec2564efc","Type":"ContainerStarted","Data":"7ff446a052ce1af021763a3095f54833fbd46cdadafe22441579c46a6d199ed0"} Oct 30 18:25:50.668507 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:50.668362 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-afc614-c4d5cff79-8rgjs" Oct 30 18:25:50.668507 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:50.668385 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-afc614-c4d5cff79-8rgjs" event={"ID":"c2ee7fe9-5c24-422e-9140-5d1ec2564efc","Type":"ContainerStarted","Data":"b3c292bc0a73a420c2abdf50981bace08925a0c31c4f56fd7829d05453835225"} Oct 30 18:25:50.670285 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:50.670256 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-afc614-78f958bbd8-dgrmm" event={"ID":"7c46dd87-b6ad-4929-bf8c-ab10fd137e5b","Type":"ContainerStarted","Data":"e3af9e7af3c5ebfbbd29bef74b71c81d7610cd3257cebc8c41cad10162e1ed05"} Oct 30 18:25:50.670417 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:50.670305 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-afc614-78f958bbd8-dgrmm" event={"ID":"7c46dd87-b6ad-4929-bf8c-ab10fd137e5b","Type":"ContainerStarted","Data":"ab2ea51dd84ec5b037f30b6feaa9dede2757a884c5d291d93b28ba85426f1182"} Oct 30 18:25:50.670417 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:50.670330 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-afc614-78f958bbd8-dgrmm" Oct 30 18:25:50.708834 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:50.708603 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-afc614-c4d5cff79-8rgjs" podStartSLOduration=2.708584445 podStartE2EDuration="2.708584445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:25:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:25:50.705435549 +0000 UTC m=+1647.795497888" watchObservedRunningTime="2026-10-30 18:25:50.708584445 +0000 UTC m=+1647.798646784" Oct 30 18:25:50.747206 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:50.747149 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-afc614-78f958bbd8-dgrmm" podStartSLOduration=1.747122209 podStartE2EDuration="1.747122209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:25:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:25:50.746424094 +0000 UTC m=+1647.836486433" watchObservedRunningTime="2026-10-30 18:25:50.747122209 +0000 UTC m=+1647.837184559" Oct 30 18:25:50.796053 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:50.796014 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-be7f9e-74c4c4589f-fv7sd"] Oct 30 18:25:50.797168 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:50.797127 4442 cpu_manager.go:414] "containerMap: removing container" podUID="1ebd21aa-152c-4087-b33b-461678c392a6" containerName="maas-api" Oct 30 18:25:50.797168 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:50.797169 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ebd21aa-152c-4087-b33b-461678c392a6" containerName="maas-api" Oct 30 18:25:50.797337 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:50.797326 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ebd21aa-152c-4087-b33b-461678c392a6" containerName="maas-api" Oct 30 18:25:50.802988 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:50.802963 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be7f9e-74c4c4589f-fv7sd" Oct 30 18:25:50.817423 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:50.817396 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-be7f9e\"" Oct 30 18:25:50.820104 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:50.820053 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-be7f9e-dockercfg-rmhgw\"" Oct 30 18:25:50.828903 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:50.823211 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-be7f9e-74c4c4589f-fv7sd"] Oct 30 18:25:51.094216 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:51.044208 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jldvv\" (UniqueName: \"kubernetes.io/projected/5db1a7fe-484f-4d61-809e-61f10e248d6d-kube-api-access-jldvv\") pod \"payload-pre-processing-e2e-praxis-a-be7f9e-74c4c4589f-fv7sd\" (UID: \"5db1a7fe-484f-4d61-809e-61f10e248d6d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be7f9e-74c4c4589f-fv7sd" Oct 30 18:25:51.104517 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:51.068727 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-be7f9e-576d7866d4-9mpwv"] Oct 30 18:25:51.108405 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:51.108379 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5db1a7fe-484f-4d61-809e-61f10e248d6d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-be7f9e-74c4c4589f-fv7sd\" (UID: \"5db1a7fe-484f-4d61-809e-61f10e248d6d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be7f9e-74c4c4589f-fv7sd" Oct 30 18:25:51.112315 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:51.112294 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-be7f9e-576d7866d4-9mpwv" Oct 30 18:25:51.130355 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:51.130324 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ebd21aa-152c-4087-b33b-461678c392a6" path="/var/lib/kubelet/pods/1ebd21aa-152c-4087-b33b-461678c392a6/volumes" Oct 30 18:25:51.130812 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:51.130791 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-be7f9e-576d7866d4-9mpwv"] Oct 30 18:25:51.210307 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:51.210269 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npqhz\" (UniqueName: \"kubernetes.io/projected/0949007d-4b27-4461-8c2a-fe96493a12aa-kube-api-access-npqhz\") pod \"payload-processing-e2e-praxis-a-be7f9e-576d7866d4-9mpwv\" (UID: \"0949007d-4b27-4461-8c2a-fe96493a12aa\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-be7f9e-576d7866d4-9mpwv" Oct 30 18:25:51.210485 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:51.210329 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5db1a7fe-484f-4d61-809e-61f10e248d6d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-be7f9e-74c4c4589f-fv7sd\" (UID: \"5db1a7fe-484f-4d61-809e-61f10e248d6d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be7f9e-74c4c4589f-fv7sd" Oct 30 18:25:51.210619 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:51.210595 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0949007d-4b27-4461-8c2a-fe96493a12aa-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-be7f9e-576d7866d4-9mpwv\" (UID: \"0949007d-4b27-4461-8c2a-fe96493a12aa\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-be7f9e-576d7866d4-9mpwv" Oct 30 18:25:51.210685 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:51.210655 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jldvv\" (UniqueName: \"kubernetes.io/projected/5db1a7fe-484f-4d61-809e-61f10e248d6d-kube-api-access-jldvv\") pod \"payload-pre-processing-e2e-praxis-a-be7f9e-74c4c4589f-fv7sd\" (UID: \"5db1a7fe-484f-4d61-809e-61f10e248d6d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be7f9e-74c4c4589f-fv7sd" Oct 30 18:25:51.211097 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:51.211078 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5db1a7fe-484f-4d61-809e-61f10e248d6d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-be7f9e-74c4c4589f-fv7sd\" (UID: \"5db1a7fe-484f-4d61-809e-61f10e248d6d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be7f9e-74c4c4589f-fv7sd" Oct 30 18:25:51.229635 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:51.229605 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jldvv\" (UniqueName: \"kubernetes.io/projected/5db1a7fe-484f-4d61-809e-61f10e248d6d-kube-api-access-jldvv\") pod \"payload-pre-processing-e2e-praxis-a-be7f9e-74c4c4589f-fv7sd\" (UID: \"5db1a7fe-484f-4d61-809e-61f10e248d6d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be7f9e-74c4c4589f-fv7sd" Oct 30 18:25:51.312045 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:51.311943 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0949007d-4b27-4461-8c2a-fe96493a12aa-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-be7f9e-576d7866d4-9mpwv\" (UID: \"0949007d-4b27-4461-8c2a-fe96493a12aa\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-be7f9e-576d7866d4-9mpwv" Oct 30 18:25:51.312045 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:51.312028 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npqhz\" (UniqueName: \"kubernetes.io/projected/0949007d-4b27-4461-8c2a-fe96493a12aa-kube-api-access-npqhz\") pod \"payload-processing-e2e-praxis-a-be7f9e-576d7866d4-9mpwv\" (UID: \"0949007d-4b27-4461-8c2a-fe96493a12aa\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-be7f9e-576d7866d4-9mpwv" Oct 30 18:25:51.313375 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:51.313349 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0949007d-4b27-4461-8c2a-fe96493a12aa-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-be7f9e-576d7866d4-9mpwv\" (UID: \"0949007d-4b27-4461-8c2a-fe96493a12aa\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-be7f9e-576d7866d4-9mpwv" Oct 30 18:25:51.338982 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:51.338941 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npqhz\" (UniqueName: \"kubernetes.io/projected/0949007d-4b27-4461-8c2a-fe96493a12aa-kube-api-access-npqhz\") pod \"payload-processing-e2e-praxis-a-be7f9e-576d7866d4-9mpwv\" (UID: \"0949007d-4b27-4461-8c2a-fe96493a12aa\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-be7f9e-576d7866d4-9mpwv" Oct 30 18:25:51.432430 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:51.432387 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be7f9e-74c4c4589f-fv7sd" Oct 30 18:25:51.436855 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:51.436805 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-be7f9e-576d7866d4-9mpwv" Oct 30 18:25:51.762370 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:25:51.762344 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0949007d_4b27_4461_8c2a_fe96493a12aa.slice/crio-998512d7e68076b832692c1b32935838cfc39b280f1a1b6df8d8f430d553249a WatchSource:0}: Error finding container 998512d7e68076b832692c1b32935838cfc39b280f1a1b6df8d8f430d553249a: Status 404 returned error can't find the container with id 998512d7e68076b832692c1b32935838cfc39b280f1a1b6df8d8f430d553249a Oct 30 18:25:51.763203 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:51.763177 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-be7f9e-576d7866d4-9mpwv"] Oct 30 18:25:51.771068 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:51.771045 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-be7f9e-74c4c4589f-fv7sd"] Oct 30 18:25:52.090853 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:52.090772 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-a4f3f0/praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk"] Oct 30 18:25:52.095213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:52.095193 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-a4f3f0/praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk" Oct 30 18:25:52.119093 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:52.119064 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-a4f3f0\"/\"openshift-service-ca.crt\"" Oct 30 18:25:52.124100 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:52.124080 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-a4f3f0/praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk"] Oct 30 18:25:52.130328 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:52.130296 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-a4f3f0\"/\"kube-root-ca.crt\"" Oct 30 18:25:52.130509 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:52.130486 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-a4f3f0\"/\"default-dockercfg-8766h\"" Oct 30 18:25:52.130509 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:52.130416 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-a4f3f0\"/\"praxis-route-a4f3f0-kserve-self-signed-certs\"" Oct 30 18:25:52.239851 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:52.239820 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2174f0e1-8712-4655-86ff-48ea2c635a96-tls-certs\") pod \"praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk\" (UID: \"2174f0e1-8712-4655-86ff-48ea2c635a96\") " pod="ai-tenant-e2e-praxis-a-a4f3f0/praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk" Oct 30 18:25:52.240051 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:52.239927 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2174f0e1-8712-4655-86ff-48ea2c635a96-home\") pod \"praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk\" (UID: \"2174f0e1-8712-4655-86ff-48ea2c635a96\") " pod="ai-tenant-e2e-praxis-a-a4f3f0/praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk" Oct 30 18:25:52.240107 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:52.240065 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2174f0e1-8712-4655-86ff-48ea2c635a96-model-cache\") pod \"praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk\" (UID: \"2174f0e1-8712-4655-86ff-48ea2c635a96\") " pod="ai-tenant-e2e-praxis-a-a4f3f0/praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk" Oct 30 18:25:52.240189 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:52.240119 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwmqt\" (UniqueName: \"kubernetes.io/projected/2174f0e1-8712-4655-86ff-48ea2c635a96-kube-api-access-nwmqt\") pod \"praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk\" (UID: \"2174f0e1-8712-4655-86ff-48ea2c635a96\") " pod="ai-tenant-e2e-praxis-a-a4f3f0/praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk" Oct 30 18:25:52.240246 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:52.240229 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2174f0e1-8712-4655-86ff-48ea2c635a96-tmp-dir\") pod \"praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk\" (UID: \"2174f0e1-8712-4655-86ff-48ea2c635a96\") " pod="ai-tenant-e2e-praxis-a-a4f3f0/praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk" Oct 30 18:25:52.240348 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:52.240332 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2174f0e1-8712-4655-86ff-48ea2c635a96-dshm\") pod \"praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk\" (UID: \"2174f0e1-8712-4655-86ff-48ea2c635a96\") " pod="ai-tenant-e2e-praxis-a-a4f3f0/praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk" Oct 30 18:25:52.356208 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:52.355401 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2174f0e1-8712-4655-86ff-48ea2c635a96-home\") pod \"praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk\" (UID: \"2174f0e1-8712-4655-86ff-48ea2c635a96\") " pod="ai-tenant-e2e-praxis-a-a4f3f0/praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk" Oct 30 18:25:52.356208 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:52.355542 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2174f0e1-8712-4655-86ff-48ea2c635a96-model-cache\") pod \"praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk\" (UID: \"2174f0e1-8712-4655-86ff-48ea2c635a96\") " pod="ai-tenant-e2e-praxis-a-a4f3f0/praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk" Oct 30 18:25:52.356208 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:52.355591 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwmqt\" (UniqueName: \"kubernetes.io/projected/2174f0e1-8712-4655-86ff-48ea2c635a96-kube-api-access-nwmqt\") pod \"praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk\" (UID: \"2174f0e1-8712-4655-86ff-48ea2c635a96\") " pod="ai-tenant-e2e-praxis-a-a4f3f0/praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk" Oct 30 18:25:52.356208 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:52.355633 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2174f0e1-8712-4655-86ff-48ea2c635a96-tmp-dir\") pod \"praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk\" (UID: \"2174f0e1-8712-4655-86ff-48ea2c635a96\") " pod="ai-tenant-e2e-praxis-a-a4f3f0/praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk" Oct 30 18:25:52.356208 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:52.355690 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2174f0e1-8712-4655-86ff-48ea2c635a96-dshm\") pod \"praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk\" (UID: \"2174f0e1-8712-4655-86ff-48ea2c635a96\") " pod="ai-tenant-e2e-praxis-a-a4f3f0/praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk" Oct 30 18:25:52.356208 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:52.355833 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2174f0e1-8712-4655-86ff-48ea2c635a96-tls-certs\") pod \"praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk\" (UID: \"2174f0e1-8712-4655-86ff-48ea2c635a96\") " pod="ai-tenant-e2e-praxis-a-a4f3f0/praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk" Oct 30 18:25:52.360157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:52.357073 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2174f0e1-8712-4655-86ff-48ea2c635a96-home\") pod \"praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk\" (UID: \"2174f0e1-8712-4655-86ff-48ea2c635a96\") " pod="ai-tenant-e2e-praxis-a-a4f3f0/praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk" Oct 30 18:25:52.360157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:52.357346 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2174f0e1-8712-4655-86ff-48ea2c635a96-model-cache\") pod \"praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk\" (UID: \"2174f0e1-8712-4655-86ff-48ea2c635a96\") " pod="ai-tenant-e2e-praxis-a-a4f3f0/praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk" Oct 30 18:25:52.360157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:52.357586 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2174f0e1-8712-4655-86ff-48ea2c635a96-tmp-dir\") pod \"praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk\" (UID: \"2174f0e1-8712-4655-86ff-48ea2c635a96\") " pod="ai-tenant-e2e-praxis-a-a4f3f0/praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk" Oct 30 18:25:52.392157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:52.371812 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2174f0e1-8712-4655-86ff-48ea2c635a96-tls-certs\") pod \"praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk\" (UID: \"2174f0e1-8712-4655-86ff-48ea2c635a96\") " pod="ai-tenant-e2e-praxis-a-a4f3f0/praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk" Oct 30 18:25:52.396062 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:52.396032 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwmqt\" (UniqueName: \"kubernetes.io/projected/2174f0e1-8712-4655-86ff-48ea2c635a96-kube-api-access-nwmqt\") pod \"praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk\" (UID: \"2174f0e1-8712-4655-86ff-48ea2c635a96\") " pod="ai-tenant-e2e-praxis-a-a4f3f0/praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk" Oct 30 18:25:52.404337 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:52.398437 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2174f0e1-8712-4655-86ff-48ea2c635a96-dshm\") pod \"praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk\" (UID: \"2174f0e1-8712-4655-86ff-48ea2c635a96\") " pod="ai-tenant-e2e-praxis-a-a4f3f0/praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk" Oct 30 18:25:52.416208 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:52.411698 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-a4f3f0/praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk" Oct 30 18:25:52.687013 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:52.686962 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-be7f9e-576d7866d4-9mpwv" event={"ID":"0949007d-4b27-4461-8c2a-fe96493a12aa","Type":"ContainerStarted","Data":"23427faa526903c67d248d2ad1a7cad5b257cde17b4e3f12a3485cc01d574238"} Oct 30 18:25:52.687231 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:52.687071 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-be7f9e-576d7866d4-9mpwv" event={"ID":"0949007d-4b27-4461-8c2a-fe96493a12aa","Type":"ContainerStarted","Data":"998512d7e68076b832692c1b32935838cfc39b280f1a1b6df8d8f430d553249a"} Oct 30 18:25:52.687231 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:52.687156 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-be7f9e-576d7866d4-9mpwv" Oct 30 18:25:52.688925 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:52.688903 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be7f9e-74c4c4589f-fv7sd" event={"ID":"5db1a7fe-484f-4d61-809e-61f10e248d6d","Type":"ContainerStarted","Data":"aaeef3ddefab5acde2e8f97e88dd81c74fade1d1ca0effa5fb82fb147c479937"} Oct 30 18:25:52.689034 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:52.688945 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be7f9e-74c4c4589f-fv7sd" event={"ID":"5db1a7fe-484f-4d61-809e-61f10e248d6d","Type":"ContainerStarted","Data":"d8757083005556eee9719d804b082d2974a0806febeba4063c6dbeecb75b152f"} Oct 30 18:25:52.689034 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:52.688977 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be7f9e-74c4c4589f-fv7sd" Oct 30 18:25:52.728759 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:52.728702 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-be7f9e-576d7866d4-9mpwv" podStartSLOduration=2.728686173 podStartE2EDuration="2.728686173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:25:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:25:52.725264895 +0000 UTC m=+1649.815327240" watchObservedRunningTime="2026-10-30 18:25:52.728686173 +0000 UTC m=+1649.818748518" Oct 30 18:25:52.763942 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:52.763916 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-a4f3f0/praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk"] Oct 30 18:25:52.764073 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:25:52.764050 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2174f0e1_8712_4655_86ff_48ea2c635a96.slice/crio-d9525a6f9b4983beeb14e6e6c1072e0b674a37f562761478bd15a2b991ba3e91 WatchSource:0}: Error finding container d9525a6f9b4983beeb14e6e6c1072e0b674a37f562761478bd15a2b991ba3e91: Status 404 returned error can't find the container with id d9525a6f9b4983beeb14e6e6c1072e0b674a37f562761478bd15a2b991ba3e91 Oct 30 18:25:52.770188 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:52.769613 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be7f9e-74c4c4589f-fv7sd" podStartSLOduration=2.7695965190000003 podStartE2EDuration="2.769596519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:25:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:25:52.764283143 +0000 UTC m=+1649.854345484" watchObservedRunningTime="2026-10-30 18:25:52.769596519 +0000 UTC m=+1649.859658865" Oct 30 18:25:53.699247 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:53.699206 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-a4f3f0/praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk" event={"ID":"2174f0e1-8712-4655-86ff-48ea2c635a96","Type":"ContainerStarted","Data":"5923385cfc4f6ab536aee2f88c74805f23ca51bb8f050248453337b31fd884fe"} Oct 30 18:25:53.699736 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:53.699256 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-a4f3f0/praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk" event={"ID":"2174f0e1-8712-4655-86ff-48ea2c635a96","Type":"ContainerStarted","Data":"d9525a6f9b4983beeb14e6e6c1072e0b674a37f562761478bd15a2b991ba3e91"} Oct 30 18:25:53.740120 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:53.740059 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-a4f3f0/praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk" podStartSLOduration=1.7400380279999998 podStartE2EDuration="1.740038028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:25:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:25:53.738969026 +0000 UTC m=+1650.829031447" watchObservedRunningTime="2026-10-30 18:25:53.740038028 +0000 UTC m=+1650.830100369" Oct 30 18:25:54.412605 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:54.412575 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-a4f3f0/praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk" Oct 30 18:25:54.649834 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:54.649806 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-f826dd-66fc94df7b-vd626_6d7a6ab3-2d04-41ec-acc3-df776501aa5e/payload-processing/0.log" Oct 30 18:25:54.857224 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:25:54.857117 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-f826dd-66fc94df7b-vd626_6d7a6ab3-2d04-41ec-acc3-df776501aa5e/payload-processing/0.log" Oct 30 18:26:01.681363 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:01.681334 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-afc614-78f958bbd8-dgrmm" Oct 30 18:26:01.681816 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:01.681426 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-afc614-c4d5cff79-8rgjs" Oct 30 18:26:02.412388 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:02.412357 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-a4f3f0/praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk" Oct 30 18:26:02.427037 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:02.427009 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-a4f3f0/praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk" Oct 30 18:26:02.794073 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:02.793986 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-a4f3f0/praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk" Oct 30 18:26:03.703182 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:03.703156 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be7f9e-74c4c4589f-fv7sd" Oct 30 18:26:03.703626 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:03.703605 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-be7f9e-576d7866d4-9mpwv" Oct 30 18:26:10.550597 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:10.550509 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-afc614-65d5c9f65f-mzfww"] Oct 30 18:26:10.556612 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:10.556585 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-afc614-65d5c9f65f-mzfww" Oct 30 18:26:10.574274 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:10.574246 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-afc614-65d5c9f65f-mzfww"] Oct 30 18:26:10.584245 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:10.579546 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-afc614-serving-cert\"" Oct 30 18:26:10.638314 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:10.638276 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc0f6b52-69e6-422e-bc46-55ee9f7c8353-metrics-certs\") pod \"maas-api-e2e-praxis-b-afc614-65d5c9f65f-mzfww\" (UID: \"bc0f6b52-69e6-422e-bc46-55ee9f7c8353\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-afc614-65d5c9f65f-mzfww" Oct 30 18:26:10.638467 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:10.638428 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc0f6b52-69e6-422e-bc46-55ee9f7c8353-maas-api-tls\") pod \"maas-api-e2e-praxis-b-afc614-65d5c9f65f-mzfww\" (UID: \"bc0f6b52-69e6-422e-bc46-55ee9f7c8353\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-afc614-65d5c9f65f-mzfww" Oct 30 18:26:10.638467 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:10.638456 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc0f6b52-69e6-422e-bc46-55ee9f7c8353-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-afc614-65d5c9f65f-mzfww\" (UID: \"bc0f6b52-69e6-422e-bc46-55ee9f7c8353\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-afc614-65d5c9f65f-mzfww" Oct 30 18:26:10.638560 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:10.638501 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6kmr\" (UniqueName: \"kubernetes.io/projected/bc0f6b52-69e6-422e-bc46-55ee9f7c8353-kube-api-access-p6kmr\") pod \"maas-api-e2e-praxis-b-afc614-65d5c9f65f-mzfww\" (UID: \"bc0f6b52-69e6-422e-bc46-55ee9f7c8353\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-afc614-65d5c9f65f-mzfww" Oct 30 18:26:10.771073 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:10.749969 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc0f6b52-69e6-422e-bc46-55ee9f7c8353-metrics-certs\") pod \"maas-api-e2e-praxis-b-afc614-65d5c9f65f-mzfww\" (UID: \"bc0f6b52-69e6-422e-bc46-55ee9f7c8353\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-afc614-65d5c9f65f-mzfww" Oct 30 18:26:10.771533 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:10.771504 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc0f6b52-69e6-422e-bc46-55ee9f7c8353-maas-api-tls\") pod \"maas-api-e2e-praxis-b-afc614-65d5c9f65f-mzfww\" (UID: \"bc0f6b52-69e6-422e-bc46-55ee9f7c8353\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-afc614-65d5c9f65f-mzfww" Oct 30 18:26:10.771683 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:10.771667 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc0f6b52-69e6-422e-bc46-55ee9f7c8353-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-afc614-65d5c9f65f-mzfww\" (UID: \"bc0f6b52-69e6-422e-bc46-55ee9f7c8353\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-afc614-65d5c9f65f-mzfww" Oct 30 18:26:10.771826 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:10.771812 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6kmr\" (UniqueName: \"kubernetes.io/projected/bc0f6b52-69e6-422e-bc46-55ee9f7c8353-kube-api-access-p6kmr\") pod \"maas-api-e2e-praxis-b-afc614-65d5c9f65f-mzfww\" (UID: \"bc0f6b52-69e6-422e-bc46-55ee9f7c8353\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-afc614-65d5c9f65f-mzfww" Oct 30 18:26:10.772545 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:26:10.772522 4442 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-afc614-serving-cert: secret "maas-api-e2e-praxis-b-afc614-serving-cert" not found Oct 30 18:26:10.772744 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:26:10.772731 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc0f6b52-69e6-422e-bc46-55ee9f7c8353-maas-api-tls podName:bc0f6b52-69e6-422e-bc46-55ee9f7c8353 nodeName:}" failed. No retries permitted until 2026-10-30 18:26:11.272705734 +0000 UTC m=+1668.362768052 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bc0f6b52-69e6-422e-bc46-55ee9f7c8353-maas-api-tls") pod "maas-api-e2e-praxis-b-afc614-65d5c9f65f-mzfww" (UID: "bc0f6b52-69e6-422e-bc46-55ee9f7c8353") : secret "maas-api-e2e-praxis-b-afc614-serving-cert" not found Oct 30 18:26:10.773785 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:10.773762 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc0f6b52-69e6-422e-bc46-55ee9f7c8353-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-afc614-65d5c9f65f-mzfww\" (UID: \"bc0f6b52-69e6-422e-bc46-55ee9f7c8353\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-afc614-65d5c9f65f-mzfww" Oct 30 18:26:10.782116 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:10.780102 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc0f6b52-69e6-422e-bc46-55ee9f7c8353-metrics-certs\") pod \"maas-api-e2e-praxis-b-afc614-65d5c9f65f-mzfww\" (UID: \"bc0f6b52-69e6-422e-bc46-55ee9f7c8353\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-afc614-65d5c9f65f-mzfww" Oct 30 18:26:10.797331 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:10.797160 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6kmr\" (UniqueName: \"kubernetes.io/projected/bc0f6b52-69e6-422e-bc46-55ee9f7c8353-kube-api-access-p6kmr\") pod \"maas-api-e2e-praxis-b-afc614-65d5c9f65f-mzfww\" (UID: \"bc0f6b52-69e6-422e-bc46-55ee9f7c8353\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-afc614-65d5c9f65f-mzfww" Oct 30 18:26:11.278285 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:11.278253 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc0f6b52-69e6-422e-bc46-55ee9f7c8353-maas-api-tls\") pod \"maas-api-e2e-praxis-b-afc614-65d5c9f65f-mzfww\" (UID: \"bc0f6b52-69e6-422e-bc46-55ee9f7c8353\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-afc614-65d5c9f65f-mzfww" Oct 30 18:26:11.280924 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:11.280886 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc0f6b52-69e6-422e-bc46-55ee9f7c8353-maas-api-tls\") pod \"maas-api-e2e-praxis-b-afc614-65d5c9f65f-mzfww\" (UID: \"bc0f6b52-69e6-422e-bc46-55ee9f7c8353\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-afc614-65d5c9f65f-mzfww" Oct 30 18:26:11.497160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:11.476821 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-afc614-65d5c9f65f-mzfww" Oct 30 18:26:11.885163 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:11.882187 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-afc614-65d5c9f65f-mzfww"] Oct 30 18:26:11.911244 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:11.911220 4442 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:26:12.925092 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:12.925059 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-afc614-65d5c9f65f-mzfww" event={"ID":"bc0f6b52-69e6-422e-bc46-55ee9f7c8353","Type":"ContainerStarted","Data":"889114b05ea042f49f0f75850c0d2944188f0c56ebd90345e3b37a2017fb04ff"} Oct 30 18:26:12.925528 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:12.925119 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-afc614-65d5c9f65f-mzfww" event={"ID":"bc0f6b52-69e6-422e-bc46-55ee9f7c8353","Type":"ContainerStarted","Data":"6f66dd97620ebe16d985d2d31dd5a3d8e4929a03dc7a00f37733001fafea0154"} Oct 30 18:26:12.925528 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:12.925165 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-afc614-65d5c9f65f-mzfww" Oct 30 18:26:12.964460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:12.964409 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-afc614-65d5c9f65f-mzfww" podStartSLOduration=2.541187346 podStartE2EDuration="2.964395592s" totalImagesPullingTime="423.208246ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:26:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:26:12.962832451 +0000 UTC m=+1670.052894791" watchObservedRunningTime="2026-10-30 18:26:12.964395592 +0000 UTC m=+1670.054457930" Oct 30 18:26:17.561958 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:26:17.561922 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:26:18.939945 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:18.939909 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-afc614-65d5c9f65f-mzfww" Oct 30 18:26:23.268422 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:23.268389 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr"] Oct 30 18:26:23.268869 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:23.268664 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" podUID="14c086ee-c4ba-48d2-a1b9-62cc8b09cc77" containerName="istio-proxy" containerID="cri-o://5c2faa22a262a69ea313b229e32c09c3980ca9bc697bd9fbfe0e8944eb29a3a5" gracePeriod=30 Oct 30 18:26:23.315688 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:23.315640 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd"] Oct 30 18:26:23.316219 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:23.316155 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" podUID="963a8b12-5921-4d72-9c90-c31daf3d2475" containerName="istio-proxy" containerID="cri-o://bde487d60584e625f8f08fc26f59e5e8ef0ef1a647ce814f3c84e7ae4fc2bf16" gracePeriod=30 Oct 30 18:26:24.715671 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:24.715640 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2"] Oct 30 18:26:24.721317 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:24.721298 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2" Oct 30 18:26:24.738963 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:24.738934 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2"] Oct 30 18:26:24.755875 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:24.744984 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-e9b146-serving-cert\"" Oct 30 18:26:24.808095 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:24.808050 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ed27292-34ae-4a6a-86cb-dbf81d903a49-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2\" (UID: \"4ed27292-34ae-4a6a-86cb-dbf81d903a49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2" Oct 30 18:26:24.808323 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:24.808188 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ed27292-34ae-4a6a-86cb-dbf81d903a49-metrics-certs\") pod \"maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2\" (UID: \"4ed27292-34ae-4a6a-86cb-dbf81d903a49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2" Oct 30 18:26:24.808323 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:24.808303 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-454v5\" (UniqueName: \"kubernetes.io/projected/4ed27292-34ae-4a6a-86cb-dbf81d903a49-kube-api-access-454v5\") pod \"maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2\" (UID: \"4ed27292-34ae-4a6a-86cb-dbf81d903a49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2" Oct 30 18:26:24.808440 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:24.808383 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ed27292-34ae-4a6a-86cb-dbf81d903a49-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2\" (UID: \"4ed27292-34ae-4a6a-86cb-dbf81d903a49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2" Oct 30 18:26:24.932123 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:24.932068 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ed27292-34ae-4a6a-86cb-dbf81d903a49-metrics-certs\") pod \"maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2\" (UID: \"4ed27292-34ae-4a6a-86cb-dbf81d903a49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2" Oct 30 18:26:24.932527 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:24.932503 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-454v5\" (UniqueName: \"kubernetes.io/projected/4ed27292-34ae-4a6a-86cb-dbf81d903a49-kube-api-access-454v5\") pod \"maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2\" (UID: \"4ed27292-34ae-4a6a-86cb-dbf81d903a49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2" Oct 30 18:26:24.932739 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:24.932708 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ed27292-34ae-4a6a-86cb-dbf81d903a49-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2\" (UID: \"4ed27292-34ae-4a6a-86cb-dbf81d903a49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2" Oct 30 18:26:24.932940 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:24.932926 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ed27292-34ae-4a6a-86cb-dbf81d903a49-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2\" (UID: \"4ed27292-34ae-4a6a-86cb-dbf81d903a49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2" Oct 30 18:26:24.934042 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:24.934018 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ed27292-34ae-4a6a-86cb-dbf81d903a49-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2\" (UID: \"4ed27292-34ae-4a6a-86cb-dbf81d903a49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2" Oct 30 18:26:24.938348 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:26:24.937913 4442 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9b146-serving-cert: secret "maas-api-e2e-ipp-a-e9b146-serving-cert" not found Oct 30 18:26:24.938567 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:26:24.938554 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ed27292-34ae-4a6a-86cb-dbf81d903a49-maas-api-tls podName:4ed27292-34ae-4a6a-86cb-dbf81d903a49 nodeName:}" failed. No retries permitted until 2026-10-30 18:26:25.438530393 +0000 UTC m=+1682.528592727 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4ed27292-34ae-4a6a-86cb-dbf81d903a49-maas-api-tls") pod "maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2" (UID: "4ed27292-34ae-4a6a-86cb-dbf81d903a49") : secret "maas-api-e2e-ipp-a-e9b146-serving-cert" not found Oct 30 18:26:24.968556 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:24.968486 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-454v5\" (UniqueName: \"kubernetes.io/projected/4ed27292-34ae-4a6a-86cb-dbf81d903a49-kube-api-access-454v5\") pod \"maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2\" (UID: \"4ed27292-34ae-4a6a-86cb-dbf81d903a49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2" Oct 30 18:26:24.990507 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:24.990477 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ed27292-34ae-4a6a-86cb-dbf81d903a49-metrics-certs\") pod \"maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2\" (UID: \"4ed27292-34ae-4a6a-86cb-dbf81d903a49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2" Oct 30 18:26:25.440543 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:25.440504 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ed27292-34ae-4a6a-86cb-dbf81d903a49-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2\" (UID: \"4ed27292-34ae-4a6a-86cb-dbf81d903a49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2" Oct 30 18:26:25.443424 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:25.443394 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ed27292-34ae-4a6a-86cb-dbf81d903a49-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2\" (UID: \"4ed27292-34ae-4a6a-86cb-dbf81d903a49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2" Oct 30 18:26:25.638286 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:25.638245 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2" Oct 30 18:26:26.012376 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:26.012350 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2"] Oct 30 18:26:26.081591 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:26.081559 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2" event={"ID":"4ed27292-34ae-4a6a-86cb-dbf81d903a49","Type":"ContainerStarted","Data":"6f7ef4c0250a9af4b63c60ead806781d58eca056b4ed9d2a9d72649c67edc301"} Oct 30 18:26:27.089509 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:27.089477 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2" event={"ID":"4ed27292-34ae-4a6a-86cb-dbf81d903a49","Type":"ContainerStarted","Data":"b7dd920b8a2155d4ee12d9c2677502e38f98dceda09eb8f0edc0c9ad6cd78368"} Oct 30 18:26:27.089893 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:27.089589 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2" Oct 30 18:26:27.129118 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:27.129055 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2" podStartSLOduration=2.781743399 podStartE2EDuration="3.12903633s" totalImagesPullingTime="347.292931ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:26:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:26:27.127781266 +0000 UTC m=+1684.217843608" watchObservedRunningTime="2026-10-30 18:26:27.12903633 +0000 UTC m=+1684.219098671" Oct 30 18:26:28.755877 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.755846 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" Oct 30 18:26:28.819216 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.818769 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" Oct 30 18:26:28.854786 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.854759 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/963a8b12-5921-4d72-9c90-c31daf3d2475-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/963a8b12-5921-4d72-9c90-c31daf3d2475-workload-socket\") pod \"963a8b12-5921-4d72-9c90-c31daf3d2475\" (UID: \"963a8b12-5921-4d72-9c90-c31daf3d2475\") " Oct 30 18:26:28.854967 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.854798 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-istiod-ca-cert\") pod \"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77\" (UID: \"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77\") " Oct 30 18:26:28.854967 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.854832 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-istio-data\") pod \"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77\" (UID: \"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77\") " Oct 30 18:26:28.854967 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.854883 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/963a8b12-5921-4d72-9c90-c31daf3d2475-kube-api-access-n7nf5\" (UniqueName: \"kubernetes.io/projected/963a8b12-5921-4d72-9c90-c31daf3d2475-kube-api-access-n7nf5\") pod \"963a8b12-5921-4d72-9c90-c31daf3d2475\" (UID: \"963a8b12-5921-4d72-9c90-c31daf3d2475\") " Oct 30 18:26:28.854967 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.854908 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-istio-token\" (UniqueName: \"kubernetes.io/projected/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-istio-token\") pod \"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77\" (UID: \"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77\") " Oct 30 18:26:28.854967 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.854955 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/963a8b12-5921-4d72-9c90-c31daf3d2475-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/963a8b12-5921-4d72-9c90-c31daf3d2475-istio-podinfo\") pod \"963a8b12-5921-4d72-9c90-c31daf3d2475\" (UID: \"963a8b12-5921-4d72-9c90-c31daf3d2475\") " Oct 30 18:26:28.855294 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.854983 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/963a8b12-5921-4d72-9c90-c31daf3d2475-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/963a8b12-5921-4d72-9c90-c31daf3d2475-workload-certs\") pod \"963a8b12-5921-4d72-9c90-c31daf3d2475\" (UID: \"963a8b12-5921-4d72-9c90-c31daf3d2475\") " Oct 30 18:26:28.855294 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.855026 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-workload-certs\") pod \"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77\" (UID: \"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77\") " Oct 30 18:26:28.855294 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.855076 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-istio-envoy\") pod \"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77\" (UID: \"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77\") " Oct 30 18:26:28.855294 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.855162 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-workload-socket\") pod \"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77\" (UID: \"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77\") " Oct 30 18:26:28.855294 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.855161 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/963a8b12-5921-4d72-9c90-c31daf3d2475-workload-socket" pod "963a8b12-5921-4d72-9c90-c31daf3d2475" (UID: "963a8b12-5921-4d72-9c90-c31daf3d2475"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:28.855294 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.855187 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-istio-data" pod "14c086ee-c4ba-48d2-a1b9-62cc8b09cc77" (UID: "14c086ee-c4ba-48d2-a1b9-62cc8b09cc77"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:28.855294 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.855235 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/963a8b12-5921-4d72-9c90-c31daf3d2475-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/963a8b12-5921-4d72-9c90-c31daf3d2475-credential-socket\") pod \"963a8b12-5921-4d72-9c90-c31daf3d2475\" (UID: \"963a8b12-5921-4d72-9c90-c31daf3d2475\") " Oct 30 18:26:28.855294 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.855277 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/963a8b12-5921-4d72-9c90-c31daf3d2475-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/963a8b12-5921-4d72-9c90-c31daf3d2475-istiod-ca-cert\") pod \"963a8b12-5921-4d72-9c90-c31daf3d2475\" (UID: \"963a8b12-5921-4d72-9c90-c31daf3d2475\") " Oct 30 18:26:28.855756 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.855295 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-istiod-ca-cert" pod "14c086ee-c4ba-48d2-a1b9-62cc8b09cc77" (UID: "14c086ee-c4ba-48d2-a1b9-62cc8b09cc77"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:28.855756 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.855346 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/963a8b12-5921-4d72-9c90-c31daf3d2475-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/963a8b12-5921-4d72-9c90-c31daf3d2475-istio-envoy\") pod \"963a8b12-5921-4d72-9c90-c31daf3d2475\" (UID: \"963a8b12-5921-4d72-9c90-c31daf3d2475\") " Oct 30 18:26:28.855756 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.855348 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-workload-certs" pod "14c086ee-c4ba-48d2-a1b9-62cc8b09cc77" (UID: "14c086ee-c4ba-48d2-a1b9-62cc8b09cc77"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:28.855756 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.855377 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/963a8b12-5921-4d72-9c90-c31daf3d2475-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/963a8b12-5921-4d72-9c90-c31daf3d2475-istio-data\") pod \"963a8b12-5921-4d72-9c90-c31daf3d2475\" (UID: \"963a8b12-5921-4d72-9c90-c31daf3d2475\") " Oct 30 18:26:28.855756 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.855436 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-kube-api-access-mlfh9\" (UniqueName: \"kubernetes.io/projected/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-kube-api-access-mlfh9\") pod \"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77\" (UID: \"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77\") " Oct 30 18:26:28.855756 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.855523 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-credential-socket\") pod \"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77\" (UID: \"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77\") " Oct 30 18:26:28.855756 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.855542 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-workload-socket" pod "14c086ee-c4ba-48d2-a1b9-62cc8b09cc77" (UID: "14c086ee-c4ba-48d2-a1b9-62cc8b09cc77"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:28.855756 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.855577 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-istio-podinfo\") pod \"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77\" (UID: \"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77\") " Oct 30 18:26:28.855756 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.855669 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/963a8b12-5921-4d72-9c90-c31daf3d2475-istio-token\" (UniqueName: \"kubernetes.io/projected/963a8b12-5921-4d72-9c90-c31daf3d2475-istio-token\") pod \"963a8b12-5921-4d72-9c90-c31daf3d2475\" (UID: \"963a8b12-5921-4d72-9c90-c31daf3d2475\") " Oct 30 18:26:28.855756 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.855694 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/963a8b12-5921-4d72-9c90-c31daf3d2475-workload-certs" pod "963a8b12-5921-4d72-9c90-c31daf3d2475" (UID: "963a8b12-5921-4d72-9c90-c31daf3d2475"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:28.856325 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.855885 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-credential-socket" pod "14c086ee-c4ba-48d2-a1b9-62cc8b09cc77" (UID: "14c086ee-c4ba-48d2-a1b9-62cc8b09cc77"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:28.856752 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.856495 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.856752 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.856521 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/963a8b12-5921-4d72-9c90-c31daf3d2475-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.856752 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.856536 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.856752 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.856550 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.856752 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.856564 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/963a8b12-5921-4d72-9c90-c31daf3d2475-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.856752 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.856577 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.856752 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.856592 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.856752 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.856691 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/963a8b12-5921-4d72-9c90-c31daf3d2475-istio-data" pod "963a8b12-5921-4d72-9c90-c31daf3d2475" (UID: "963a8b12-5921-4d72-9c90-c31daf3d2475"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:28.857287 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.856985 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/963a8b12-5921-4d72-9c90-c31daf3d2475-credential-socket" pod "963a8b12-5921-4d72-9c90-c31daf3d2475" (UID: "963a8b12-5921-4d72-9c90-c31daf3d2475"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:28.857287 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.857068 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/963a8b12-5921-4d72-9c90-c31daf3d2475-istiod-ca-cert" pod "963a8b12-5921-4d72-9c90-c31daf3d2475" (UID: "963a8b12-5921-4d72-9c90-c31daf3d2475"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:28.858912 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.858648 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-istio-podinfo" pod "14c086ee-c4ba-48d2-a1b9-62cc8b09cc77" (UID: "14c086ee-c4ba-48d2-a1b9-62cc8b09cc77"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:26:28.858912 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.858651 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-istio-token" pod "14c086ee-c4ba-48d2-a1b9-62cc8b09cc77" (UID: "14c086ee-c4ba-48d2-a1b9-62cc8b09cc77"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:28.858912 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.858826 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-istio-envoy" pod "14c086ee-c4ba-48d2-a1b9-62cc8b09cc77" (UID: "14c086ee-c4ba-48d2-a1b9-62cc8b09cc77"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:28.861903 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.861212 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/963a8b12-5921-4d72-9c90-c31daf3d2475-istio-token" pod "963a8b12-5921-4d72-9c90-c31daf3d2475" (UID: "963a8b12-5921-4d72-9c90-c31daf3d2475"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:28.861903 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.861350 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/963a8b12-5921-4d72-9c90-c31daf3d2475-istio-envoy" pod "963a8b12-5921-4d72-9c90-c31daf3d2475" (UID: "963a8b12-5921-4d72-9c90-c31daf3d2475"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:28.861903 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.861437 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/963a8b12-5921-4d72-9c90-c31daf3d2475-kube-api-access-n7nf5" pod "963a8b12-5921-4d72-9c90-c31daf3d2475" (UID: "963a8b12-5921-4d72-9c90-c31daf3d2475"). InnerVolumeSpecName "kube-api-access-n7nf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:28.861903 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.861626 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-kube-api-access-mlfh9" pod "14c086ee-c4ba-48d2-a1b9-62cc8b09cc77" (UID: "14c086ee-c4ba-48d2-a1b9-62cc8b09cc77"). InnerVolumeSpecName "kube-api-access-mlfh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:28.862903 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.862882 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/963a8b12-5921-4d72-9c90-c31daf3d2475-istio-podinfo" pod "963a8b12-5921-4d72-9c90-c31daf3d2475" (UID: "963a8b12-5921-4d72-9c90-c31daf3d2475"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:26:28.957557 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.957522 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.957557 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.957551 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/963a8b12-5921-4d72-9c90-c31daf3d2475-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.957557 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.957561 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/963a8b12-5921-4d72-9c90-c31daf3d2475-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.957811 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.957571 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/963a8b12-5921-4d72-9c90-c31daf3d2475-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.957811 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.957579 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/963a8b12-5921-4d72-9c90-c31daf3d2475-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.957811 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.957588 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlfh9\" (UniqueName: \"kubernetes.io/projected/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-kube-api-access-mlfh9\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.957811 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.957597 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.957811 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.957605 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/963a8b12-5921-4d72-9c90-c31daf3d2475-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.957811 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.957612 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7nf5\" (UniqueName: \"kubernetes.io/projected/963a8b12-5921-4d72-9c90-c31daf3d2475-kube-api-access-n7nf5\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.957811 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.957620 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.957811 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:28.957627 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/963a8b12-5921-4d72-9c90-c31daf3d2475-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:26:29.105430 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:29.105400 4442 generic.go:356] "Generic (PLEG): container finished" podID="963a8b12-5921-4d72-9c90-c31daf3d2475" containerID="bde487d60584e625f8f08fc26f59e5e8ef0ef1a647ce814f3c84e7ae4fc2bf16" exitCode=0 Oct 30 18:26:29.105566 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:29.105442 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" Oct 30 18:26:29.105566 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:29.105490 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" event={"ID":"963a8b12-5921-4d72-9c90-c31daf3d2475","Type":"ContainerDied","Data":"bde487d60584e625f8f08fc26f59e5e8ef0ef1a647ce814f3c84e7ae4fc2bf16"} Oct 30 18:26:29.105566 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:29.105562 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd" event={"ID":"963a8b12-5921-4d72-9c90-c31daf3d2475","Type":"ContainerDied","Data":"5c3c6b24cc14f426422770564cc006eab3613f8b6c92339b98306582f296bfec"} Oct 30 18:26:29.105749 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:29.105597 4442 scope.go:122] "RemoveContainer" containerID="bde487d60584e625f8f08fc26f59e5e8ef0ef1a647ce814f3c84e7ae4fc2bf16" Oct 30 18:26:29.107105 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:29.107079 4442 generic.go:356] "Generic (PLEG): container finished" podID="14c086ee-c4ba-48d2-a1b9-62cc8b09cc77" containerID="5c2faa22a262a69ea313b229e32c09c3980ca9bc697bd9fbfe0e8944eb29a3a5" exitCode=0 Oct 30 18:26:29.107229 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:29.107167 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" Oct 30 18:26:29.107297 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:29.107164 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" event={"ID":"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77","Type":"ContainerDied","Data":"5c2faa22a262a69ea313b229e32c09c3980ca9bc697bd9fbfe0e8944eb29a3a5"} Oct 30 18:26:29.107357 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:29.107308 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr" event={"ID":"14c086ee-c4ba-48d2-a1b9-62cc8b09cc77","Type":"ContainerDied","Data":"415a184bc7530bbb58630723ac2c21a8761fd6464061f1b34e91f911aa186f98"} Oct 30 18:26:29.117022 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:29.117003 4442 scope.go:122] "RemoveContainer" containerID="bde487d60584e625f8f08fc26f59e5e8ef0ef1a647ce814f3c84e7ae4fc2bf16" Oct 30 18:26:29.117337 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:26:29.117317 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bde487d60584e625f8f08fc26f59e5e8ef0ef1a647ce814f3c84e7ae4fc2bf16\": container with ID starting with bde487d60584e625f8f08fc26f59e5e8ef0ef1a647ce814f3c84e7ae4fc2bf16 not found: ID does not exist" containerID="bde487d60584e625f8f08fc26f59e5e8ef0ef1a647ce814f3c84e7ae4fc2bf16" Oct 30 18:26:29.117385 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:29.117346 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bde487d60584e625f8f08fc26f59e5e8ef0ef1a647ce814f3c84e7ae4fc2bf16"} err="failed to get container status \"bde487d60584e625f8f08fc26f59e5e8ef0ef1a647ce814f3c84e7ae4fc2bf16\": rpc error: code = NotFound desc = could not find container \"bde487d60584e625f8f08fc26f59e5e8ef0ef1a647ce814f3c84e7ae4fc2bf16\": container with ID starting with bde487d60584e625f8f08fc26f59e5e8ef0ef1a647ce814f3c84e7ae4fc2bf16 not found: ID does not exist" Oct 30 18:26:29.117385 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:29.117363 4442 scope.go:122] "RemoveContainer" containerID="5c2faa22a262a69ea313b229e32c09c3980ca9bc697bd9fbfe0e8944eb29a3a5" Oct 30 18:26:29.132083 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:29.132061 4442 scope.go:122] "RemoveContainer" containerID="5c2faa22a262a69ea313b229e32c09c3980ca9bc697bd9fbfe0e8944eb29a3a5" Oct 30 18:26:29.132490 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:26:29.132467 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c2faa22a262a69ea313b229e32c09c3980ca9bc697bd9fbfe0e8944eb29a3a5\": container with ID starting with 5c2faa22a262a69ea313b229e32c09c3980ca9bc697bd9fbfe0e8944eb29a3a5 not found: ID does not exist" containerID="5c2faa22a262a69ea313b229e32c09c3980ca9bc697bd9fbfe0e8944eb29a3a5" Oct 30 18:26:29.132566 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:29.132497 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c2faa22a262a69ea313b229e32c09c3980ca9bc697bd9fbfe0e8944eb29a3a5"} err="failed to get container status \"5c2faa22a262a69ea313b229e32c09c3980ca9bc697bd9fbfe0e8944eb29a3a5\": rpc error: code = NotFound desc = could not find container \"5c2faa22a262a69ea313b229e32c09c3980ca9bc697bd9fbfe0e8944eb29a3a5\": container with ID starting with 5c2faa22a262a69ea313b229e32c09c3980ca9bc697bd9fbfe0e8944eb29a3a5 not found: ID does not exist" Oct 30 18:26:29.183553 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:29.183522 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr"] Oct 30 18:26:29.187309 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:29.187285 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fkxzr"] Oct 30 18:26:29.218879 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:29.218849 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd"] Oct 30 18:26:29.223622 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:29.223601 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mrvcd"] Oct 30 18:26:31.088896 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:31.088856 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14c086ee-c4ba-48d2-a1b9-62cc8b09cc77" path="/var/lib/kubelet/pods/14c086ee-c4ba-48d2-a1b9-62cc8b09cc77/volumes" Oct 30 18:26:31.089475 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:31.089459 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="963a8b12-5921-4d72-9c90-c31daf3d2475" path="/var/lib/kubelet/pods/963a8b12-5921-4d72-9c90-c31daf3d2475/volumes" Oct 30 18:26:33.099670 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:33.099625 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2" podUID="4ed27292-34ae-4a6a-86cb-dbf81d903a49" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.168:8443/health\": dial tcp 10.134.0.168:8443: connect: connection refused" Oct 30 18:26:37.276217 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:37.276184 4442 generic.go:356] "Generic (PLEG): container finished" podID="4ed27292-34ae-4a6a-86cb-dbf81d903a49" containerID="b7dd920b8a2155d4ee12d9c2677502e38f98dceda09eb8f0edc0c9ad6cd78368" exitCode=1 Oct 30 18:26:37.276723 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:37.276321 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2" event={"ID":"4ed27292-34ae-4a6a-86cb-dbf81d903a49","Type":"ContainerDied","Data":"b7dd920b8a2155d4ee12d9c2677502e38f98dceda09eb8f0edc0c9ad6cd78368"} Oct 30 18:26:37.276845 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:37.276827 4442 scope.go:122] "RemoveContainer" containerID="b7dd920b8a2155d4ee12d9c2677502e38f98dceda09eb8f0edc0c9ad6cd78368" Oct 30 18:26:38.287753 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:38.287715 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2" event={"ID":"4ed27292-34ae-4a6a-86cb-dbf81d903a49","Type":"ContainerStarted","Data":"49eae0041851137c3ea92be92e86b5c3f4d066f21827ce2652b260ffb0900216"} Oct 30 18:26:38.288330 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:38.287922 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2" Oct 30 18:26:39.298016 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:39.297982 4442 generic.go:356] "Generic (PLEG): container finished" podID="29c8e32e-e676-4b79-9643-25c88f1a6f1a" containerID="190401a7ae3d22201b3afd9adfbff94e809554cf2cd17a81774e38683963bb1b" exitCode=0 Oct 30 18:26:39.298571 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:39.298058 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" event={"ID":"29c8e32e-e676-4b79-9643-25c88f1a6f1a","Type":"ContainerDied","Data":"190401a7ae3d22201b3afd9adfbff94e809554cf2cd17a81774e38683963bb1b"} Oct 30 18:26:39.298645 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:39.298621 4442 scope.go:122] "RemoveContainer" containerID="190401a7ae3d22201b3afd9adfbff94e809554cf2cd17a81774e38683963bb1b" Oct 30 18:26:39.300420 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:39.300389 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:26:39.300551 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:39.300450 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:26:39.300551 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:39.300483 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:26:40.308087 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:40.308057 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" event={"ID":"29c8e32e-e676-4b79-9643-25c88f1a6f1a","Type":"ContainerStarted","Data":"c3a8fcecf38d66506e58a5204238f86d39b9e3ee766042afd1b206c1bf7aea89"} Oct 30 18:26:40.317001 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:40.316968 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" Oct 30 18:26:40.317194 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:40.317091 4442 patch_prober.go:29] interesting pod/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 30 18:26:40.317277 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:40.317216 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" podUID="29c8e32e-e676-4b79-9643-25c88f1a6f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 30 18:26:40.317789 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:40.317762 4442 generic.go:356] "Generic (PLEG): container finished" podID="54fe8523-0ed9-477e-a23d-8c7d624ba2d4" containerID="be3cc516aab817d15f7d3dfd489b7545d6353c939bff37985ee155f6f00919df" exitCode=0 Oct 30 18:26:40.317890 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:40.317824 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" event={"ID":"54fe8523-0ed9-477e-a23d-8c7d624ba2d4","Type":"ContainerDied","Data":"be3cc516aab817d15f7d3dfd489b7545d6353c939bff37985ee155f6f00919df"} Oct 30 18:26:40.318504 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:40.318481 4442 scope.go:122] "RemoveContainer" containerID="be3cc516aab817d15f7d3dfd489b7545d6353c939bff37985ee155f6f00919df" Oct 30 18:26:40.320410 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:40.320382 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:26:40.320472 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:40.320451 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:26:40.320507 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:40.320492 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:26:41.317166 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:41.317094 4442 patch_prober.go:29] interesting pod/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 30 18:26:41.317683 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:41.317200 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" podUID="29c8e32e-e676-4b79-9643-25c88f1a6f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 30 18:26:41.335518 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:41.335475 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" event={"ID":"54fe8523-0ed9-477e-a23d-8c7d624ba2d4","Type":"ContainerStarted","Data":"76f1b7feda32ae9413cc5ec4e83026335b1a37fc05265995feb28e3c248f0e14"} Oct 30 18:26:41.663790 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:41.663754 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" Oct 30 18:26:41.664167 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:41.664115 4442 patch_prober.go:29] interesting pod/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 18:26:41.664382 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:41.664356 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" podUID="54fe8523-0ed9-477e-a23d-8c7d624ba2d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 18:26:42.317113 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:42.317075 4442 patch_prober.go:29] interesting pod/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 30 18:26:42.317304 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:42.317147 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" podUID="29c8e32e-e676-4b79-9643-25c88f1a6f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 30 18:26:42.657599 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:42.657561 4442 patch_prober.go:29] interesting pod/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 18:26:42.657914 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:42.657886 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" podUID="54fe8523-0ed9-477e-a23d-8c7d624ba2d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 18:26:43.317261 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:43.317229 4442 patch_prober.go:29] interesting pod/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 30 18:26:43.317448 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:43.317290 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" podUID="29c8e32e-e676-4b79-9643-25c88f1a6f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 30 18:26:43.656888 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:43.656851 4442 patch_prober.go:29] interesting pod/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 18:26:43.657206 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:43.657177 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" podUID="54fe8523-0ed9-477e-a23d-8c7d624ba2d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 18:26:44.303953 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:44.303925 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2" Oct 30 18:26:44.317225 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:44.317197 4442 patch_prober.go:29] interesting pod/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 30 18:26:44.317415 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:44.317396 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" podUID="29c8e32e-e676-4b79-9643-25c88f1a6f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 30 18:26:44.656971 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:44.656844 4442 patch_prober.go:29] interesting pod/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 18:26:44.657695 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:44.657657 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" podUID="54fe8523-0ed9-477e-a23d-8c7d624ba2d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 18:26:45.317324 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:45.317271 4442 patch_prober.go:29] interesting pod/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 30 18:26:45.317514 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:45.317342 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" podUID="29c8e32e-e676-4b79-9643-25c88f1a6f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 30 18:26:45.586580 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:45.586477 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" Oct 30 18:26:45.658592 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:45.658552 4442 patch_prober.go:29] interesting pod/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 18:26:45.659226 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:45.659185 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" podUID="54fe8523-0ed9-477e-a23d-8c7d624ba2d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 18:26:46.321760 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:46.321736 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" Oct 30 18:26:46.410827 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:46.410798 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" Oct 30 18:26:46.593698 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:46.593613 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" Oct 30 18:26:46.656716 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:46.656684 4442 patch_prober.go:29] interesting pod/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 18:26:46.656902 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:46.656740 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" podUID="54fe8523-0ed9-477e-a23d-8c7d624ba2d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 18:26:47.561627 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:26:47.561589 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:26:47.662108 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:47.662077 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" Oct 30 18:26:47.663520 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:47.663497 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" Oct 30 18:26:49.877268 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:49.877230 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f42594-666cc4c777-dm62x"] Oct 30 18:26:49.878096 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:49.878049 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f42594-666cc4c777-dm62x" podUID="f99c368a-c8a3-49be-afd7-c68980a7fa39" containerName="maas-api" containerID="cri-o://2fc426065f9cb3e2637ddd913495ca5ea97494588545b1a9e82a8c576126afbd" gracePeriod=30 Oct 30 18:26:50.544028 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:50.543837 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f42594-666cc4c777-dm62x" Oct 30 18:26:50.549971 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:50.549822 4442 generic.go:356] "Generic (PLEG): container finished" podID="f99c368a-c8a3-49be-afd7-c68980a7fa39" containerID="2fc426065f9cb3e2637ddd913495ca5ea97494588545b1a9e82a8c576126afbd" exitCode=0 Oct 30 18:26:50.549971 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:50.549876 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f42594-666cc4c777-dm62x" event={"ID":"f99c368a-c8a3-49be-afd7-c68980a7fa39","Type":"ContainerDied","Data":"2fc426065f9cb3e2637ddd913495ca5ea97494588545b1a9e82a8c576126afbd"} Oct 30 18:26:50.549971 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:50.549954 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f42594-666cc4c777-dm62x" event={"ID":"f99c368a-c8a3-49be-afd7-c68980a7fa39","Type":"ContainerDied","Data":"527a8d52504eccbef3d4d421c2236c0607e18eb1a039bd45832c77e54ccfdb77"} Oct 30 18:26:50.549971 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:50.549978 4442 scope.go:122] "RemoveContainer" containerID="2fc426065f9cb3e2637ddd913495ca5ea97494588545b1a9e82a8c576126afbd" Oct 30 18:26:50.570537 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:50.570441 4442 scope.go:122] "RemoveContainer" containerID="2fc426065f9cb3e2637ddd913495ca5ea97494588545b1a9e82a8c576126afbd" Oct 30 18:26:50.570845 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:26:50.570821 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fc426065f9cb3e2637ddd913495ca5ea97494588545b1a9e82a8c576126afbd\": container with ID starting with 2fc426065f9cb3e2637ddd913495ca5ea97494588545b1a9e82a8c576126afbd not found: ID does not exist" containerID="2fc426065f9cb3e2637ddd913495ca5ea97494588545b1a9e82a8c576126afbd" Oct 30 18:26:50.570956 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:50.570861 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fc426065f9cb3e2637ddd913495ca5ea97494588545b1a9e82a8c576126afbd"} err="failed to get container status \"2fc426065f9cb3e2637ddd913495ca5ea97494588545b1a9e82a8c576126afbd\": rpc error: code = NotFound desc = could not find container \"2fc426065f9cb3e2637ddd913495ca5ea97494588545b1a9e82a8c576126afbd\": container with ID starting with 2fc426065f9cb3e2637ddd913495ca5ea97494588545b1a9e82a8c576126afbd not found: ID does not exist" Oct 30 18:26:50.619749 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:50.619027 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f99c368a-c8a3-49be-afd7-c68980a7fa39-kube-api-access-phtls\" (UniqueName: \"kubernetes.io/projected/f99c368a-c8a3-49be-afd7-c68980a7fa39-kube-api-access-phtls\") pod \"f99c368a-c8a3-49be-afd7-c68980a7fa39\" (UID: \"f99c368a-c8a3-49be-afd7-c68980a7fa39\") " Oct 30 18:26:50.620204 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:50.620173 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f99c368a-c8a3-49be-afd7-c68980a7fa39-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f99c368a-c8a3-49be-afd7-c68980a7fa39-metrics-certs\") pod \"f99c368a-c8a3-49be-afd7-c68980a7fa39\" (UID: \"f99c368a-c8a3-49be-afd7-c68980a7fa39\") " Oct 30 18:26:50.620393 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:50.620333 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f99c368a-c8a3-49be-afd7-c68980a7fa39-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f99c368a-c8a3-49be-afd7-c68980a7fa39-maas-api-tls\") pod \"f99c368a-c8a3-49be-afd7-c68980a7fa39\" (UID: \"f99c368a-c8a3-49be-afd7-c68980a7fa39\") " Oct 30 18:26:50.620520 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:50.620394 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f99c368a-c8a3-49be-afd7-c68980a7fa39-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f99c368a-c8a3-49be-afd7-c68980a7fa39-openshift-service-ca\") pod \"f99c368a-c8a3-49be-afd7-c68980a7fa39\" (UID: \"f99c368a-c8a3-49be-afd7-c68980a7fa39\") " Oct 30 18:26:50.621526 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:50.621214 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f99c368a-c8a3-49be-afd7-c68980a7fa39-openshift-service-ca" pod "f99c368a-c8a3-49be-afd7-c68980a7fa39" (UID: "f99c368a-c8a3-49be-afd7-c68980a7fa39"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:50.641618 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:50.641567 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f99c368a-c8a3-49be-afd7-c68980a7fa39-metrics-certs" pod "f99c368a-c8a3-49be-afd7-c68980a7fa39" (UID: "f99c368a-c8a3-49be-afd7-c68980a7fa39"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:26:50.641845 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:50.641802 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f99c368a-c8a3-49be-afd7-c68980a7fa39-kube-api-access-phtls" pod "f99c368a-c8a3-49be-afd7-c68980a7fa39" (UID: "f99c368a-c8a3-49be-afd7-c68980a7fa39"). InnerVolumeSpecName "kube-api-access-phtls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:50.643377 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:50.643343 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f99c368a-c8a3-49be-afd7-c68980a7fa39-maas-api-tls" pod "f99c368a-c8a3-49be-afd7-c68980a7fa39" (UID: "f99c368a-c8a3-49be-afd7-c68980a7fa39"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:26:50.721812 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:50.721779 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f99c368a-c8a3-49be-afd7-c68980a7fa39-metrics-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:26:50.721812 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:50.721805 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f99c368a-c8a3-49be-afd7-c68980a7fa39-maas-api-tls\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:26:50.721812 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:50.721815 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f99c368a-c8a3-49be-afd7-c68980a7fa39-openshift-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:26:50.722069 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:50.721826 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phtls\" (UniqueName: \"kubernetes.io/projected/f99c368a-c8a3-49be-afd7-c68980a7fa39-kube-api-access-phtls\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:26:51.559188 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:51.559152 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f42594-666cc4c777-dm62x" Oct 30 18:26:51.611594 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:51.611560 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f42594-666cc4c777-dm62x"] Oct 30 18:26:51.622669 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:51.622638 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f42594-666cc4c777-dm62x"] Oct 30 18:26:53.069045 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:26:53.069007 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f99c368a-c8a3-49be-afd7-c68980a7fa39" path="/var/lib/kubelet/pods/f99c368a-c8a3-49be-afd7-c68980a7fa39/volumes" Oct 30 18:27:01.700906 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:01.700872 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-50b278-fd5695c7b-mdddw"] Oct 30 18:27:01.734798 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:01.734759 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-50b278-99494fc99-wv8sj"] Oct 30 18:27:06.701991 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:06.701947 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-50b278-fd5695c7b-mdddw" podUID="6567cc2e-3472-44f3-a360-5c08b825abf8" containerName="payload-processing" containerID="cri-o://2e74e652582d54d764d2f470cfb05249d7d4801e062d954add9534ecad969a24" gracePeriod=25 Oct 30 18:27:06.735394 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:06.735334 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-50b278-99494fc99-wv8sj" podUID="1f2c8508-03f8-4615-9c58-a9e4a4c0093b" containerName="payload-pre-processing" containerID="cri-o://124bcfbf713d2ef4ae5a6bb105d060fb2d437cd7c6ff9c79dea92f68c6ba0d65" gracePeriod=25 Oct 30 18:27:07.045740 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:07.045713 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-50b278-fd5695c7b-mdddw" Oct 30 18:27:07.072467 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:07.072432 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-50b278-99494fc99-wv8sj" Oct 30 18:27:07.143402 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:07.143372 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6567cc2e-3472-44f3-a360-5c08b825abf8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6567cc2e-3472-44f3-a360-5c08b825abf8-plugins-config-volume\") pod \"6567cc2e-3472-44f3-a360-5c08b825abf8\" (UID: \"6567cc2e-3472-44f3-a360-5c08b825abf8\") " Oct 30 18:27:07.143586 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:07.143508 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6567cc2e-3472-44f3-a360-5c08b825abf8-kube-api-access-hjjpm\" (UniqueName: \"kubernetes.io/projected/6567cc2e-3472-44f3-a360-5c08b825abf8-kube-api-access-hjjpm\") pod \"6567cc2e-3472-44f3-a360-5c08b825abf8\" (UID: \"6567cc2e-3472-44f3-a360-5c08b825abf8\") " Oct 30 18:27:07.143586 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:07.143554 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f2c8508-03f8-4615-9c58-a9e4a4c0093b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f2c8508-03f8-4615-9c58-a9e4a4c0093b-plugins-config-volume\") pod \"1f2c8508-03f8-4615-9c58-a9e4a4c0093b\" (UID: \"1f2c8508-03f8-4615-9c58-a9e4a4c0093b\") " Oct 30 18:27:07.143716 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:07.143615 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f2c8508-03f8-4615-9c58-a9e4a4c0093b-kube-api-access-rzl57\" (UniqueName: \"kubernetes.io/projected/1f2c8508-03f8-4615-9c58-a9e4a4c0093b-kube-api-access-rzl57\") pod \"1f2c8508-03f8-4615-9c58-a9e4a4c0093b\" (UID: \"1f2c8508-03f8-4615-9c58-a9e4a4c0093b\") " Oct 30 18:27:07.143787 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:07.143763 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6567cc2e-3472-44f3-a360-5c08b825abf8-plugins-config-volume" pod "6567cc2e-3472-44f3-a360-5c08b825abf8" (UID: "6567cc2e-3472-44f3-a360-5c08b825abf8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:07.144286 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:07.144211 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f2c8508-03f8-4615-9c58-a9e4a4c0093b-plugins-config-volume" pod "1f2c8508-03f8-4615-9c58-a9e4a4c0093b" (UID: "1f2c8508-03f8-4615-9c58-a9e4a4c0093b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:07.145490 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:07.145463 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f2c8508-03f8-4615-9c58-a9e4a4c0093b-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:27:07.145612 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:07.145493 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6567cc2e-3472-44f3-a360-5c08b825abf8-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:27:07.146332 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:07.146308 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6567cc2e-3472-44f3-a360-5c08b825abf8-kube-api-access-hjjpm" pod "6567cc2e-3472-44f3-a360-5c08b825abf8" (UID: "6567cc2e-3472-44f3-a360-5c08b825abf8"). InnerVolumeSpecName "kube-api-access-hjjpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:07.146982 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:07.146958 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f2c8508-03f8-4615-9c58-a9e4a4c0093b-kube-api-access-rzl57" pod "1f2c8508-03f8-4615-9c58-a9e4a4c0093b" (UID: "1f2c8508-03f8-4615-9c58-a9e4a4c0093b"). InnerVolumeSpecName "kube-api-access-rzl57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:07.247371 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:07.247275 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjjpm\" (UniqueName: \"kubernetes.io/projected/6567cc2e-3472-44f3-a360-5c08b825abf8-kube-api-access-hjjpm\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:27:07.247371 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:07.247302 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzl57\" (UniqueName: \"kubernetes.io/projected/1f2c8508-03f8-4615-9c58-a9e4a4c0093b-kube-api-access-rzl57\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:27:07.817740 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:07.817699 4442 generic.go:356] "Generic (PLEG): container finished" podID="1f2c8508-03f8-4615-9c58-a9e4a4c0093b" containerID="124bcfbf713d2ef4ae5a6bb105d060fb2d437cd7c6ff9c79dea92f68c6ba0d65" exitCode=0 Oct 30 18:27:07.818289 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:07.817762 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-50b278-99494fc99-wv8sj" Oct 30 18:27:07.818289 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:07.817784 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-50b278-99494fc99-wv8sj" event={"ID":"1f2c8508-03f8-4615-9c58-a9e4a4c0093b","Type":"ContainerDied","Data":"124bcfbf713d2ef4ae5a6bb105d060fb2d437cd7c6ff9c79dea92f68c6ba0d65"} Oct 30 18:27:07.818289 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:07.817893 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-50b278-99494fc99-wv8sj" event={"ID":"1f2c8508-03f8-4615-9c58-a9e4a4c0093b","Type":"ContainerDied","Data":"014c97110c0db1645cfac001ae4b605a42c5d5c069be55b0fcf70e7b26d756df"} Oct 30 18:27:07.818289 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:07.817929 4442 scope.go:122] "RemoveContainer" containerID="124bcfbf713d2ef4ae5a6bb105d060fb2d437cd7c6ff9c79dea92f68c6ba0d65" Oct 30 18:27:07.819858 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:07.819729 4442 generic.go:356] "Generic (PLEG): container finished" podID="6567cc2e-3472-44f3-a360-5c08b825abf8" containerID="2e74e652582d54d764d2f470cfb05249d7d4801e062d954add9534ecad969a24" exitCode=0 Oct 30 18:27:07.819858 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:07.819815 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-50b278-fd5695c7b-mdddw" Oct 30 18:27:07.819858 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:07.819822 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-50b278-fd5695c7b-mdddw" event={"ID":"6567cc2e-3472-44f3-a360-5c08b825abf8","Type":"ContainerDied","Data":"2e74e652582d54d764d2f470cfb05249d7d4801e062d954add9534ecad969a24"} Oct 30 18:27:07.820026 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:07.819874 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-50b278-fd5695c7b-mdddw" event={"ID":"6567cc2e-3472-44f3-a360-5c08b825abf8","Type":"ContainerDied","Data":"a6ca581e6a773083db9b65ffa24c61100c4232e34260c295ecbf3dd68ed9d429"} Oct 30 18:27:07.831738 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:07.831720 4442 scope.go:122] "RemoveContainer" containerID="124bcfbf713d2ef4ae5a6bb105d060fb2d437cd7c6ff9c79dea92f68c6ba0d65" Oct 30 18:27:07.832246 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:27:07.832225 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"124bcfbf713d2ef4ae5a6bb105d060fb2d437cd7c6ff9c79dea92f68c6ba0d65\": container with ID starting with 124bcfbf713d2ef4ae5a6bb105d060fb2d437cd7c6ff9c79dea92f68c6ba0d65 not found: ID does not exist" containerID="124bcfbf713d2ef4ae5a6bb105d060fb2d437cd7c6ff9c79dea92f68c6ba0d65" Oct 30 18:27:07.832353 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:07.832286 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"124bcfbf713d2ef4ae5a6bb105d060fb2d437cd7c6ff9c79dea92f68c6ba0d65"} err="failed to get container status \"124bcfbf713d2ef4ae5a6bb105d060fb2d437cd7c6ff9c79dea92f68c6ba0d65\": rpc error: code = NotFound desc = could not find container \"124bcfbf713d2ef4ae5a6bb105d060fb2d437cd7c6ff9c79dea92f68c6ba0d65\": container with ID starting with 124bcfbf713d2ef4ae5a6bb105d060fb2d437cd7c6ff9c79dea92f68c6ba0d65 not found: ID does not exist" Oct 30 18:27:07.832353 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:07.832313 4442 scope.go:122] "RemoveContainer" containerID="2e74e652582d54d764d2f470cfb05249d7d4801e062d954add9534ecad969a24" Oct 30 18:27:07.846813 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:07.846792 4442 scope.go:122] "RemoveContainer" containerID="2e74e652582d54d764d2f470cfb05249d7d4801e062d954add9534ecad969a24" Oct 30 18:27:07.847208 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:27:07.847184 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e74e652582d54d764d2f470cfb05249d7d4801e062d954add9534ecad969a24\": container with ID starting with 2e74e652582d54d764d2f470cfb05249d7d4801e062d954add9534ecad969a24 not found: ID does not exist" containerID="2e74e652582d54d764d2f470cfb05249d7d4801e062d954add9534ecad969a24" Oct 30 18:27:07.847282 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:07.847222 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e74e652582d54d764d2f470cfb05249d7d4801e062d954add9534ecad969a24"} err="failed to get container status \"2e74e652582d54d764d2f470cfb05249d7d4801e062d954add9534ecad969a24\": rpc error: code = NotFound desc = could not find container \"2e74e652582d54d764d2f470cfb05249d7d4801e062d954add9534ecad969a24\": container with ID starting with 2e74e652582d54d764d2f470cfb05249d7d4801e062d954add9534ecad969a24 not found: ID does not exist" Oct 30 18:27:07.869850 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:07.869820 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-50b278-fd5695c7b-mdddw"] Oct 30 18:27:07.880986 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:07.880952 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-50b278-fd5695c7b-mdddw"] Oct 30 18:27:07.920258 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:07.920230 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-50b278-99494fc99-wv8sj"] Oct 30 18:27:07.929483 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:07.929458 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-50b278-99494fc99-wv8sj"] Oct 30 18:27:09.067699 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:09.067670 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f2c8508-03f8-4615-9c58-a9e4a4c0093b" path="/var/lib/kubelet/pods/1f2c8508-03f8-4615-9c58-a9e4a4c0093b/volumes" Oct 30 18:27:09.068124 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:09.068061 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6567cc2e-3472-44f3-a360-5c08b825abf8" path="/var/lib/kubelet/pods/6567cc2e-3472-44f3-a360-5c08b825abf8/volumes" Oct 30 18:27:11.239271 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:11.239233 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-50b278-5b75c7cb45-mmhbl"] Oct 30 18:27:11.239732 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:11.239465 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-50b278-5b75c7cb45-mmhbl" podUID="ce947c77-f4b8-4178-9834-cb8e330b9070" containerName="maas-api" containerID="cri-o://cb03064fb2da85273fa791a13b8d583042d9856e8ef43e17aca1e526eddeb37a" gracePeriod=30 Oct 30 18:27:11.600677 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:11.600648 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-50b278-5b75c7cb45-mmhbl" Oct 30 18:27:11.683178 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:11.683122 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce947c77-f4b8-4178-9834-cb8e330b9070-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce947c77-f4b8-4178-9834-cb8e330b9070-metrics-certs\") pod \"ce947c77-f4b8-4178-9834-cb8e330b9070\" (UID: \"ce947c77-f4b8-4178-9834-cb8e330b9070\") " Oct 30 18:27:11.730238 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:11.723285 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce947c77-f4b8-4178-9834-cb8e330b9070-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce947c77-f4b8-4178-9834-cb8e330b9070-openshift-service-ca\") pod \"ce947c77-f4b8-4178-9834-cb8e330b9070\" (UID: \"ce947c77-f4b8-4178-9834-cb8e330b9070\") " Oct 30 18:27:11.730238 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:11.723426 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce947c77-f4b8-4178-9834-cb8e330b9070-kube-api-access-6pj65\" (UniqueName: \"kubernetes.io/projected/ce947c77-f4b8-4178-9834-cb8e330b9070-kube-api-access-6pj65\") pod \"ce947c77-f4b8-4178-9834-cb8e330b9070\" (UID: \"ce947c77-f4b8-4178-9834-cb8e330b9070\") " Oct 30 18:27:11.730238 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:11.723467 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce947c77-f4b8-4178-9834-cb8e330b9070-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce947c77-f4b8-4178-9834-cb8e330b9070-maas-api-tls\") pod \"ce947c77-f4b8-4178-9834-cb8e330b9070\" (UID: \"ce947c77-f4b8-4178-9834-cb8e330b9070\") " Oct 30 18:27:11.743165 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:11.733334 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce947c77-f4b8-4178-9834-cb8e330b9070-openshift-service-ca" pod "ce947c77-f4b8-4178-9834-cb8e330b9070" (UID: "ce947c77-f4b8-4178-9834-cb8e330b9070"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:11.776159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:11.751904 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce947c77-f4b8-4178-9834-cb8e330b9070-openshift-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:27:11.843675 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:11.843631 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce947c77-f4b8-4178-9834-cb8e330b9070-maas-api-tls" pod "ce947c77-f4b8-4178-9834-cb8e330b9070" (UID: "ce947c77-f4b8-4178-9834-cb8e330b9070"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:27:11.843870 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:11.843680 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce947c77-f4b8-4178-9834-cb8e330b9070-kube-api-access-6pj65" pod "ce947c77-f4b8-4178-9834-cb8e330b9070" (UID: "ce947c77-f4b8-4178-9834-cb8e330b9070"). InnerVolumeSpecName "kube-api-access-6pj65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:11.845628 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:11.845594 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce947c77-f4b8-4178-9834-cb8e330b9070-metrics-certs" pod "ce947c77-f4b8-4178-9834-cb8e330b9070" (UID: "ce947c77-f4b8-4178-9834-cb8e330b9070"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:27:11.865001 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:11.864967 4442 generic.go:356] "Generic (PLEG): container finished" podID="ce947c77-f4b8-4178-9834-cb8e330b9070" containerID="cb03064fb2da85273fa791a13b8d583042d9856e8ef43e17aca1e526eddeb37a" exitCode=0 Oct 30 18:27:11.865217 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:11.865038 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-50b278-5b75c7cb45-mmhbl" event={"ID":"ce947c77-f4b8-4178-9834-cb8e330b9070","Type":"ContainerDied","Data":"cb03064fb2da85273fa791a13b8d583042d9856e8ef43e17aca1e526eddeb37a"} Oct 30 18:27:11.865217 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:11.865108 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-50b278-5b75c7cb45-mmhbl" Oct 30 18:27:11.865217 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:11.865117 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-50b278-5b75c7cb45-mmhbl" event={"ID":"ce947c77-f4b8-4178-9834-cb8e330b9070","Type":"ContainerDied","Data":"f25932dc64a70fb6ab573776dacfb1348468ba512ade00799c102e8a88f024be"} Oct 30 18:27:11.865217 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:11.865163 4442 scope.go:122] "RemoveContainer" containerID="cb03064fb2da85273fa791a13b8d583042d9856e8ef43e17aca1e526eddeb37a" Oct 30 18:27:11.879089 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:11.879068 4442 scope.go:122] "RemoveContainer" containerID="cb03064fb2da85273fa791a13b8d583042d9856e8ef43e17aca1e526eddeb37a" Oct 30 18:27:11.879594 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:27:11.879573 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb03064fb2da85273fa791a13b8d583042d9856e8ef43e17aca1e526eddeb37a\": container with ID starting with cb03064fb2da85273fa791a13b8d583042d9856e8ef43e17aca1e526eddeb37a not found: ID does not exist" containerID="cb03064fb2da85273fa791a13b8d583042d9856e8ef43e17aca1e526eddeb37a" Oct 30 18:27:11.879678 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:11.879603 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb03064fb2da85273fa791a13b8d583042d9856e8ef43e17aca1e526eddeb37a"} err="failed to get container status \"cb03064fb2da85273fa791a13b8d583042d9856e8ef43e17aca1e526eddeb37a\": rpc error: code = NotFound desc = could not find container \"cb03064fb2da85273fa791a13b8d583042d9856e8ef43e17aca1e526eddeb37a\": container with ID starting with cb03064fb2da85273fa791a13b8d583042d9856e8ef43e17aca1e526eddeb37a not found: ID does not exist" Oct 30 18:27:11.893813 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:11.893783 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pj65\" (UniqueName: \"kubernetes.io/projected/ce947c77-f4b8-4178-9834-cb8e330b9070-kube-api-access-6pj65\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:27:11.893813 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:11.893810 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce947c77-f4b8-4178-9834-cb8e330b9070-maas-api-tls\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:27:11.893813 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:11.893818 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce947c77-f4b8-4178-9834-cb8e330b9070-metrics-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:27:11.916366 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:11.916341 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-50b278-5b75c7cb45-mmhbl"] Oct 30 18:27:11.930348 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:11.930296 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-50b278-5b75c7cb45-mmhbl"] Oct 30 18:27:13.069373 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:13.069343 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce947c77-f4b8-4178-9834-cb8e330b9070" path="/var/lib/kubelet/pods/ce947c77-f4b8-4178-9834-cb8e330b9070/volumes" Oct 30 18:27:17.562266 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:27:17.562228 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:27:30.656004 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.655965 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z"] Oct 30 18:27:30.657112 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.657088 4442 cpu_manager.go:414] "containerMap: removing container" podUID="f99c368a-c8a3-49be-afd7-c68980a7fa39" containerName="maas-api" Oct 30 18:27:30.657266 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.657117 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f99c368a-c8a3-49be-afd7-c68980a7fa39" containerName="maas-api" Oct 30 18:27:30.657266 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.657152 4442 cpu_manager.go:414] "containerMap: removing container" podUID="6567cc2e-3472-44f3-a360-5c08b825abf8" containerName="payload-processing" Oct 30 18:27:30.657266 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.657161 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6567cc2e-3472-44f3-a360-5c08b825abf8" containerName="payload-processing" Oct 30 18:27:30.657266 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.657177 4442 cpu_manager.go:414] "containerMap: removing container" podUID="963a8b12-5921-4d72-9c90-c31daf3d2475" containerName="istio-proxy" Oct 30 18:27:30.657266 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.657219 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="963a8b12-5921-4d72-9c90-c31daf3d2475" containerName="istio-proxy" Oct 30 18:27:30.657266 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.657229 4442 cpu_manager.go:414] "containerMap: removing container" podUID="14c086ee-c4ba-48d2-a1b9-62cc8b09cc77" containerName="istio-proxy" Oct 30 18:27:30.657266 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.657238 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14c086ee-c4ba-48d2-a1b9-62cc8b09cc77" containerName="istio-proxy" Oct 30 18:27:30.657266 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.657250 4442 cpu_manager.go:414] "containerMap: removing container" podUID="ce947c77-f4b8-4178-9834-cb8e330b9070" containerName="maas-api" Oct 30 18:27:30.657266 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.657257 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce947c77-f4b8-4178-9834-cb8e330b9070" containerName="maas-api" Oct 30 18:27:30.657615 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.657279 4442 cpu_manager.go:414] "containerMap: removing container" podUID="1f2c8508-03f8-4615-9c58-a9e4a4c0093b" containerName="payload-pre-processing" Oct 30 18:27:30.657615 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.657287 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f2c8508-03f8-4615-9c58-a9e4a4c0093b" containerName="payload-pre-processing" Oct 30 18:27:30.657615 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.657513 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="f99c368a-c8a3-49be-afd7-c68980a7fa39" containerName="maas-api" Oct 30 18:27:30.657615 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.657527 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="6567cc2e-3472-44f3-a360-5c08b825abf8" containerName="payload-processing" Oct 30 18:27:30.657615 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.657539 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="963a8b12-5921-4d72-9c90-c31daf3d2475" containerName="istio-proxy" Oct 30 18:27:30.657615 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.657547 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="14c086ee-c4ba-48d2-a1b9-62cc8b09cc77" containerName="istio-proxy" Oct 30 18:27:30.657615 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.657555 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce947c77-f4b8-4178-9834-cb8e330b9070" containerName="maas-api" Oct 30 18:27:30.657615 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.657567 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f2c8508-03f8-4615-9c58-a9e4a4c0093b" containerName="payload-pre-processing" Oct 30 18:27:30.663755 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.663731 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" Oct 30 18:27:30.676963 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.676934 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z"] Oct 30 18:27:30.682612 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.682582 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-44379a-openshift-default-dockercfg-xg44l\"" Oct 30 18:27:30.767373 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.767331 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-workload-certs\") pod \"e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z\" (UID: \"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690\") " pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" Oct 30 18:27:30.767562 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.767470 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-credential-socket\") pod \"e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z\" (UID: \"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690\") " pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" Oct 30 18:27:30.767562 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.767510 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-workload-socket\") pod \"e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z\" (UID: \"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690\") " pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" Oct 30 18:27:30.767694 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.767570 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-istio-podinfo\") pod \"e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z\" (UID: \"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690\") " pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" Oct 30 18:27:30.767756 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.767729 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-istio-data\") pod \"e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z\" (UID: \"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690\") " pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" Oct 30 18:27:30.767813 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.767778 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slv4c\" (UniqueName: \"kubernetes.io/projected/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-kube-api-access-slv4c\") pod \"e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z\" (UID: \"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690\") " pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" Oct 30 18:27:30.767868 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.767846 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-istio-envoy\") pod \"e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z\" (UID: \"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690\") " pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" Oct 30 18:27:30.767914 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.767902 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-istio-token\") pod \"e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z\" (UID: \"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690\") " pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" Oct 30 18:27:30.767960 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.767934 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-istiod-ca-cert\") pod \"e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z\" (UID: \"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690\") " pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" Oct 30 18:27:30.872116 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.872077 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-istio-podinfo\") pod \"e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z\" (UID: \"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690\") " pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" Oct 30 18:27:30.873018 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.872992 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-istio-data\") pod \"e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z\" (UID: \"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690\") " pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" Oct 30 18:27:30.873234 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.873215 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slv4c\" (UniqueName: \"kubernetes.io/projected/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-kube-api-access-slv4c\") pod \"e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z\" (UID: \"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690\") " pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" Oct 30 18:27:30.873369 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.873354 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-istio-envoy\") pod \"e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z\" (UID: \"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690\") " pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" Oct 30 18:27:30.873514 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.873499 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-istio-token\") pod \"e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z\" (UID: \"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690\") " pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" Oct 30 18:27:30.873628 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.873614 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-istiod-ca-cert\") pod \"e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z\" (UID: \"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690\") " pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" Oct 30 18:27:30.873775 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.873762 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-workload-certs\") pod \"e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z\" (UID: \"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690\") " pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" Oct 30 18:27:30.873954 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.873938 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-credential-socket\") pod \"e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z\" (UID: \"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690\") " pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" Oct 30 18:27:30.874092 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.874074 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-workload-socket\") pod \"e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z\" (UID: \"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690\") " pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" Oct 30 18:27:30.878160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.874738 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-workload-socket\") pod \"e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z\" (UID: \"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690\") " pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" Oct 30 18:27:30.886801 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.878800 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-istio-podinfo\") pod \"e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z\" (UID: \"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690\") " pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" Oct 30 18:27:30.886964 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.878917 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-istio-data\") pod \"e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z\" (UID: \"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690\") " pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" Oct 30 18:27:30.887053 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.879053 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-workload-certs\") pod \"e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z\" (UID: \"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690\") " pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" Oct 30 18:27:30.887152 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.879778 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-credential-socket\") pod \"e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z\" (UID: \"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690\") " pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" Oct 30 18:27:30.887247 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.885706 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-istio-envoy\") pod \"e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z\" (UID: \"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690\") " pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" Oct 30 18:27:30.887321 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.886733 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-istiod-ca-cert\") pod \"e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z\" (UID: \"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690\") " pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" Oct 30 18:27:30.914204 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.913863 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slv4c\" (UniqueName: \"kubernetes.io/projected/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-kube-api-access-slv4c\") pod \"e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z\" (UID: \"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690\") " pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" Oct 30 18:27:30.918157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.917047 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-istio-token\") pod \"e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z\" (UID: \"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690\") " pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" Oct 30 18:27:30.982158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:30.981634 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" Oct 30 18:27:31.360056 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:27:31.360026 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43c061bf_f41c_4bb8_93ca_fe9a3ee5e690.slice/crio-ca4ba257793da66045da454b9256e8e381fbc64c40a6084f0e890c5f0b1b7e56 WatchSource:0}: Error finding container ca4ba257793da66045da454b9256e8e381fbc64c40a6084f0e890c5f0b1b7e56: Status 404 returned error can't find the container with id ca4ba257793da66045da454b9256e8e381fbc64c40a6084f0e890c5f0b1b7e56 Oct 30 18:27:31.361314 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:31.361286 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z"] Oct 30 18:27:31.362938 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:31.362902 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:27:31.363022 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:31.362984 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:27:31.363059 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:31.363027 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:27:32.114062 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:32.114026 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" event={"ID":"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690","Type":"ContainerStarted","Data":"2a05cfd6b5ffdfdacede4a79d5df7c56a2863053ddba693efa47ea469237e376"} Oct 30 18:27:32.114485 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:32.114090 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" event={"ID":"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690","Type":"ContainerStarted","Data":"ca4ba257793da66045da454b9256e8e381fbc64c40a6084f0e890c5f0b1b7e56"} Oct 30 18:27:32.167988 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:32.167931 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" podStartSLOduration=2.167916938 podStartE2EDuration="2.167916938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:27:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:27:32.164610996 +0000 UTC m=+1749.254673335" watchObservedRunningTime="2026-10-30 18:27:32.167916938 +0000 UTC m=+1749.257979277" Oct 30 18:27:32.983252 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:32.982783 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" Oct 30 18:27:33.042212 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:33.037764 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" Oct 30 18:27:33.121150 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:33.121110 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" Oct 30 18:27:33.122198 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:33.122180 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" Oct 30 18:27:40.395731 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:40.395688 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a4f3f0-594b877768-ctrsw"] Oct 30 18:27:40.472240 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:40.472198 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a4f3f0-7c77897bf9-7mxbc"] Oct 30 18:27:45.396832 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:45.396786 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-a4f3f0-594b877768-ctrsw" podUID="2816cb76-b775-4ee2-8fe1-063879e21902" containerName="payload-processing" containerID="cri-o://22402fadba1d91990f046a6b8fbb18e552d656be12f061039f60cf471e7fcbcf" gracePeriod=25 Oct 30 18:27:45.474430 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:45.474370 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a4f3f0-7c77897bf9-7mxbc" podUID="f613d06d-c822-4949-9f01-73d99044a3a5" containerName="payload-pre-processing" containerID="cri-o://d8e2b9b14300e5caf3c7933bd357e4ddbbe4ffcc108aeeaa230d1cc74eab7b34" gracePeriod=25 Oct 30 18:27:47.562820 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:27:47.562778 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:27:49.888750 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:49.888712 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-afc614-78f958bbd8-dgrmm"] Oct 30 18:27:49.919669 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:49.919623 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-afc614-c4d5cff79-8rgjs"] Oct 30 18:27:51.243428 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:51.243393 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a4f3f0-7c77897bf9-7mxbc" podUID="f613d06d-c822-4949-9f01-73d99044a3a5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:27:51.243938 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:51.243493 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-a4f3f0-594b877768-ctrsw" podUID="2816cb76-b775-4ee2-8fe1-063879e21902" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:27:51.721365 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:51.721331 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-afc614-65d5c9f65f-mzfww"] Oct 30 18:27:51.721606 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:51.721582 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-afc614-65d5c9f65f-mzfww" podUID="bc0f6b52-69e6-422e-bc46-55ee9f7c8353" containerName="maas-api" containerID="cri-o://889114b05ea042f49f0f75850c0d2944188f0c56ebd90345e3b37a2017fb04ff" gracePeriod=30 Oct 30 18:27:52.073443 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:52.061450 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-afc614-65d5c9f65f-mzfww" Oct 30 18:27:52.217996 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:52.217950 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc0f6b52-69e6-422e-bc46-55ee9f7c8353-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc0f6b52-69e6-422e-bc46-55ee9f7c8353-metrics-certs\") pod \"bc0f6b52-69e6-422e-bc46-55ee9f7c8353\" (UID: \"bc0f6b52-69e6-422e-bc46-55ee9f7c8353\") " Oct 30 18:27:52.218217 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:52.218024 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc0f6b52-69e6-422e-bc46-55ee9f7c8353-kube-api-access-p6kmr\" (UniqueName: \"kubernetes.io/projected/bc0f6b52-69e6-422e-bc46-55ee9f7c8353-kube-api-access-p6kmr\") pod \"bc0f6b52-69e6-422e-bc46-55ee9f7c8353\" (UID: \"bc0f6b52-69e6-422e-bc46-55ee9f7c8353\") " Oct 30 18:27:52.218217 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:52.218073 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc0f6b52-69e6-422e-bc46-55ee9f7c8353-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc0f6b52-69e6-422e-bc46-55ee9f7c8353-maas-api-tls\") pod \"bc0f6b52-69e6-422e-bc46-55ee9f7c8353\" (UID: \"bc0f6b52-69e6-422e-bc46-55ee9f7c8353\") " Oct 30 18:27:52.218217 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:52.218112 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc0f6b52-69e6-422e-bc46-55ee9f7c8353-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc0f6b52-69e6-422e-bc46-55ee9f7c8353-openshift-service-ca\") pod \"bc0f6b52-69e6-422e-bc46-55ee9f7c8353\" (UID: \"bc0f6b52-69e6-422e-bc46-55ee9f7c8353\") " Oct 30 18:27:52.218690 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:52.218655 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc0f6b52-69e6-422e-bc46-55ee9f7c8353-openshift-service-ca" pod "bc0f6b52-69e6-422e-bc46-55ee9f7c8353" (UID: "bc0f6b52-69e6-422e-bc46-55ee9f7c8353"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:52.221937 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:52.221891 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc0f6b52-69e6-422e-bc46-55ee9f7c8353-maas-api-tls" pod "bc0f6b52-69e6-422e-bc46-55ee9f7c8353" (UID: "bc0f6b52-69e6-422e-bc46-55ee9f7c8353"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:27:52.222053 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:52.221997 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc0f6b52-69e6-422e-bc46-55ee9f7c8353-kube-api-access-p6kmr" pod "bc0f6b52-69e6-422e-bc46-55ee9f7c8353" (UID: "bc0f6b52-69e6-422e-bc46-55ee9f7c8353"). InnerVolumeSpecName "kube-api-access-p6kmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:52.222185 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:52.222076 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc0f6b52-69e6-422e-bc46-55ee9f7c8353-metrics-certs" pod "bc0f6b52-69e6-422e-bc46-55ee9f7c8353" (UID: "bc0f6b52-69e6-422e-bc46-55ee9f7c8353"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:27:52.327255 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:52.323082 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc0f6b52-69e6-422e-bc46-55ee9f7c8353-metrics-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:27:52.327255 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:52.323121 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6kmr\" (UniqueName: \"kubernetes.io/projected/bc0f6b52-69e6-422e-bc46-55ee9f7c8353-kube-api-access-p6kmr\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:27:52.327255 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:52.323155 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc0f6b52-69e6-422e-bc46-55ee9f7c8353-maas-api-tls\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:27:52.327255 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:52.323168 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc0f6b52-69e6-422e-bc46-55ee9f7c8353-openshift-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:27:52.403514 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:52.403471 4442 generic.go:356] "Generic (PLEG): container finished" podID="bc0f6b52-69e6-422e-bc46-55ee9f7c8353" containerID="889114b05ea042f49f0f75850c0d2944188f0c56ebd90345e3b37a2017fb04ff" exitCode=0 Oct 30 18:27:52.403718 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:52.403526 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-afc614-65d5c9f65f-mzfww" Oct 30 18:27:52.403718 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:52.403553 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-afc614-65d5c9f65f-mzfww" event={"ID":"bc0f6b52-69e6-422e-bc46-55ee9f7c8353","Type":"ContainerDied","Data":"889114b05ea042f49f0f75850c0d2944188f0c56ebd90345e3b37a2017fb04ff"} Oct 30 18:27:52.403718 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:52.403620 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-afc614-65d5c9f65f-mzfww" event={"ID":"bc0f6b52-69e6-422e-bc46-55ee9f7c8353","Type":"ContainerDied","Data":"6f66dd97620ebe16d985d2d31dd5a3d8e4929a03dc7a00f37733001fafea0154"} Oct 30 18:27:52.403718 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:52.403651 4442 scope.go:122] "RemoveContainer" containerID="889114b05ea042f49f0f75850c0d2944188f0c56ebd90345e3b37a2017fb04ff" Oct 30 18:27:52.415336 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:52.415312 4442 scope.go:122] "RemoveContainer" containerID="889114b05ea042f49f0f75850c0d2944188f0c56ebd90345e3b37a2017fb04ff" Oct 30 18:27:52.415651 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:27:52.415625 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"889114b05ea042f49f0f75850c0d2944188f0c56ebd90345e3b37a2017fb04ff\": container with ID starting with 889114b05ea042f49f0f75850c0d2944188f0c56ebd90345e3b37a2017fb04ff not found: ID does not exist" containerID="889114b05ea042f49f0f75850c0d2944188f0c56ebd90345e3b37a2017fb04ff" Oct 30 18:27:52.415754 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:52.415657 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"889114b05ea042f49f0f75850c0d2944188f0c56ebd90345e3b37a2017fb04ff"} err="failed to get container status \"889114b05ea042f49f0f75850c0d2944188f0c56ebd90345e3b37a2017fb04ff\": rpc error: code = NotFound desc = could not find container \"889114b05ea042f49f0f75850c0d2944188f0c56ebd90345e3b37a2017fb04ff\": container with ID starting with 889114b05ea042f49f0f75850c0d2944188f0c56ebd90345e3b37a2017fb04ff not found: ID does not exist" Oct 30 18:27:52.456076 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:52.456037 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-afc614-65d5c9f65f-mzfww"] Oct 30 18:27:52.460853 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:52.460828 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-afc614-65d5c9f65f-mzfww"] Oct 30 18:27:53.065918 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:53.065875 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc0f6b52-69e6-422e-bc46-55ee9f7c8353" path="/var/lib/kubelet/pods/bc0f6b52-69e6-422e-bc46-55ee9f7c8353/volumes" Oct 30 18:27:54.890040 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:54.889999 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-afc614-78f958bbd8-dgrmm" podUID="7c46dd87-b6ad-4929-bf8c-ab10fd137e5b" containerName="payload-processing" containerID="cri-o://e3af9e7af3c5ebfbbd29bef74b71c81d7610cd3257cebc8c41cad10162e1ed05" gracePeriod=25 Oct 30 18:27:54.920453 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:54.920419 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-afc614-c4d5cff79-8rgjs" podUID="c2ee7fe9-5c24-422e-9140-5d1ec2564efc" containerName="payload-pre-processing" containerID="cri-o://7ff446a052ce1af021763a3095f54833fbd46cdadafe22441579c46a6d199ed0" gracePeriod=25 Oct 30 18:27:55.242787 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:55.242759 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-afc614-78f958bbd8-dgrmm" Oct 30 18:27:55.259633 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:55.259609 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-afc614-c4d5cff79-8rgjs" Oct 30 18:27:55.294870 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:55.294838 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2ee7fe9-5c24-422e-9140-5d1ec2564efc-kube-api-access-klgj9\" (UniqueName: \"kubernetes.io/projected/c2ee7fe9-5c24-422e-9140-5d1ec2564efc-kube-api-access-klgj9\") pod \"c2ee7fe9-5c24-422e-9140-5d1ec2564efc\" (UID: \"c2ee7fe9-5c24-422e-9140-5d1ec2564efc\") " Oct 30 18:27:55.295054 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:55.294948 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c46dd87-b6ad-4929-bf8c-ab10fd137e5b-kube-api-access-w25xd\" (UniqueName: \"kubernetes.io/projected/7c46dd87-b6ad-4929-bf8c-ab10fd137e5b-kube-api-access-w25xd\") pod \"7c46dd87-b6ad-4929-bf8c-ab10fd137e5b\" (UID: \"7c46dd87-b6ad-4929-bf8c-ab10fd137e5b\") " Oct 30 18:27:55.295054 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:55.294985 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2ee7fe9-5c24-422e-9140-5d1ec2564efc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2ee7fe9-5c24-422e-9140-5d1ec2564efc-plugins-config-volume\") pod \"c2ee7fe9-5c24-422e-9140-5d1ec2564efc\" (UID: \"c2ee7fe9-5c24-422e-9140-5d1ec2564efc\") " Oct 30 18:27:55.295054 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:55.295037 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c46dd87-b6ad-4929-bf8c-ab10fd137e5b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c46dd87-b6ad-4929-bf8c-ab10fd137e5b-plugins-config-volume\") pod \"7c46dd87-b6ad-4929-bf8c-ab10fd137e5b\" (UID: \"7c46dd87-b6ad-4929-bf8c-ab10fd137e5b\") " Oct 30 18:27:55.296188 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:55.296112 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2ee7fe9-5c24-422e-9140-5d1ec2564efc-plugins-config-volume" pod "c2ee7fe9-5c24-422e-9140-5d1ec2564efc" (UID: "c2ee7fe9-5c24-422e-9140-5d1ec2564efc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:55.297545 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:55.297517 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c46dd87-b6ad-4929-bf8c-ab10fd137e5b-plugins-config-volume" pod "7c46dd87-b6ad-4929-bf8c-ab10fd137e5b" (UID: "7c46dd87-b6ad-4929-bf8c-ab10fd137e5b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:55.298636 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:55.298613 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2ee7fe9-5c24-422e-9140-5d1ec2564efc-kube-api-access-klgj9" pod "c2ee7fe9-5c24-422e-9140-5d1ec2564efc" (UID: "c2ee7fe9-5c24-422e-9140-5d1ec2564efc"). InnerVolumeSpecName "kube-api-access-klgj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:55.299537 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:55.299516 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c46dd87-b6ad-4929-bf8c-ab10fd137e5b-kube-api-access-w25xd" pod "7c46dd87-b6ad-4929-bf8c-ab10fd137e5b" (UID: "7c46dd87-b6ad-4929-bf8c-ab10fd137e5b"). InnerVolumeSpecName "kube-api-access-w25xd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:55.396828 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:55.396781 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klgj9\" (UniqueName: \"kubernetes.io/projected/c2ee7fe9-5c24-422e-9140-5d1ec2564efc-kube-api-access-klgj9\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:27:55.396828 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:55.396819 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w25xd\" (UniqueName: \"kubernetes.io/projected/7c46dd87-b6ad-4929-bf8c-ab10fd137e5b-kube-api-access-w25xd\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:27:55.396828 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:55.396831 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2ee7fe9-5c24-422e-9140-5d1ec2564efc-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:27:55.397088 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:55.396862 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c46dd87-b6ad-4929-bf8c-ab10fd137e5b-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:27:55.428467 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:55.428371 4442 generic.go:356] "Generic (PLEG): container finished" podID="c2ee7fe9-5c24-422e-9140-5d1ec2564efc" containerID="7ff446a052ce1af021763a3095f54833fbd46cdadafe22441579c46a6d199ed0" exitCode=0 Oct 30 18:27:55.428467 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:55.428432 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-afc614-c4d5cff79-8rgjs" Oct 30 18:27:55.428467 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:55.428451 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-afc614-c4d5cff79-8rgjs" event={"ID":"c2ee7fe9-5c24-422e-9140-5d1ec2564efc","Type":"ContainerDied","Data":"7ff446a052ce1af021763a3095f54833fbd46cdadafe22441579c46a6d199ed0"} Oct 30 18:27:55.428710 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:55.428503 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-afc614-c4d5cff79-8rgjs" event={"ID":"c2ee7fe9-5c24-422e-9140-5d1ec2564efc","Type":"ContainerDied","Data":"b3c292bc0a73a420c2abdf50981bace08925a0c31c4f56fd7829d05453835225"} Oct 30 18:27:55.428710 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:55.428531 4442 scope.go:122] "RemoveContainer" containerID="7ff446a052ce1af021763a3095f54833fbd46cdadafe22441579c46a6d199ed0" Oct 30 18:27:55.430092 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:55.430060 4442 generic.go:356] "Generic (PLEG): container finished" podID="7c46dd87-b6ad-4929-bf8c-ab10fd137e5b" containerID="e3af9e7af3c5ebfbbd29bef74b71c81d7610cd3257cebc8c41cad10162e1ed05" exitCode=0 Oct 30 18:27:55.430236 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:55.430091 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-afc614-78f958bbd8-dgrmm" event={"ID":"7c46dd87-b6ad-4929-bf8c-ab10fd137e5b","Type":"ContainerDied","Data":"e3af9e7af3c5ebfbbd29bef74b71c81d7610cd3257cebc8c41cad10162e1ed05"} Oct 30 18:27:55.430296 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:55.430110 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-afc614-78f958bbd8-dgrmm" Oct 30 18:27:55.430351 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:55.430331 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-afc614-78f958bbd8-dgrmm" event={"ID":"7c46dd87-b6ad-4929-bf8c-ab10fd137e5b","Type":"ContainerDied","Data":"ab2ea51dd84ec5b037f30b6feaa9dede2757a884c5d291d93b28ba85426f1182"} Oct 30 18:27:55.440095 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:55.440073 4442 scope.go:122] "RemoveContainer" containerID="7ff446a052ce1af021763a3095f54833fbd46cdadafe22441579c46a6d199ed0" Oct 30 18:27:55.440509 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:27:55.440486 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ff446a052ce1af021763a3095f54833fbd46cdadafe22441579c46a6d199ed0\": container with ID starting with 7ff446a052ce1af021763a3095f54833fbd46cdadafe22441579c46a6d199ed0 not found: ID does not exist" containerID="7ff446a052ce1af021763a3095f54833fbd46cdadafe22441579c46a6d199ed0" Oct 30 18:27:55.440607 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:55.440525 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ff446a052ce1af021763a3095f54833fbd46cdadafe22441579c46a6d199ed0"} err="failed to get container status \"7ff446a052ce1af021763a3095f54833fbd46cdadafe22441579c46a6d199ed0\": rpc error: code = NotFound desc = could not find container \"7ff446a052ce1af021763a3095f54833fbd46cdadafe22441579c46a6d199ed0\": container with ID starting with 7ff446a052ce1af021763a3095f54833fbd46cdadafe22441579c46a6d199ed0 not found: ID does not exist" Oct 30 18:27:55.440607 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:55.440549 4442 scope.go:122] "RemoveContainer" containerID="e3af9e7af3c5ebfbbd29bef74b71c81d7610cd3257cebc8c41cad10162e1ed05" Oct 30 18:27:55.452219 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:55.452191 4442 scope.go:122] "RemoveContainer" containerID="e3af9e7af3c5ebfbbd29bef74b71c81d7610cd3257cebc8c41cad10162e1ed05" Oct 30 18:27:55.452559 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:27:55.452537 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3af9e7af3c5ebfbbd29bef74b71c81d7610cd3257cebc8c41cad10162e1ed05\": container with ID starting with e3af9e7af3c5ebfbbd29bef74b71c81d7610cd3257cebc8c41cad10162e1ed05 not found: ID does not exist" containerID="e3af9e7af3c5ebfbbd29bef74b71c81d7610cd3257cebc8c41cad10162e1ed05" Oct 30 18:27:55.452628 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:55.452572 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3af9e7af3c5ebfbbd29bef74b71c81d7610cd3257cebc8c41cad10162e1ed05"} err="failed to get container status \"e3af9e7af3c5ebfbbd29bef74b71c81d7610cd3257cebc8c41cad10162e1ed05\": rpc error: code = NotFound desc = could not find container \"e3af9e7af3c5ebfbbd29bef74b71c81d7610cd3257cebc8c41cad10162e1ed05\": container with ID starting with e3af9e7af3c5ebfbbd29bef74b71c81d7610cd3257cebc8c41cad10162e1ed05 not found: ID does not exist" Oct 30 18:27:55.480468 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:55.480356 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-afc614-78f958bbd8-dgrmm"] Oct 30 18:27:55.485198 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:55.485172 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-afc614-78f958bbd8-dgrmm"] Oct 30 18:27:55.514522 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:55.514492 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-afc614-c4d5cff79-8rgjs"] Oct 30 18:27:55.521995 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:55.521968 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-afc614-c4d5cff79-8rgjs"] Oct 30 18:27:57.064786 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:57.064747 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c46dd87-b6ad-4929-bf8c-ab10fd137e5b" path="/var/lib/kubelet/pods/7c46dd87-b6ad-4929-bf8c-ab10fd137e5b/volumes" Oct 30 18:27:57.065451 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:57.065428 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2ee7fe9-5c24-422e-9140-5d1ec2564efc" path="/var/lib/kubelet/pods/c2ee7fe9-5c24-422e-9140-5d1ec2564efc/volumes" Oct 30 18:27:59.769769 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:59.769730 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw"] Oct 30 18:27:59.770370 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:27:59.770037 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" podUID="29c8e32e-e676-4b79-9643-25c88f1a6f1a" containerName="istio-proxy" containerID="cri-o://c3a8fcecf38d66506e58a5204238f86d39b9e3ee766042afd1b206c1bf7aea89" gracePeriod=30 Oct 30 18:28:01.243669 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:01.243629 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-a4f3f0-594b877768-ctrsw" podUID="2816cb76-b775-4ee2-8fe1-063879e21902" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:28:01.244199 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:01.243761 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a4f3f0-7c77897bf9-7mxbc" podUID="f613d06d-c822-4949-9f01-73d99044a3a5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:28:01.412853 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:01.412816 4442 patch_prober.go:29] interesting pod/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:28:01.413027 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:01.412894 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" podUID="29c8e32e-e676-4b79-9643-25c88f1a6f1a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:28:05.051909 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.051843 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" Oct 30 18:28:05.171087 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.171051 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29c8e32e-e676-4b79-9643-25c88f1a6f1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29c8e32e-e676-4b79-9643-25c88f1a6f1a-workload-certs\") pod \"29c8e32e-e676-4b79-9643-25c88f1a6f1a\" (UID: \"29c8e32e-e676-4b79-9643-25c88f1a6f1a\") " Oct 30 18:28:05.171326 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.171169 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29c8e32e-e676-4b79-9643-25c88f1a6f1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c8e32e-e676-4b79-9643-25c88f1a6f1a-credential-socket\") pod \"29c8e32e-e676-4b79-9643-25c88f1a6f1a\" (UID: \"29c8e32e-e676-4b79-9643-25c88f1a6f1a\") " Oct 30 18:28:05.171326 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.171198 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29c8e32e-e676-4b79-9643-25c88f1a6f1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29c8e32e-e676-4b79-9643-25c88f1a6f1a-istio-envoy\") pod \"29c8e32e-e676-4b79-9643-25c88f1a6f1a\" (UID: \"29c8e32e-e676-4b79-9643-25c88f1a6f1a\") " Oct 30 18:28:05.171326 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.171301 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29c8e32e-e676-4b79-9643-25c88f1a6f1a-istio-token\" (UniqueName: \"kubernetes.io/projected/29c8e32e-e676-4b79-9643-25c88f1a6f1a-istio-token\") pod \"29c8e32e-e676-4b79-9643-25c88f1a6f1a\" (UID: \"29c8e32e-e676-4b79-9643-25c88f1a6f1a\") " Oct 30 18:28:05.171326 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.171314 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29c8e32e-e676-4b79-9643-25c88f1a6f1a-workload-certs" pod "29c8e32e-e676-4b79-9643-25c88f1a6f1a" (UID: "29c8e32e-e676-4b79-9643-25c88f1a6f1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:05.171561 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.171350 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29c8e32e-e676-4b79-9643-25c88f1a6f1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29c8e32e-e676-4b79-9643-25c88f1a6f1a-istio-podinfo\") pod \"29c8e32e-e676-4b79-9643-25c88f1a6f1a\" (UID: \"29c8e32e-e676-4b79-9643-25c88f1a6f1a\") " Oct 30 18:28:05.171561 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.171442 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29c8e32e-e676-4b79-9643-25c88f1a6f1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29c8e32e-e676-4b79-9643-25c88f1a6f1a-istio-data\") pod \"29c8e32e-e676-4b79-9643-25c88f1a6f1a\" (UID: \"29c8e32e-e676-4b79-9643-25c88f1a6f1a\") " Oct 30 18:28:05.171561 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.171489 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29c8e32e-e676-4b79-9643-25c88f1a6f1a-kube-api-access-n9s5m\" (UniqueName: \"kubernetes.io/projected/29c8e32e-e676-4b79-9643-25c88f1a6f1a-kube-api-access-n9s5m\") pod \"29c8e32e-e676-4b79-9643-25c88f1a6f1a\" (UID: \"29c8e32e-e676-4b79-9643-25c88f1a6f1a\") " Oct 30 18:28:05.171561 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.171496 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29c8e32e-e676-4b79-9643-25c88f1a6f1a-credential-socket" pod "29c8e32e-e676-4b79-9643-25c88f1a6f1a" (UID: "29c8e32e-e676-4b79-9643-25c88f1a6f1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:05.171561 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.171532 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29c8e32e-e676-4b79-9643-25c88f1a6f1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c8e32e-e676-4b79-9643-25c88f1a6f1a-workload-socket\") pod \"29c8e32e-e676-4b79-9643-25c88f1a6f1a\" (UID: \"29c8e32e-e676-4b79-9643-25c88f1a6f1a\") " Oct 30 18:28:05.171837 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.171564 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29c8e32e-e676-4b79-9643-25c88f1a6f1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29c8e32e-e676-4b79-9643-25c88f1a6f1a-istiod-ca-cert\") pod \"29c8e32e-e676-4b79-9643-25c88f1a6f1a\" (UID: \"29c8e32e-e676-4b79-9643-25c88f1a6f1a\") " Oct 30 18:28:05.171893 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.171872 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29c8e32e-e676-4b79-9643-25c88f1a6f1a-workload-socket" pod "29c8e32e-e676-4b79-9643-25c88f1a6f1a" (UID: "29c8e32e-e676-4b79-9643-25c88f1a6f1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:05.171998 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.171975 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29c8e32e-e676-4b79-9643-25c88f1a6f1a-istiod-ca-cert" pod "29c8e32e-e676-4b79-9643-25c88f1a6f1a" (UID: "29c8e32e-e676-4b79-9643-25c88f1a6f1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:05.172377 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.172354 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c8e32e-e676-4b79-9643-25c88f1a6f1a-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:28:05.172504 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.172382 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29c8e32e-e676-4b79-9643-25c88f1a6f1a-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:28:05.172504 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.172415 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29c8e32e-e676-4b79-9643-25c88f1a6f1a-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:28:05.172504 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.172429 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c8e32e-e676-4b79-9643-25c88f1a6f1a-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:28:05.172668 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.172626 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29c8e32e-e676-4b79-9643-25c88f1a6f1a-istio-data" pod "29c8e32e-e676-4b79-9643-25c88f1a6f1a" (UID: "29c8e32e-e676-4b79-9643-25c88f1a6f1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:05.174170 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.174115 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29c8e32e-e676-4b79-9643-25c88f1a6f1a-istio-podinfo" pod "29c8e32e-e676-4b79-9643-25c88f1a6f1a" (UID: "29c8e32e-e676-4b79-9643-25c88f1a6f1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:28:05.174694 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.174671 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29c8e32e-e676-4b79-9643-25c88f1a6f1a-kube-api-access-n9s5m" pod "29c8e32e-e676-4b79-9643-25c88f1a6f1a" (UID: "29c8e32e-e676-4b79-9643-25c88f1a6f1a"). InnerVolumeSpecName "kube-api-access-n9s5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:05.175038 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.175009 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29c8e32e-e676-4b79-9643-25c88f1a6f1a-istio-token" pod "29c8e32e-e676-4b79-9643-25c88f1a6f1a" (UID: "29c8e32e-e676-4b79-9643-25c88f1a6f1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:05.175104 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.175085 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29c8e32e-e676-4b79-9643-25c88f1a6f1a-istio-envoy" pod "29c8e32e-e676-4b79-9643-25c88f1a6f1a" (UID: "29c8e32e-e676-4b79-9643-25c88f1a6f1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:05.274182 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.274125 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29c8e32e-e676-4b79-9643-25c88f1a6f1a-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:28:05.274182 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.274174 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29c8e32e-e676-4b79-9643-25c88f1a6f1a-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:28:05.274182 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.274182 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29c8e32e-e676-4b79-9643-25c88f1a6f1a-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:28:05.274182 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.274191 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9s5m\" (UniqueName: \"kubernetes.io/projected/29c8e32e-e676-4b79-9643-25c88f1a6f1a-kube-api-access-n9s5m\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:28:05.274465 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.274201 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29c8e32e-e676-4b79-9643-25c88f1a6f1a-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:28:05.513507 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.513474 4442 generic.go:356] "Generic (PLEG): container finished" podID="29c8e32e-e676-4b79-9643-25c88f1a6f1a" containerID="c3a8fcecf38d66506e58a5204238f86d39b9e3ee766042afd1b206c1bf7aea89" exitCode=0 Oct 30 18:28:05.513684 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.513524 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" Oct 30 18:28:05.513684 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.513561 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" event={"ID":"29c8e32e-e676-4b79-9643-25c88f1a6f1a","Type":"ContainerDied","Data":"c3a8fcecf38d66506e58a5204238f86d39b9e3ee766042afd1b206c1bf7aea89"} Oct 30 18:28:05.513684 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.513616 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw" event={"ID":"29c8e32e-e676-4b79-9643-25c88f1a6f1a","Type":"ContainerDied","Data":"bab7deec04d160b291a4ffd3e0d60525dc03e2a2c8181fd64ff7df16740424fc"} Oct 30 18:28:05.513684 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.513641 4442 scope.go:122] "RemoveContainer" containerID="c3a8fcecf38d66506e58a5204238f86d39b9e3ee766042afd1b206c1bf7aea89" Oct 30 18:28:05.515542 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.515524 4442 generic.go:356] "Generic (PLEG): container finished" podID="2816cb76-b775-4ee2-8fe1-063879e21902" containerID="22402fadba1d91990f046a6b8fbb18e552d656be12f061039f60cf471e7fcbcf" exitCode=0 Oct 30 18:28:05.515649 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.515600 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a4f3f0-594b877768-ctrsw" event={"ID":"2816cb76-b775-4ee2-8fe1-063879e21902","Type":"ContainerDied","Data":"22402fadba1d91990f046a6b8fbb18e552d656be12f061039f60cf471e7fcbcf"} Oct 30 18:28:05.578250 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.578210 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw"] Oct 30 18:28:05.582907 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.582878 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-afc614-openshift-default-55bd7c8977-vd6kw"] Oct 30 18:28:05.590338 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.590312 4442 scope.go:122] "RemoveContainer" containerID="190401a7ae3d22201b3afd9adfbff94e809554cf2cd17a81774e38683963bb1b" Oct 30 18:28:05.603879 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.603757 4442 scope.go:122] "RemoveContainer" containerID="c3a8fcecf38d66506e58a5204238f86d39b9e3ee766042afd1b206c1bf7aea89" Oct 30 18:28:05.604264 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:28:05.604233 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3a8fcecf38d66506e58a5204238f86d39b9e3ee766042afd1b206c1bf7aea89\": container with ID starting with c3a8fcecf38d66506e58a5204238f86d39b9e3ee766042afd1b206c1bf7aea89 not found: ID does not exist" containerID="c3a8fcecf38d66506e58a5204238f86d39b9e3ee766042afd1b206c1bf7aea89" Oct 30 18:28:05.604386 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.604276 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3a8fcecf38d66506e58a5204238f86d39b9e3ee766042afd1b206c1bf7aea89"} err="failed to get container status \"c3a8fcecf38d66506e58a5204238f86d39b9e3ee766042afd1b206c1bf7aea89\": rpc error: code = NotFound desc = could not find container \"c3a8fcecf38d66506e58a5204238f86d39b9e3ee766042afd1b206c1bf7aea89\": container with ID starting with c3a8fcecf38d66506e58a5204238f86d39b9e3ee766042afd1b206c1bf7aea89 not found: ID does not exist" Oct 30 18:28:05.604386 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.604299 4442 scope.go:122] "RemoveContainer" containerID="190401a7ae3d22201b3afd9adfbff94e809554cf2cd17a81774e38683963bb1b" Oct 30 18:28:05.604639 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:28:05.604602 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"190401a7ae3d22201b3afd9adfbff94e809554cf2cd17a81774e38683963bb1b\": container with ID starting with 190401a7ae3d22201b3afd9adfbff94e809554cf2cd17a81774e38683963bb1b not found: ID does not exist" containerID="190401a7ae3d22201b3afd9adfbff94e809554cf2cd17a81774e38683963bb1b" Oct 30 18:28:05.604740 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.604634 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"190401a7ae3d22201b3afd9adfbff94e809554cf2cd17a81774e38683963bb1b"} err="failed to get container status \"190401a7ae3d22201b3afd9adfbff94e809554cf2cd17a81774e38683963bb1b\": rpc error: code = NotFound desc = could not find container \"190401a7ae3d22201b3afd9adfbff94e809554cf2cd17a81774e38683963bb1b\": container with ID starting with 190401a7ae3d22201b3afd9adfbff94e809554cf2cd17a81774e38683963bb1b not found: ID does not exist" Oct 30 18:28:05.757338 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.757230 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a4f3f0-594b877768-ctrsw" Oct 30 18:28:05.840288 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.840263 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a4f3f0-7c77897bf9-7mxbc" Oct 30 18:28:05.883386 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.883349 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2816cb76-b775-4ee2-8fe1-063879e21902-kube-api-access-fdkxs\" (UniqueName: \"kubernetes.io/projected/2816cb76-b775-4ee2-8fe1-063879e21902-kube-api-access-fdkxs\") pod \"2816cb76-b775-4ee2-8fe1-063879e21902\" (UID: \"2816cb76-b775-4ee2-8fe1-063879e21902\") " Oct 30 18:28:05.883576 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.883460 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2816cb76-b775-4ee2-8fe1-063879e21902-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2816cb76-b775-4ee2-8fe1-063879e21902-plugins-config-volume\") pod \"2816cb76-b775-4ee2-8fe1-063879e21902\" (UID: \"2816cb76-b775-4ee2-8fe1-063879e21902\") " Oct 30 18:28:05.884044 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.884012 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2816cb76-b775-4ee2-8fe1-063879e21902-plugins-config-volume" pod "2816cb76-b775-4ee2-8fe1-063879e21902" (UID: "2816cb76-b775-4ee2-8fe1-063879e21902"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:05.886125 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.886101 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2816cb76-b775-4ee2-8fe1-063879e21902-kube-api-access-fdkxs" pod "2816cb76-b775-4ee2-8fe1-063879e21902" (UID: "2816cb76-b775-4ee2-8fe1-063879e21902"). InnerVolumeSpecName "kube-api-access-fdkxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:05.996247 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.987669 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f613d06d-c822-4949-9f01-73d99044a3a5-kube-api-access-gcv8k\" (UniqueName: \"kubernetes.io/projected/f613d06d-c822-4949-9f01-73d99044a3a5-kube-api-access-gcv8k\") pod \"f613d06d-c822-4949-9f01-73d99044a3a5\" (UID: \"f613d06d-c822-4949-9f01-73d99044a3a5\") " Oct 30 18:28:05.996247 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.987806 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f613d06d-c822-4949-9f01-73d99044a3a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f613d06d-c822-4949-9f01-73d99044a3a5-plugins-config-volume\") pod \"f613d06d-c822-4949-9f01-73d99044a3a5\" (UID: \"f613d06d-c822-4949-9f01-73d99044a3a5\") " Oct 30 18:28:05.996247 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.988292 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2816cb76-b775-4ee2-8fe1-063879e21902-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:28:05.996247 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:05.988312 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdkxs\" (UniqueName: \"kubernetes.io/projected/2816cb76-b775-4ee2-8fe1-063879e21902-kube-api-access-fdkxs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:28:06.008947 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:06.001991 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f613d06d-c822-4949-9f01-73d99044a3a5-plugins-config-volume" pod "f613d06d-c822-4949-9f01-73d99044a3a5" (UID: "f613d06d-c822-4949-9f01-73d99044a3a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:06.012340 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:06.011451 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f613d06d-c822-4949-9f01-73d99044a3a5-kube-api-access-gcv8k" pod "f613d06d-c822-4949-9f01-73d99044a3a5" (UID: "f613d06d-c822-4949-9f01-73d99044a3a5"). InnerVolumeSpecName "kube-api-access-gcv8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:06.089418 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:06.089382 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f613d06d-c822-4949-9f01-73d99044a3a5-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:28:06.089418 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:06.089409 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcv8k\" (UniqueName: \"kubernetes.io/projected/f613d06d-c822-4949-9f01-73d99044a3a5-kube-api-access-gcv8k\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:28:06.526373 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:06.526341 4442 generic.go:356] "Generic (PLEG): container finished" podID="f613d06d-c822-4949-9f01-73d99044a3a5" containerID="d8e2b9b14300e5caf3c7933bd357e4ddbbe4ffcc108aeeaa230d1cc74eab7b34" exitCode=0 Oct 30 18:28:06.526581 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:06.526422 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a4f3f0-7c77897bf9-7mxbc" Oct 30 18:28:06.526581 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:06.526433 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a4f3f0-7c77897bf9-7mxbc" event={"ID":"f613d06d-c822-4949-9f01-73d99044a3a5","Type":"ContainerDied","Data":"d8e2b9b14300e5caf3c7933bd357e4ddbbe4ffcc108aeeaa230d1cc74eab7b34"} Oct 30 18:28:06.526581 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:06.526497 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a4f3f0-7c77897bf9-7mxbc" event={"ID":"f613d06d-c822-4949-9f01-73d99044a3a5","Type":"ContainerDied","Data":"694cea101d665717e61b2a4e20dbdde86ff5d2d24245e92200ea47d6a0cfbd7e"} Oct 30 18:28:06.526581 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:06.526530 4442 scope.go:122] "RemoveContainer" containerID="d8e2b9b14300e5caf3c7933bd357e4ddbbe4ffcc108aeeaa230d1cc74eab7b34" Oct 30 18:28:06.529952 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:06.529929 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a4f3f0-594b877768-ctrsw" Oct 30 18:28:06.530101 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:06.529970 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a4f3f0-594b877768-ctrsw" event={"ID":"2816cb76-b775-4ee2-8fe1-063879e21902","Type":"ContainerDied","Data":"a622a7728af9fbf32e4137e6dee912f913c2efa12134eea1b672f4ecd2ff43fa"} Oct 30 18:28:06.540393 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:06.540370 4442 scope.go:122] "RemoveContainer" containerID="d8e2b9b14300e5caf3c7933bd357e4ddbbe4ffcc108aeeaa230d1cc74eab7b34" Oct 30 18:28:06.541000 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:28:06.540976 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8e2b9b14300e5caf3c7933bd357e4ddbbe4ffcc108aeeaa230d1cc74eab7b34\": container with ID starting with d8e2b9b14300e5caf3c7933bd357e4ddbbe4ffcc108aeeaa230d1cc74eab7b34 not found: ID does not exist" containerID="d8e2b9b14300e5caf3c7933bd357e4ddbbe4ffcc108aeeaa230d1cc74eab7b34" Oct 30 18:28:06.541102 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:06.541007 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8e2b9b14300e5caf3c7933bd357e4ddbbe4ffcc108aeeaa230d1cc74eab7b34"} err="failed to get container status \"d8e2b9b14300e5caf3c7933bd357e4ddbbe4ffcc108aeeaa230d1cc74eab7b34\": rpc error: code = NotFound desc = could not find container \"d8e2b9b14300e5caf3c7933bd357e4ddbbe4ffcc108aeeaa230d1cc74eab7b34\": container with ID starting with d8e2b9b14300e5caf3c7933bd357e4ddbbe4ffcc108aeeaa230d1cc74eab7b34 not found: ID does not exist" Oct 30 18:28:06.541102 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:06.541027 4442 scope.go:122] "RemoveContainer" containerID="22402fadba1d91990f046a6b8fbb18e552d656be12f061039f60cf471e7fcbcf" Oct 30 18:28:06.578104 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:06.578073 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a4f3f0-594b877768-ctrsw"] Oct 30 18:28:06.589047 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:06.589020 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a4f3f0-594b877768-ctrsw"] Oct 30 18:28:06.613724 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:06.613688 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a4f3f0-7c77897bf9-7mxbc"] Oct 30 18:28:06.617043 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:06.617022 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a4f3f0-7c77897bf9-7mxbc"] Oct 30 18:28:06.883523 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:06.883490 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-be7f9e-576d7866d4-9mpwv"] Oct 30 18:28:06.923255 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:06.923226 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-be7f9e-74c4c4589f-fv7sd"] Oct 30 18:28:07.068771 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:07.068737 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2816cb76-b775-4ee2-8fe1-063879e21902" path="/var/lib/kubelet/pods/2816cb76-b775-4ee2-8fe1-063879e21902/volumes" Oct 30 18:28:07.069234 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:07.069220 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29c8e32e-e676-4b79-9643-25c88f1a6f1a" path="/var/lib/kubelet/pods/29c8e32e-e676-4b79-9643-25c88f1a6f1a/volumes" Oct 30 18:28:07.069708 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:07.069692 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f613d06d-c822-4949-9f01-73d99044a3a5" path="/var/lib/kubelet/pods/f613d06d-c822-4949-9f01-73d99044a3a5/volumes" Oct 30 18:28:11.884786 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:11.884740 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-be7f9e-576d7866d4-9mpwv" podUID="0949007d-4b27-4461-8c2a-fe96493a12aa" containerName="payload-processing" containerID="cri-o://23427faa526903c67d248d2ad1a7cad5b257cde17b4e3f12a3485cc01d574238" gracePeriod=25 Oct 30 18:28:11.924386 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:11.924345 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be7f9e-74c4c4589f-fv7sd" podUID="5db1a7fe-484f-4d61-809e-61f10e248d6d" containerName="payload-pre-processing" containerID="cri-o://aaeef3ddefab5acde2e8f97e88dd81c74fade1d1ca0effa5fb82fb147c479937" gracePeriod=25 Oct 30 18:28:12.213897 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:12.213866 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-be7f9e-576d7866d4-9mpwv" Oct 30 18:28:12.244665 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:12.244644 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be7f9e-74c4c4589f-fv7sd" Oct 30 18:28:12.284406 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:12.284373 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5db1a7fe-484f-4d61-809e-61f10e248d6d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5db1a7fe-484f-4d61-809e-61f10e248d6d-plugins-config-volume\") pod \"5db1a7fe-484f-4d61-809e-61f10e248d6d\" (UID: \"5db1a7fe-484f-4d61-809e-61f10e248d6d\") " Oct 30 18:28:12.284553 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:12.284439 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0949007d-4b27-4461-8c2a-fe96493a12aa-kube-api-access-npqhz\" (UniqueName: \"kubernetes.io/projected/0949007d-4b27-4461-8c2a-fe96493a12aa-kube-api-access-npqhz\") pod \"0949007d-4b27-4461-8c2a-fe96493a12aa\" (UID: \"0949007d-4b27-4461-8c2a-fe96493a12aa\") " Oct 30 18:28:12.284553 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:12.284501 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0949007d-4b27-4461-8c2a-fe96493a12aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0949007d-4b27-4461-8c2a-fe96493a12aa-plugins-config-volume\") pod \"0949007d-4b27-4461-8c2a-fe96493a12aa\" (UID: \"0949007d-4b27-4461-8c2a-fe96493a12aa\") " Oct 30 18:28:12.284661 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:12.284592 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5db1a7fe-484f-4d61-809e-61f10e248d6d-kube-api-access-jldvv\" (UniqueName: \"kubernetes.io/projected/5db1a7fe-484f-4d61-809e-61f10e248d6d-kube-api-access-jldvv\") pod \"5db1a7fe-484f-4d61-809e-61f10e248d6d\" (UID: \"5db1a7fe-484f-4d61-809e-61f10e248d6d\") " Oct 30 18:28:12.284881 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:12.284851 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5db1a7fe-484f-4d61-809e-61f10e248d6d-plugins-config-volume" pod "5db1a7fe-484f-4d61-809e-61f10e248d6d" (UID: "5db1a7fe-484f-4d61-809e-61f10e248d6d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:12.285004 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:12.284940 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0949007d-4b27-4461-8c2a-fe96493a12aa-plugins-config-volume" pod "0949007d-4b27-4461-8c2a-fe96493a12aa" (UID: "0949007d-4b27-4461-8c2a-fe96493a12aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:12.285202 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:12.285176 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5db1a7fe-484f-4d61-809e-61f10e248d6d-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:28:12.285202 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:12.285200 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0949007d-4b27-4461-8c2a-fe96493a12aa-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:28:12.286818 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:12.286792 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0949007d-4b27-4461-8c2a-fe96493a12aa-kube-api-access-npqhz" pod "0949007d-4b27-4461-8c2a-fe96493a12aa" (UID: "0949007d-4b27-4461-8c2a-fe96493a12aa"). InnerVolumeSpecName "kube-api-access-npqhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:12.287004 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:12.286926 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5db1a7fe-484f-4d61-809e-61f10e248d6d-kube-api-access-jldvv" pod "5db1a7fe-484f-4d61-809e-61f10e248d6d" (UID: "5db1a7fe-484f-4d61-809e-61f10e248d6d"). InnerVolumeSpecName "kube-api-access-jldvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:12.386637 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:12.386602 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jldvv\" (UniqueName: \"kubernetes.io/projected/5db1a7fe-484f-4d61-809e-61f10e248d6d-kube-api-access-jldvv\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:28:12.386637 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:12.386632 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npqhz\" (UniqueName: \"kubernetes.io/projected/0949007d-4b27-4461-8c2a-fe96493a12aa-kube-api-access-npqhz\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:28:12.617005 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:12.616970 4442 generic.go:356] "Generic (PLEG): container finished" podID="0949007d-4b27-4461-8c2a-fe96493a12aa" containerID="23427faa526903c67d248d2ad1a7cad5b257cde17b4e3f12a3485cc01d574238" exitCode=0 Oct 30 18:28:12.617220 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:12.617044 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-be7f9e-576d7866d4-9mpwv" Oct 30 18:28:12.617220 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:12.617057 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-be7f9e-576d7866d4-9mpwv" event={"ID":"0949007d-4b27-4461-8c2a-fe96493a12aa","Type":"ContainerDied","Data":"23427faa526903c67d248d2ad1a7cad5b257cde17b4e3f12a3485cc01d574238"} Oct 30 18:28:12.617220 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:12.617170 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-be7f9e-576d7866d4-9mpwv" event={"ID":"0949007d-4b27-4461-8c2a-fe96493a12aa","Type":"ContainerDied","Data":"998512d7e68076b832692c1b32935838cfc39b280f1a1b6df8d8f430d553249a"} Oct 30 18:28:12.617220 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:12.617205 4442 scope.go:122] "RemoveContainer" containerID="23427faa526903c67d248d2ad1a7cad5b257cde17b4e3f12a3485cc01d574238" Oct 30 18:28:12.618791 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:12.618766 4442 generic.go:356] "Generic (PLEG): container finished" podID="5db1a7fe-484f-4d61-809e-61f10e248d6d" containerID="aaeef3ddefab5acde2e8f97e88dd81c74fade1d1ca0effa5fb82fb147c479937" exitCode=0 Oct 30 18:28:12.618875 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:12.618805 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be7f9e-74c4c4589f-fv7sd" event={"ID":"5db1a7fe-484f-4d61-809e-61f10e248d6d","Type":"ContainerDied","Data":"aaeef3ddefab5acde2e8f97e88dd81c74fade1d1ca0effa5fb82fb147c479937"} Oct 30 18:28:12.618875 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:12.618827 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be7f9e-74c4c4589f-fv7sd" Oct 30 18:28:12.618875 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:12.618846 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be7f9e-74c4c4589f-fv7sd" event={"ID":"5db1a7fe-484f-4d61-809e-61f10e248d6d","Type":"ContainerDied","Data":"d8757083005556eee9719d804b082d2974a0806febeba4063c6dbeecb75b152f"} Oct 30 18:28:12.630537 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:12.630515 4442 scope.go:122] "RemoveContainer" containerID="23427faa526903c67d248d2ad1a7cad5b257cde17b4e3f12a3485cc01d574238" Oct 30 18:28:12.630870 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:28:12.630849 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23427faa526903c67d248d2ad1a7cad5b257cde17b4e3f12a3485cc01d574238\": container with ID starting with 23427faa526903c67d248d2ad1a7cad5b257cde17b4e3f12a3485cc01d574238 not found: ID does not exist" containerID="23427faa526903c67d248d2ad1a7cad5b257cde17b4e3f12a3485cc01d574238" Oct 30 18:28:12.630927 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:12.630879 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23427faa526903c67d248d2ad1a7cad5b257cde17b4e3f12a3485cc01d574238"} err="failed to get container status \"23427faa526903c67d248d2ad1a7cad5b257cde17b4e3f12a3485cc01d574238\": rpc error: code = NotFound desc = could not find container \"23427faa526903c67d248d2ad1a7cad5b257cde17b4e3f12a3485cc01d574238\": container with ID starting with 23427faa526903c67d248d2ad1a7cad5b257cde17b4e3f12a3485cc01d574238 not found: ID does not exist" Oct 30 18:28:12.630927 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:12.630896 4442 scope.go:122] "RemoveContainer" containerID="aaeef3ddefab5acde2e8f97e88dd81c74fade1d1ca0effa5fb82fb147c479937" Oct 30 18:28:12.643261 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:12.643239 4442 scope.go:122] "RemoveContainer" containerID="aaeef3ddefab5acde2e8f97e88dd81c74fade1d1ca0effa5fb82fb147c479937" Oct 30 18:28:12.643646 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:28:12.643608 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaeef3ddefab5acde2e8f97e88dd81c74fade1d1ca0effa5fb82fb147c479937\": container with ID starting with aaeef3ddefab5acde2e8f97e88dd81c74fade1d1ca0effa5fb82fb147c479937 not found: ID does not exist" containerID="aaeef3ddefab5acde2e8f97e88dd81c74fade1d1ca0effa5fb82fb147c479937" Oct 30 18:28:12.643750 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:12.643655 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaeef3ddefab5acde2e8f97e88dd81c74fade1d1ca0effa5fb82fb147c479937"} err="failed to get container status \"aaeef3ddefab5acde2e8f97e88dd81c74fade1d1ca0effa5fb82fb147c479937\": rpc error: code = NotFound desc = could not find container \"aaeef3ddefab5acde2e8f97e88dd81c74fade1d1ca0effa5fb82fb147c479937\": container with ID starting with aaeef3ddefab5acde2e8f97e88dd81c74fade1d1ca0effa5fb82fb147c479937 not found: ID does not exist" Oct 30 18:28:12.668727 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:12.668693 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-be7f9e-74c4c4589f-fv7sd"] Oct 30 18:28:12.680252 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:12.680219 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-be7f9e-74c4c4589f-fv7sd"] Oct 30 18:28:12.705074 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:12.705037 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-be7f9e-576d7866d4-9mpwv"] Oct 30 18:28:12.708656 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:12.708625 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-be7f9e-576d7866d4-9mpwv"] Oct 30 18:28:13.109312 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:13.109279 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0949007d-4b27-4461-8c2a-fe96493a12aa" path="/var/lib/kubelet/pods/0949007d-4b27-4461-8c2a-fe96493a12aa/volumes" Oct 30 18:28:13.110497 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:13.110475 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5db1a7fe-484f-4d61-809e-61f10e248d6d" path="/var/lib/kubelet/pods/5db1a7fe-484f-4d61-809e-61f10e248d6d/volumes" Oct 30 18:28:17.562662 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:28:17.562590 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:28:22.865084 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:22.852809 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r"] Oct 30 18:28:22.865084 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:22.853187 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" podUID="54fe8523-0ed9-477e-a23d-8c7d624ba2d4" containerName="istio-proxy" containerID="cri-o://76f1b7feda32ae9413cc5ec4e83026335b1a37fc05265995feb28e3c248f0e14" gracePeriod=30 Oct 30 18:28:23.420595 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:23.420564 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52rn6_516c6b6c-fc40-49cf-93b5-1c4471b02cf0/ovn-acl-logging/0.log" Oct 30 18:28:23.422868 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:23.422847 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52rn6_516c6b6c-fc40-49cf-93b5-1c4471b02cf0/ovn-acl-logging/0.log" Oct 30 18:28:28.295654 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:28.295619 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" Oct 30 18:28:28.313336 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:28.313300 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-istio-envoy\") pod \"54fe8523-0ed9-477e-a23d-8c7d624ba2d4\" (UID: \"54fe8523-0ed9-477e-a23d-8c7d624ba2d4\") " Oct 30 18:28:28.313555 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:28.313351 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-istio-podinfo\") pod \"54fe8523-0ed9-477e-a23d-8c7d624ba2d4\" (UID: \"54fe8523-0ed9-477e-a23d-8c7d624ba2d4\") " Oct 30 18:28:28.313555 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:28.313492 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-istio-token\" (UniqueName: \"kubernetes.io/projected/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-istio-token\") pod \"54fe8523-0ed9-477e-a23d-8c7d624ba2d4\" (UID: \"54fe8523-0ed9-477e-a23d-8c7d624ba2d4\") " Oct 30 18:28:28.313686 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:28.313581 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-kube-api-access-s9dj6\" (UniqueName: \"kubernetes.io/projected/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-kube-api-access-s9dj6\") pod \"54fe8523-0ed9-477e-a23d-8c7d624ba2d4\" (UID: \"54fe8523-0ed9-477e-a23d-8c7d624ba2d4\") " Oct 30 18:28:28.313686 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:28.313613 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-credential-socket\") pod \"54fe8523-0ed9-477e-a23d-8c7d624ba2d4\" (UID: \"54fe8523-0ed9-477e-a23d-8c7d624ba2d4\") " Oct 30 18:28:28.313686 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:28.313649 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-workload-socket\") pod \"54fe8523-0ed9-477e-a23d-8c7d624ba2d4\" (UID: \"54fe8523-0ed9-477e-a23d-8c7d624ba2d4\") " Oct 30 18:28:28.313852 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:28.313693 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-workload-certs\") pod \"54fe8523-0ed9-477e-a23d-8c7d624ba2d4\" (UID: \"54fe8523-0ed9-477e-a23d-8c7d624ba2d4\") " Oct 30 18:28:28.313852 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:28.313724 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-istiod-ca-cert\") pod \"54fe8523-0ed9-477e-a23d-8c7d624ba2d4\" (UID: \"54fe8523-0ed9-477e-a23d-8c7d624ba2d4\") " Oct 30 18:28:28.313852 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:28.313760 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-istio-data\") pod \"54fe8523-0ed9-477e-a23d-8c7d624ba2d4\" (UID: \"54fe8523-0ed9-477e-a23d-8c7d624ba2d4\") " Oct 30 18:28:28.314027 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:28.313854 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-credential-socket" pod "54fe8523-0ed9-477e-a23d-8c7d624ba2d4" (UID: "54fe8523-0ed9-477e-a23d-8c7d624ba2d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:28.314027 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:28.314002 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-workload-certs" pod "54fe8523-0ed9-477e-a23d-8c7d624ba2d4" (UID: "54fe8523-0ed9-477e-a23d-8c7d624ba2d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:28.314161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:28.314047 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-istiod-ca-cert" pod "54fe8523-0ed9-477e-a23d-8c7d624ba2d4" (UID: "54fe8523-0ed9-477e-a23d-8c7d624ba2d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:28.314393 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:28.314173 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-workload-socket" pod "54fe8523-0ed9-477e-a23d-8c7d624ba2d4" (UID: "54fe8523-0ed9-477e-a23d-8c7d624ba2d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:28.314393 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:28.314193 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-istio-data" pod "54fe8523-0ed9-477e-a23d-8c7d624ba2d4" (UID: "54fe8523-0ed9-477e-a23d-8c7d624ba2d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:28.314393 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:28.314352 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:28:28.314393 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:28.314373 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:28:28.314393 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:28.314389 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:28:28.314670 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:28.314403 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:28:28.314670 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:28.314415 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:28:28.316058 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:28.316028 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-istio-token" pod "54fe8523-0ed9-477e-a23d-8c7d624ba2d4" (UID: "54fe8523-0ed9-477e-a23d-8c7d624ba2d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:28.316330 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:28.316309 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-istio-envoy" pod "54fe8523-0ed9-477e-a23d-8c7d624ba2d4" (UID: "54fe8523-0ed9-477e-a23d-8c7d624ba2d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:28.316476 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:28.316458 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-kube-api-access-s9dj6" pod "54fe8523-0ed9-477e-a23d-8c7d624ba2d4" (UID: "54fe8523-0ed9-477e-a23d-8c7d624ba2d4"). InnerVolumeSpecName "kube-api-access-s9dj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:28.316533 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:28.316471 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-istio-podinfo" pod "54fe8523-0ed9-477e-a23d-8c7d624ba2d4" (UID: "54fe8523-0ed9-477e-a23d-8c7d624ba2d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:28:28.448207 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:28.448067 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:28:28.448207 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:28.448107 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9dj6\" (UniqueName: \"kubernetes.io/projected/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-kube-api-access-s9dj6\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:28:28.448207 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:28.448122 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:28:28.448207 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:28.448154 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54fe8523-0ed9-477e-a23d-8c7d624ba2d4-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:28:28.894529 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:28.893906 4442 generic.go:356] "Generic (PLEG): container finished" podID="54fe8523-0ed9-477e-a23d-8c7d624ba2d4" containerID="76f1b7feda32ae9413cc5ec4e83026335b1a37fc05265995feb28e3c248f0e14" exitCode=0 Oct 30 18:28:28.894529 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:28.894045 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" event={"ID":"54fe8523-0ed9-477e-a23d-8c7d624ba2d4","Type":"ContainerDied","Data":"76f1b7feda32ae9413cc5ec4e83026335b1a37fc05265995feb28e3c248f0e14"} Oct 30 18:28:28.894529 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:28.894099 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" event={"ID":"54fe8523-0ed9-477e-a23d-8c7d624ba2d4","Type":"ContainerDied","Data":"a4d9dd87156c4cd21798944c948ce8816bb1f8e59954fdc32a9cbc976b6f6e30"} Oct 30 18:28:28.894529 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:28.894152 4442 scope.go:122] "RemoveContainer" containerID="76f1b7feda32ae9413cc5ec4e83026335b1a37fc05265995feb28e3c248f0e14" Oct 30 18:28:28.894529 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:28.894342 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r" Oct 30 18:28:28.918792 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:28.918764 4442 scope.go:122] "RemoveContainer" containerID="be3cc516aab817d15f7d3dfd489b7545d6353c939bff37985ee155f6f00919df" Oct 30 18:28:28.930269 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:28.930216 4442 scope.go:122] "RemoveContainer" containerID="76f1b7feda32ae9413cc5ec4e83026335b1a37fc05265995feb28e3c248f0e14" Oct 30 18:28:28.930601 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:28:28.930576 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76f1b7feda32ae9413cc5ec4e83026335b1a37fc05265995feb28e3c248f0e14\": container with ID starting with 76f1b7feda32ae9413cc5ec4e83026335b1a37fc05265995feb28e3c248f0e14 not found: ID does not exist" containerID="76f1b7feda32ae9413cc5ec4e83026335b1a37fc05265995feb28e3c248f0e14" Oct 30 18:28:28.930675 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:28.930610 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76f1b7feda32ae9413cc5ec4e83026335b1a37fc05265995feb28e3c248f0e14"} err="failed to get container status \"76f1b7feda32ae9413cc5ec4e83026335b1a37fc05265995feb28e3c248f0e14\": rpc error: code = NotFound desc = could not find container \"76f1b7feda32ae9413cc5ec4e83026335b1a37fc05265995feb28e3c248f0e14\": container with ID starting with 76f1b7feda32ae9413cc5ec4e83026335b1a37fc05265995feb28e3c248f0e14 not found: ID does not exist" Oct 30 18:28:28.930675 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:28.930628 4442 scope.go:122] "RemoveContainer" containerID="be3cc516aab817d15f7d3dfd489b7545d6353c939bff37985ee155f6f00919df" Oct 30 18:28:28.930925 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:28:28.930907 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be3cc516aab817d15f7d3dfd489b7545d6353c939bff37985ee155f6f00919df\": container with ID starting with be3cc516aab817d15f7d3dfd489b7545d6353c939bff37985ee155f6f00919df not found: ID does not exist" containerID="be3cc516aab817d15f7d3dfd489b7545d6353c939bff37985ee155f6f00919df" Oct 30 18:28:28.930989 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:28.930932 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be3cc516aab817d15f7d3dfd489b7545d6353c939bff37985ee155f6f00919df"} err="failed to get container status \"be3cc516aab817d15f7d3dfd489b7545d6353c939bff37985ee155f6f00919df\": rpc error: code = NotFound desc = could not find container \"be3cc516aab817d15f7d3dfd489b7545d6353c939bff37985ee155f6f00919df\": container with ID starting with be3cc516aab817d15f7d3dfd489b7545d6353c939bff37985ee155f6f00919df not found: ID does not exist" Oct 30 18:28:28.958196 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:28.957253 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r"] Oct 30 18:28:28.969024 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:28.968980 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-be7f9e-openshift-default-7cb56cd649-8sn5r"] Oct 30 18:28:29.103194 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:29.103125 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54fe8523-0ed9-477e-a23d-8c7d624ba2d4" path="/var/lib/kubelet/pods/54fe8523-0ed9-477e-a23d-8c7d624ba2d4/volumes" Oct 30 18:28:31.919348 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:31.919312 4442 generic.go:356] "Generic (PLEG): container finished" podID="43c061bf-f41c-4bb8-93ca-fe9a3ee5e690" containerID="2a05cfd6b5ffdfdacede4a79d5df7c56a2863053ddba693efa47ea469237e376" exitCode=0 Oct 30 18:28:31.919834 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:31.919397 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" event={"ID":"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690","Type":"ContainerDied","Data":"2a05cfd6b5ffdfdacede4a79d5df7c56a2863053ddba693efa47ea469237e376"} Oct 30 18:28:31.919964 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:31.919944 4442 scope.go:122] "RemoveContainer" containerID="2a05cfd6b5ffdfdacede4a79d5df7c56a2863053ddba693efa47ea469237e376" Oct 30 18:28:31.921571 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:31.921548 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:28:31.921646 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:31.921612 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:28:31.921646 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:31.921639 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:28:32.952351 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:32.952302 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" event={"ID":"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690","Type":"ContainerStarted","Data":"5f2d2b9fe82ddbf3708053bdb244c27c1b9f9c1b55e09646017b5cb6f7440a3e"} Oct 30 18:28:32.982313 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:32.982285 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" Oct 30 18:28:33.121378 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:33.121344 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" Oct 30 18:28:33.984358 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:33.984193 4442 patch_prober.go:29] interesting pod/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 18:28:33.984358 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:33.984256 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" podUID="43c061bf-f41c-4bb8-93ca-fe9a3ee5e690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 18:28:34.311723 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:34.311647 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-a4f3f0/praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk"] Oct 30 18:28:34.694333 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:34.694292 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f826dd-7cf496bf8b-qwhl9"] Oct 30 18:28:34.826160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:34.814497 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f826dd-7cf496bf8b-qwhl9" podUID="34802091-fbcd-4b48-90da-96ee88ec3aae" containerName="maas-api" containerID="cri-o://68239458ce1d0966b6543a4fbe1f9ad8fa7c89e8514b52b4fb81fd6865d0f43e" gracePeriod=30 Oct 30 18:28:34.993174 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:34.984465 4442 patch_prober.go:29] interesting pod/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 18:28:34.993174 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:34.984554 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" podUID="43c061bf-f41c-4bb8-93ca-fe9a3ee5e690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 18:28:35.023061 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:35.023026 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f826dd-66fc94df7b-vd626"] Oct 30 18:28:35.040661 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:35.040597 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f826dd-66fc94df7b-vd626" podUID="6d7a6ab3-2d04-41ec-acc3-df776501aa5e" containerName="payload-processing" containerID="cri-o://f48281dac0ece7e15609356abc48e21c3bc35a5a76dbc031707142b299ae5638" gracePeriod=30 Oct 30 18:28:35.085820 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:35.085699 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-675f9fd755-t8pkx"] Oct 30 18:28:35.086356 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:35.086321 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-675f9fd755-t8pkx" podUID="9de8d55a-7a7c-4e6d-9ec4-43a1d779f30e" containerName="payload-pre-processing" containerID="cri-o://c1ee8edb689eb9ca12f5cebde8e8b080b4bba4fec1e67a722344a2734c370287" gracePeriod=30 Oct 30 18:28:35.565874 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:35.565363 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f826dd-7cf496bf8b-qwhl9" Oct 30 18:28:35.664276 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:35.664246 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f826dd-66fc94df7b-vd626" Oct 30 18:28:35.666546 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:35.666523 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34802091-fbcd-4b48-90da-96ee88ec3aae-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34802091-fbcd-4b48-90da-96ee88ec3aae-openshift-service-ca\") pod \"34802091-fbcd-4b48-90da-96ee88ec3aae\" (UID: \"34802091-fbcd-4b48-90da-96ee88ec3aae\") " Oct 30 18:28:35.666665 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:35.666597 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34802091-fbcd-4b48-90da-96ee88ec3aae-kube-api-access-nxjsj\" (UniqueName: \"kubernetes.io/projected/34802091-fbcd-4b48-90da-96ee88ec3aae-kube-api-access-nxjsj\") pod \"34802091-fbcd-4b48-90da-96ee88ec3aae\" (UID: \"34802091-fbcd-4b48-90da-96ee88ec3aae\") " Oct 30 18:28:35.666665 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:35.666633 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34802091-fbcd-4b48-90da-96ee88ec3aae-metrics-certs\" (UniqueName: \"kubernetes.io/secret/34802091-fbcd-4b48-90da-96ee88ec3aae-metrics-certs\") pod \"34802091-fbcd-4b48-90da-96ee88ec3aae\" (UID: \"34802091-fbcd-4b48-90da-96ee88ec3aae\") " Oct 30 18:28:35.666780 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:35.666669 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34802091-fbcd-4b48-90da-96ee88ec3aae-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34802091-fbcd-4b48-90da-96ee88ec3aae-maas-api-tls\") pod \"34802091-fbcd-4b48-90da-96ee88ec3aae\" (UID: \"34802091-fbcd-4b48-90da-96ee88ec3aae\") " Oct 30 18:28:35.666942 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:35.666913 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34802091-fbcd-4b48-90da-96ee88ec3aae-openshift-service-ca" pod "34802091-fbcd-4b48-90da-96ee88ec3aae" (UID: "34802091-fbcd-4b48-90da-96ee88ec3aae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:35.667235 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:35.667214 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34802091-fbcd-4b48-90da-96ee88ec3aae-openshift-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:28:35.669410 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:35.669378 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34802091-fbcd-4b48-90da-96ee88ec3aae-maas-api-tls" pod "34802091-fbcd-4b48-90da-96ee88ec3aae" (UID: "34802091-fbcd-4b48-90da-96ee88ec3aae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:28:35.670055 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:35.669794 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34802091-fbcd-4b48-90da-96ee88ec3aae-kube-api-access-nxjsj" pod "34802091-fbcd-4b48-90da-96ee88ec3aae" (UID: "34802091-fbcd-4b48-90da-96ee88ec3aae"). InnerVolumeSpecName "kube-api-access-nxjsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:35.670055 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:35.669932 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34802091-fbcd-4b48-90da-96ee88ec3aae-metrics-certs" pod "34802091-fbcd-4b48-90da-96ee88ec3aae" (UID: "34802091-fbcd-4b48-90da-96ee88ec3aae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:28:35.670456 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:35.670410 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-675f9fd755-t8pkx" Oct 30 18:28:35.769019 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:35.768886 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d7a6ab3-2d04-41ec-acc3-df776501aa5e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d7a6ab3-2d04-41ec-acc3-df776501aa5e-plugins-config-volume\") pod \"6d7a6ab3-2d04-41ec-acc3-df776501aa5e\" (UID: \"6d7a6ab3-2d04-41ec-acc3-df776501aa5e\") " Oct 30 18:28:35.769019 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:35.769011 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d7a6ab3-2d04-41ec-acc3-df776501aa5e-kube-api-access-982dh\" (UniqueName: \"kubernetes.io/projected/6d7a6ab3-2d04-41ec-acc3-df776501aa5e-kube-api-access-982dh\") pod \"6d7a6ab3-2d04-41ec-acc3-df776501aa5e\" (UID: \"6d7a6ab3-2d04-41ec-acc3-df776501aa5e\") " Oct 30 18:28:35.771613 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:35.769214 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9de8d55a-7a7c-4e6d-9ec4-43a1d779f30e-kube-api-access-q7tkb\" (UniqueName: \"kubernetes.io/projected/9de8d55a-7a7c-4e6d-9ec4-43a1d779f30e-kube-api-access-q7tkb\") pod \"9de8d55a-7a7c-4e6d-9ec4-43a1d779f30e\" (UID: \"9de8d55a-7a7c-4e6d-9ec4-43a1d779f30e\") " Oct 30 18:28:35.771613 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:35.769391 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9de8d55a-7a7c-4e6d-9ec4-43a1d779f30e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9de8d55a-7a7c-4e6d-9ec4-43a1d779f30e-plugins-config-volume\") pod \"9de8d55a-7a7c-4e6d-9ec4-43a1d779f30e\" (UID: \"9de8d55a-7a7c-4e6d-9ec4-43a1d779f30e\") " Oct 30 18:28:35.771613 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:35.769853 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxjsj\" (UniqueName: \"kubernetes.io/projected/34802091-fbcd-4b48-90da-96ee88ec3aae-kube-api-access-nxjsj\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:28:35.771613 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:35.769872 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34802091-fbcd-4b48-90da-96ee88ec3aae-metrics-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:28:35.771613 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:35.769888 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34802091-fbcd-4b48-90da-96ee88ec3aae-maas-api-tls\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:28:35.771613 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:35.770315 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9de8d55a-7a7c-4e6d-9ec4-43a1d779f30e-plugins-config-volume" pod "9de8d55a-7a7c-4e6d-9ec4-43a1d779f30e" (UID: "9de8d55a-7a7c-4e6d-9ec4-43a1d779f30e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:35.771613 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:35.770763 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d7a6ab3-2d04-41ec-acc3-df776501aa5e-plugins-config-volume" pod "6d7a6ab3-2d04-41ec-acc3-df776501aa5e" (UID: "6d7a6ab3-2d04-41ec-acc3-df776501aa5e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:35.773581 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:35.773558 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d7a6ab3-2d04-41ec-acc3-df776501aa5e-kube-api-access-982dh" pod "6d7a6ab3-2d04-41ec-acc3-df776501aa5e" (UID: "6d7a6ab3-2d04-41ec-acc3-df776501aa5e"). InnerVolumeSpecName "kube-api-access-982dh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:35.773880 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:35.773865 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9de8d55a-7a7c-4e6d-9ec4-43a1d779f30e-kube-api-access-q7tkb" pod "9de8d55a-7a7c-4e6d-9ec4-43a1d779f30e" (UID: "9de8d55a-7a7c-4e6d-9ec4-43a1d779f30e"). InnerVolumeSpecName "kube-api-access-q7tkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:35.871436 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:35.871404 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9de8d55a-7a7c-4e6d-9ec4-43a1d779f30e-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:28:35.871436 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:35.871435 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d7a6ab3-2d04-41ec-acc3-df776501aa5e-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:28:35.871646 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:35.871448 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-982dh\" (UniqueName: \"kubernetes.io/projected/6d7a6ab3-2d04-41ec-acc3-df776501aa5e-kube-api-access-982dh\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:28:35.871646 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:35.871461 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7tkb\" (UniqueName: \"kubernetes.io/projected/9de8d55a-7a7c-4e6d-9ec4-43a1d779f30e-kube-api-access-q7tkb\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:28:35.996159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:35.995281 4442 patch_prober.go:29] interesting pod/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 18:28:35.996159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:35.995417 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" podUID="43c061bf-f41c-4bb8-93ca-fe9a3ee5e690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 18:28:36.016308 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:36.016272 4442 generic.go:356] "Generic (PLEG): container finished" podID="6d7a6ab3-2d04-41ec-acc3-df776501aa5e" containerID="f48281dac0ece7e15609356abc48e21c3bc35a5a76dbc031707142b299ae5638" exitCode=0 Oct 30 18:28:36.016526 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:36.016354 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f826dd-66fc94df7b-vd626" event={"ID":"6d7a6ab3-2d04-41ec-acc3-df776501aa5e","Type":"ContainerDied","Data":"f48281dac0ece7e15609356abc48e21c3bc35a5a76dbc031707142b299ae5638"} Oct 30 18:28:36.016526 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:36.016401 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f826dd-66fc94df7b-vd626" event={"ID":"6d7a6ab3-2d04-41ec-acc3-df776501aa5e","Type":"ContainerDied","Data":"f28da4d374eddaa54a2e279b9d37b181d9601675aedc228c32cb4dc357930a59"} Oct 30 18:28:36.016526 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:36.016430 4442 scope.go:122] "RemoveContainer" containerID="f48281dac0ece7e15609356abc48e21c3bc35a5a76dbc031707142b299ae5638" Oct 30 18:28:36.016526 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:36.016526 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f826dd-66fc94df7b-vd626" Oct 30 18:28:36.019323 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:36.019254 4442 generic.go:356] "Generic (PLEG): container finished" podID="9de8d55a-7a7c-4e6d-9ec4-43a1d779f30e" containerID="c1ee8edb689eb9ca12f5cebde8e8b080b4bba4fec1e67a722344a2734c370287" exitCode=0 Oct 30 18:28:36.019490 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:36.019331 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-675f9fd755-t8pkx" Oct 30 18:28:36.019490 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:36.019449 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-675f9fd755-t8pkx" event={"ID":"9de8d55a-7a7c-4e6d-9ec4-43a1d779f30e","Type":"ContainerDied","Data":"c1ee8edb689eb9ca12f5cebde8e8b080b4bba4fec1e67a722344a2734c370287"} Oct 30 18:28:36.019663 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:36.019495 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-675f9fd755-t8pkx" event={"ID":"9de8d55a-7a7c-4e6d-9ec4-43a1d779f30e","Type":"ContainerDied","Data":"55756afdf229a126e2d4bc7ca669a00dd5fffc273a9ac4307655fe40582f5eb5"} Oct 30 18:28:36.021606 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:36.021534 4442 generic.go:356] "Generic (PLEG): container finished" podID="34802091-fbcd-4b48-90da-96ee88ec3aae" containerID="68239458ce1d0966b6543a4fbe1f9ad8fa7c89e8514b52b4fb81fd6865d0f43e" exitCode=0 Oct 30 18:28:36.021606 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:36.021601 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f826dd-7cf496bf8b-qwhl9" event={"ID":"34802091-fbcd-4b48-90da-96ee88ec3aae","Type":"ContainerDied","Data":"68239458ce1d0966b6543a4fbe1f9ad8fa7c89e8514b52b4fb81fd6865d0f43e"} Oct 30 18:28:36.021800 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:36.021642 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f826dd-7cf496bf8b-qwhl9" event={"ID":"34802091-fbcd-4b48-90da-96ee88ec3aae","Type":"ContainerDied","Data":"c505402738ab30ec52106e27c67fb27b14d8a870b85a46e051391af93d868b09"} Oct 30 18:28:36.021800 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:36.021705 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f826dd-7cf496bf8b-qwhl9" Oct 30 18:28:36.065612 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:36.061311 4442 scope.go:122] "RemoveContainer" containerID="f48281dac0ece7e15609356abc48e21c3bc35a5a76dbc031707142b299ae5638" Oct 30 18:28:36.065612 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:28:36.062341 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f48281dac0ece7e15609356abc48e21c3bc35a5a76dbc031707142b299ae5638\": container with ID starting with f48281dac0ece7e15609356abc48e21c3bc35a5a76dbc031707142b299ae5638 not found: ID does not exist" containerID="f48281dac0ece7e15609356abc48e21c3bc35a5a76dbc031707142b299ae5638" Oct 30 18:28:36.065612 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:36.062369 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f48281dac0ece7e15609356abc48e21c3bc35a5a76dbc031707142b299ae5638"} err="failed to get container status \"f48281dac0ece7e15609356abc48e21c3bc35a5a76dbc031707142b299ae5638\": rpc error: code = NotFound desc = could not find container \"f48281dac0ece7e15609356abc48e21c3bc35a5a76dbc031707142b299ae5638\": container with ID starting with f48281dac0ece7e15609356abc48e21c3bc35a5a76dbc031707142b299ae5638 not found: ID does not exist" Oct 30 18:28:36.065612 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:36.062386 4442 scope.go:122] "RemoveContainer" containerID="c1ee8edb689eb9ca12f5cebde8e8b080b4bba4fec1e67a722344a2734c370287" Oct 30 18:28:36.077168 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:36.077112 4442 scope.go:122] "RemoveContainer" containerID="c1ee8edb689eb9ca12f5cebde8e8b080b4bba4fec1e67a722344a2734c370287" Oct 30 18:28:36.077473 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:28:36.077449 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1ee8edb689eb9ca12f5cebde8e8b080b4bba4fec1e67a722344a2734c370287\": container with ID starting with c1ee8edb689eb9ca12f5cebde8e8b080b4bba4fec1e67a722344a2734c370287 not found: ID does not exist" containerID="c1ee8edb689eb9ca12f5cebde8e8b080b4bba4fec1e67a722344a2734c370287" Oct 30 18:28:36.077541 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:36.077487 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1ee8edb689eb9ca12f5cebde8e8b080b4bba4fec1e67a722344a2734c370287"} err="failed to get container status \"c1ee8edb689eb9ca12f5cebde8e8b080b4bba4fec1e67a722344a2734c370287\": rpc error: code = NotFound desc = could not find container \"c1ee8edb689eb9ca12f5cebde8e8b080b4bba4fec1e67a722344a2734c370287\": container with ID starting with c1ee8edb689eb9ca12f5cebde8e8b080b4bba4fec1e67a722344a2734c370287 not found: ID does not exist" Oct 30 18:28:36.077541 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:36.077512 4442 scope.go:122] "RemoveContainer" containerID="68239458ce1d0966b6543a4fbe1f9ad8fa7c89e8514b52b4fb81fd6865d0f43e" Oct 30 18:28:36.087332 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:36.087311 4442 scope.go:122] "RemoveContainer" containerID="68239458ce1d0966b6543a4fbe1f9ad8fa7c89e8514b52b4fb81fd6865d0f43e" Oct 30 18:28:36.087774 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:28:36.087747 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68239458ce1d0966b6543a4fbe1f9ad8fa7c89e8514b52b4fb81fd6865d0f43e\": container with ID starting with 68239458ce1d0966b6543a4fbe1f9ad8fa7c89e8514b52b4fb81fd6865d0f43e not found: ID does not exist" containerID="68239458ce1d0966b6543a4fbe1f9ad8fa7c89e8514b52b4fb81fd6865d0f43e" Oct 30 18:28:36.087858 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:36.087776 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68239458ce1d0966b6543a4fbe1f9ad8fa7c89e8514b52b4fb81fd6865d0f43e"} err="failed to get container status \"68239458ce1d0966b6543a4fbe1f9ad8fa7c89e8514b52b4fb81fd6865d0f43e\": rpc error: code = NotFound desc = could not find container \"68239458ce1d0966b6543a4fbe1f9ad8fa7c89e8514b52b4fb81fd6865d0f43e\": container with ID starting with 68239458ce1d0966b6543a4fbe1f9ad8fa7c89e8514b52b4fb81fd6865d0f43e not found: ID does not exist" Oct 30 18:28:36.115332 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:36.115301 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f826dd-7cf496bf8b-qwhl9"] Oct 30 18:28:36.118666 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:36.118645 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f826dd-7cf496bf8b-qwhl9"] Oct 30 18:28:36.161892 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:36.161810 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-675f9fd755-t8pkx"] Oct 30 18:28:36.164892 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:36.164868 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f826dd-675f9fd755-t8pkx"] Oct 30 18:28:36.196903 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:36.196867 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f826dd-66fc94df7b-vd626"] Oct 30 18:28:36.202180 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:36.202150 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f826dd-66fc94df7b-vd626"] Oct 30 18:28:36.982937 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:36.982898 4442 patch_prober.go:29] interesting pod/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 18:28:36.983119 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:36.982969 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" podUID="43c061bf-f41c-4bb8-93ca-fe9a3ee5e690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 18:28:37.064940 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:37.064905 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34802091-fbcd-4b48-90da-96ee88ec3aae" path="/var/lib/kubelet/pods/34802091-fbcd-4b48-90da-96ee88ec3aae/volumes" Oct 30 18:28:37.065417 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:37.065401 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d7a6ab3-2d04-41ec-acc3-df776501aa5e" path="/var/lib/kubelet/pods/6d7a6ab3-2d04-41ec-acc3-df776501aa5e/volumes" Oct 30 18:28:37.065749 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:37.065736 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9de8d55a-7a7c-4e6d-9ec4-43a1d779f30e" path="/var/lib/kubelet/pods/9de8d55a-7a7c-4e6d-9ec4-43a1d779f30e/volumes" Oct 30 18:28:37.982960 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:37.982914 4442 patch_prober.go:29] interesting pod/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 18:28:37.983201 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:37.983000 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" podUID="43c061bf-f41c-4bb8-93ca-fe9a3ee5e690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 18:28:38.235233 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.235111 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww"] Oct 30 18:28:38.236019 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.236000 4442 cpu_manager.go:414] "containerMap: removing container" podUID="29c8e32e-e676-4b79-9643-25c88f1a6f1a" containerName="istio-proxy" Oct 30 18:28:38.236074 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.236025 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29c8e32e-e676-4b79-9643-25c88f1a6f1a" containerName="istio-proxy" Oct 30 18:28:38.236074 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.236041 4442 cpu_manager.go:414] "containerMap: removing container" podUID="0949007d-4b27-4461-8c2a-fe96493a12aa" containerName="payload-processing" Oct 30 18:28:38.236074 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.236052 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0949007d-4b27-4461-8c2a-fe96493a12aa" containerName="payload-processing" Oct 30 18:28:38.236074 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.236066 4442 cpu_manager.go:414] "containerMap: removing container" podUID="29c8e32e-e676-4b79-9643-25c88f1a6f1a" containerName="istio-proxy" Oct 30 18:28:38.236242 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.236074 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29c8e32e-e676-4b79-9643-25c88f1a6f1a" containerName="istio-proxy" Oct 30 18:28:38.236242 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.236086 4442 cpu_manager.go:414] "containerMap: removing container" podUID="5db1a7fe-484f-4d61-809e-61f10e248d6d" containerName="payload-pre-processing" Oct 30 18:28:38.236242 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.236094 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5db1a7fe-484f-4d61-809e-61f10e248d6d" containerName="payload-pre-processing" Oct 30 18:28:38.236242 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.236125 4442 cpu_manager.go:414] "containerMap: removing container" podUID="9de8d55a-7a7c-4e6d-9ec4-43a1d779f30e" containerName="payload-pre-processing" Oct 30 18:28:38.236242 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.236150 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9de8d55a-7a7c-4e6d-9ec4-43a1d779f30e" containerName="payload-pre-processing" Oct 30 18:28:38.236242 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.236162 4442 cpu_manager.go:414] "containerMap: removing container" podUID="f613d06d-c822-4949-9f01-73d99044a3a5" containerName="payload-pre-processing" Oct 30 18:28:38.236242 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.236170 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f613d06d-c822-4949-9f01-73d99044a3a5" containerName="payload-pre-processing" Oct 30 18:28:38.236242 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.236184 4442 cpu_manager.go:414] "containerMap: removing container" podUID="54fe8523-0ed9-477e-a23d-8c7d624ba2d4" containerName="istio-proxy" Oct 30 18:28:38.236242 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.236191 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54fe8523-0ed9-477e-a23d-8c7d624ba2d4" containerName="istio-proxy" Oct 30 18:28:38.236242 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.236210 4442 cpu_manager.go:414] "containerMap: removing container" podUID="7c46dd87-b6ad-4929-bf8c-ab10fd137e5b" containerName="payload-processing" Oct 30 18:28:38.236242 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.236217 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c46dd87-b6ad-4929-bf8c-ab10fd137e5b" containerName="payload-processing" Oct 30 18:28:38.236242 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.236236 4442 cpu_manager.go:414] "containerMap: removing container" podUID="54fe8523-0ed9-477e-a23d-8c7d624ba2d4" containerName="istio-proxy" Oct 30 18:28:38.236242 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.236243 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54fe8523-0ed9-477e-a23d-8c7d624ba2d4" containerName="istio-proxy" Oct 30 18:28:38.236611 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.236264 4442 cpu_manager.go:414] "containerMap: removing container" podUID="34802091-fbcd-4b48-90da-96ee88ec3aae" containerName="maas-api" Oct 30 18:28:38.236611 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.236271 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34802091-fbcd-4b48-90da-96ee88ec3aae" containerName="maas-api" Oct 30 18:28:38.236611 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.236286 4442 cpu_manager.go:414] "containerMap: removing container" podUID="6d7a6ab3-2d04-41ec-acc3-df776501aa5e" containerName="payload-processing" Oct 30 18:28:38.236611 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.236293 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d7a6ab3-2d04-41ec-acc3-df776501aa5e" containerName="payload-processing" Oct 30 18:28:38.236611 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.236304 4442 cpu_manager.go:414] "containerMap: removing container" podUID="c2ee7fe9-5c24-422e-9140-5d1ec2564efc" containerName="payload-pre-processing" Oct 30 18:28:38.236611 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.236311 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2ee7fe9-5c24-422e-9140-5d1ec2564efc" containerName="payload-pre-processing" Oct 30 18:28:38.236611 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.236330 4442 cpu_manager.go:414] "containerMap: removing container" podUID="bc0f6b52-69e6-422e-bc46-55ee9f7c8353" containerName="maas-api" Oct 30 18:28:38.236611 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.236336 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc0f6b52-69e6-422e-bc46-55ee9f7c8353" containerName="maas-api" Oct 30 18:28:38.236611 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.236350 4442 cpu_manager.go:414] "containerMap: removing container" podUID="2816cb76-b775-4ee2-8fe1-063879e21902" containerName="payload-processing" Oct 30 18:28:38.236611 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.236357 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2816cb76-b775-4ee2-8fe1-063879e21902" containerName="payload-processing" Oct 30 18:28:38.236611 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.236495 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="29c8e32e-e676-4b79-9643-25c88f1a6f1a" containerName="istio-proxy" Oct 30 18:28:38.236611 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.236509 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="0949007d-4b27-4461-8c2a-fe96493a12aa" containerName="payload-processing" Oct 30 18:28:38.236611 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.236519 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="29c8e32e-e676-4b79-9643-25c88f1a6f1a" containerName="istio-proxy" Oct 30 18:28:38.236611 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.236530 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="5db1a7fe-484f-4d61-809e-61f10e248d6d" containerName="payload-pre-processing" Oct 30 18:28:38.236611 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.236542 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="9de8d55a-7a7c-4e6d-9ec4-43a1d779f30e" containerName="payload-pre-processing" Oct 30 18:28:38.236611 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.236551 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="f613d06d-c822-4949-9f01-73d99044a3a5" containerName="payload-pre-processing" Oct 30 18:28:38.236611 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.236559 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="54fe8523-0ed9-477e-a23d-8c7d624ba2d4" containerName="istio-proxy" Oct 30 18:28:38.236611 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.236569 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c46dd87-b6ad-4929-bf8c-ab10fd137e5b" containerName="payload-processing" Oct 30 18:28:38.236611 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.236579 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="54fe8523-0ed9-477e-a23d-8c7d624ba2d4" containerName="istio-proxy" Oct 30 18:28:38.236611 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.236591 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="34802091-fbcd-4b48-90da-96ee88ec3aae" containerName="maas-api" Oct 30 18:28:38.236611 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.236600 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d7a6ab3-2d04-41ec-acc3-df776501aa5e" containerName="payload-processing" Oct 30 18:28:38.236611 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.236608 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2ee7fe9-5c24-422e-9140-5d1ec2564efc" containerName="payload-pre-processing" Oct 30 18:28:38.236611 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.236619 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc0f6b52-69e6-422e-bc46-55ee9f7c8353" containerName="maas-api" Oct 30 18:28:38.237288 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.236628 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="2816cb76-b775-4ee2-8fe1-063879e21902" containerName="payload-processing" Oct 30 18:28:38.240469 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.240448 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" Oct 30 18:28:38.266359 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.266312 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww"] Oct 30 18:28:38.308793 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.308754 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdckw\" (UniqueName: \"kubernetes.io/projected/ffc6ceba-80a2-4722-a064-d409fe90fde4-kube-api-access-hdckw\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww\" (UID: \"ffc6ceba-80a2-4722-a064-d409fe90fde4\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" Oct 30 18:28:38.308988 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.308950 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffc6ceba-80a2-4722-a064-d409fe90fde4-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww\" (UID: \"ffc6ceba-80a2-4722-a064-d409fe90fde4\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" Oct 30 18:28:38.309032 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.309012 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffc6ceba-80a2-4722-a064-d409fe90fde4-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww\" (UID: \"ffc6ceba-80a2-4722-a064-d409fe90fde4\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" Oct 30 18:28:38.309067 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.309043 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffc6ceba-80a2-4722-a064-d409fe90fde4-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww\" (UID: \"ffc6ceba-80a2-4722-a064-d409fe90fde4\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" Oct 30 18:28:38.309162 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.309104 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffc6ceba-80a2-4722-a064-d409fe90fde4-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww\" (UID: \"ffc6ceba-80a2-4722-a064-d409fe90fde4\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" Oct 30 18:28:38.309250 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.309207 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc6ceba-80a2-4722-a064-d409fe90fde4-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww\" (UID: \"ffc6ceba-80a2-4722-a064-d409fe90fde4\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" Oct 30 18:28:38.309298 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.309255 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffc6ceba-80a2-4722-a064-d409fe90fde4-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww\" (UID: \"ffc6ceba-80a2-4722-a064-d409fe90fde4\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" Oct 30 18:28:38.309298 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.309290 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc6ceba-80a2-4722-a064-d409fe90fde4-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww\" (UID: \"ffc6ceba-80a2-4722-a064-d409fe90fde4\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" Oct 30 18:28:38.309415 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.309360 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffc6ceba-80a2-4722-a064-d409fe90fde4-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww\" (UID: \"ffc6ceba-80a2-4722-a064-d409fe90fde4\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" Oct 30 18:28:38.411215 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.411178 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffc6ceba-80a2-4722-a064-d409fe90fde4-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww\" (UID: \"ffc6ceba-80a2-4722-a064-d409fe90fde4\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" Oct 30 18:28:38.411401 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.411234 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc6ceba-80a2-4722-a064-d409fe90fde4-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww\" (UID: \"ffc6ceba-80a2-4722-a064-d409fe90fde4\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" Oct 30 18:28:38.411401 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.411303 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffc6ceba-80a2-4722-a064-d409fe90fde4-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww\" (UID: \"ffc6ceba-80a2-4722-a064-d409fe90fde4\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" Oct 30 18:28:38.411401 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.411338 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdckw\" (UniqueName: \"kubernetes.io/projected/ffc6ceba-80a2-4722-a064-d409fe90fde4-kube-api-access-hdckw\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww\" (UID: \"ffc6ceba-80a2-4722-a064-d409fe90fde4\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" Oct 30 18:28:38.411580 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.411449 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffc6ceba-80a2-4722-a064-d409fe90fde4-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww\" (UID: \"ffc6ceba-80a2-4722-a064-d409fe90fde4\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" Oct 30 18:28:38.411580 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.411501 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffc6ceba-80a2-4722-a064-d409fe90fde4-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww\" (UID: \"ffc6ceba-80a2-4722-a064-d409fe90fde4\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" Oct 30 18:28:38.411580 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.411529 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffc6ceba-80a2-4722-a064-d409fe90fde4-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww\" (UID: \"ffc6ceba-80a2-4722-a064-d409fe90fde4\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" Oct 30 18:28:38.411764 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.411592 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffc6ceba-80a2-4722-a064-d409fe90fde4-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww\" (UID: \"ffc6ceba-80a2-4722-a064-d409fe90fde4\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" Oct 30 18:28:38.411764 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.411658 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc6ceba-80a2-4722-a064-d409fe90fde4-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww\" (UID: \"ffc6ceba-80a2-4722-a064-d409fe90fde4\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" Oct 30 18:28:38.411764 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.411686 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc6ceba-80a2-4722-a064-d409fe90fde4-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww\" (UID: \"ffc6ceba-80a2-4722-a064-d409fe90fde4\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" Oct 30 18:28:38.411948 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.411878 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffc6ceba-80a2-4722-a064-d409fe90fde4-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww\" (UID: \"ffc6ceba-80a2-4722-a064-d409fe90fde4\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" Oct 30 18:28:38.412045 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.412019 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffc6ceba-80a2-4722-a064-d409fe90fde4-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww\" (UID: \"ffc6ceba-80a2-4722-a064-d409fe90fde4\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" Oct 30 18:28:38.412106 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.412049 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc6ceba-80a2-4722-a064-d409fe90fde4-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww\" (UID: \"ffc6ceba-80a2-4722-a064-d409fe90fde4\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" Oct 30 18:28:38.412106 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.412082 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffc6ceba-80a2-4722-a064-d409fe90fde4-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww\" (UID: \"ffc6ceba-80a2-4722-a064-d409fe90fde4\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" Oct 30 18:28:38.413904 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.413884 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffc6ceba-80a2-4722-a064-d409fe90fde4-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww\" (UID: \"ffc6ceba-80a2-4722-a064-d409fe90fde4\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" Oct 30 18:28:38.414168 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.414119 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffc6ceba-80a2-4722-a064-d409fe90fde4-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww\" (UID: \"ffc6ceba-80a2-4722-a064-d409fe90fde4\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" Oct 30 18:28:38.430805 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.430776 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdckw\" (UniqueName: \"kubernetes.io/projected/ffc6ceba-80a2-4722-a064-d409fe90fde4-kube-api-access-hdckw\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww\" (UID: \"ffc6ceba-80a2-4722-a064-d409fe90fde4\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" Oct 30 18:28:38.431095 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.431076 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffc6ceba-80a2-4722-a064-d409fe90fde4-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww\" (UID: \"ffc6ceba-80a2-4722-a064-d409fe90fde4\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" Oct 30 18:28:38.556095 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.555999 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" Oct 30 18:28:38.873519 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.873486 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww"] Oct 30 18:28:38.874287 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:28:38.874262 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffc6ceba_80a2_4722_a064_d409fe90fde4.slice/crio-1d2b2917757aa278bbac94c713841acfea88bf970aca9a5432848696a3df586e WatchSource:0}: Error finding container 1d2b2917757aa278bbac94c713841acfea88bf970aca9a5432848696a3df586e: Status 404 returned error can't find the container with id 1d2b2917757aa278bbac94c713841acfea88bf970aca9a5432848696a3df586e Oct 30 18:28:38.876893 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.876855 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:28:38.877003 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.876938 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:28:38.877003 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.876985 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:28:38.988784 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.988754 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" Oct 30 18:28:38.990428 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:38.990406 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" Oct 30 18:28:39.078299 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:39.078262 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" event={"ID":"ffc6ceba-80a2-4722-a064-d409fe90fde4","Type":"ContainerStarted","Data":"d84f0e5b32a1944e7fd2cb71934548e8c0524a97ef7823dcec34cd9299253a12"} Oct 30 18:28:39.078508 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:39.078325 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" event={"ID":"ffc6ceba-80a2-4722-a064-d409fe90fde4","Type":"ContainerStarted","Data":"1d2b2917757aa278bbac94c713841acfea88bf970aca9a5432848696a3df586e"} Oct 30 18:28:39.120981 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:39.120864 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" podStartSLOduration=1.120845696 podStartE2EDuration="1.120845696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:28:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:28:39.119056732 +0000 UTC m=+1816.209119072" watchObservedRunningTime="2026-10-30 18:28:39.120845696 +0000 UTC m=+1816.210908036" Oct 30 18:28:39.556532 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:39.556442 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" Oct 30 18:28:39.558385 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:39.558359 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 18:28:39.558517 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:39.558412 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" podUID="ffc6ceba-80a2-4722-a064-d409fe90fde4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 18:28:40.556769 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:40.556731 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 18:28:40.557406 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:40.557368 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" podUID="ffc6ceba-80a2-4722-a064-d409fe90fde4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 18:28:41.458057 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:41.458013 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-86bb6c4f5-q8xgr"] Oct 30 18:28:41.462221 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:41.462193 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-86bb6c4f5-q8xgr" Oct 30 18:28:41.493968 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:41.493937 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-44379a\"" Oct 30 18:28:41.511506 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:41.503593 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-86bb6c4f5-q8xgr"] Oct 30 18:28:41.518166 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:41.515558 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-44379a-dockercfg-l84wj\"" Oct 30 18:28:41.571716 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:41.571684 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 18:28:41.596351 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:41.596304 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" podUID="ffc6ceba-80a2-4722-a064-d409fe90fde4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 18:28:41.597473 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:41.597441 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lmch\" (UniqueName: \"kubernetes.io/projected/3f32ada3-60e5-44ce-92ad-cf17f0ce9184-kube-api-access-4lmch\") pod \"payload-pre-processing-e2e-ipp-b-44379a-86bb6c4f5-q8xgr\" (UID: \"3f32ada3-60e5-44ce-92ad-cf17f0ce9184\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-86bb6c4f5-q8xgr" Oct 30 18:28:41.597832 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:41.597811 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f32ada3-60e5-44ce-92ad-cf17f0ce9184-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-44379a-86bb6c4f5-q8xgr\" (UID: \"3f32ada3-60e5-44ce-92ad-cf17f0ce9184\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-86bb6c4f5-q8xgr" Oct 30 18:28:41.601682 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:41.601654 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-44379a-644486f9dd-jgqh6"] Oct 30 18:28:41.610932 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:41.610903 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-44379a-644486f9dd-jgqh6"] Oct 30 18:28:41.611109 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:41.611003 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-44379a-644486f9dd-jgqh6" Oct 30 18:28:41.699551 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:41.699496 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f32ada3-60e5-44ce-92ad-cf17f0ce9184-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-44379a-86bb6c4f5-q8xgr\" (UID: \"3f32ada3-60e5-44ce-92ad-cf17f0ce9184\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-86bb6c4f5-q8xgr" Oct 30 18:28:41.699784 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:41.699685 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2978a571-5e62-4f52-ac09-e628c08425ae-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-44379a-644486f9dd-jgqh6\" (UID: \"2978a571-5e62-4f52-ac09-e628c08425ae\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-44379a-644486f9dd-jgqh6" Oct 30 18:28:41.699784 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:41.699721 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lmch\" (UniqueName: \"kubernetes.io/projected/3f32ada3-60e5-44ce-92ad-cf17f0ce9184-kube-api-access-4lmch\") pod \"payload-pre-processing-e2e-ipp-b-44379a-86bb6c4f5-q8xgr\" (UID: \"3f32ada3-60e5-44ce-92ad-cf17f0ce9184\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-86bb6c4f5-q8xgr" Oct 30 18:28:41.699784 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:41.699756 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjzvk\" (UniqueName: \"kubernetes.io/projected/2978a571-5e62-4f52-ac09-e628c08425ae-kube-api-access-gjzvk\") pod \"payload-processing-e2e-ipp-b-44379a-644486f9dd-jgqh6\" (UID: \"2978a571-5e62-4f52-ac09-e628c08425ae\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-44379a-644486f9dd-jgqh6" Oct 30 18:28:41.700931 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:41.700898 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f32ada3-60e5-44ce-92ad-cf17f0ce9184-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-44379a-86bb6c4f5-q8xgr\" (UID: \"3f32ada3-60e5-44ce-92ad-cf17f0ce9184\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-86bb6c4f5-q8xgr" Oct 30 18:28:41.742234 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:41.723174 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lmch\" (UniqueName: \"kubernetes.io/projected/3f32ada3-60e5-44ce-92ad-cf17f0ce9184-kube-api-access-4lmch\") pod \"payload-pre-processing-e2e-ipp-b-44379a-86bb6c4f5-q8xgr\" (UID: \"3f32ada3-60e5-44ce-92ad-cf17f0ce9184\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-86bb6c4f5-q8xgr" Oct 30 18:28:41.801397 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:41.801211 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2978a571-5e62-4f52-ac09-e628c08425ae-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-44379a-644486f9dd-jgqh6\" (UID: \"2978a571-5e62-4f52-ac09-e628c08425ae\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-44379a-644486f9dd-jgqh6" Oct 30 18:28:41.801397 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:41.801255 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjzvk\" (UniqueName: \"kubernetes.io/projected/2978a571-5e62-4f52-ac09-e628c08425ae-kube-api-access-gjzvk\") pod \"payload-processing-e2e-ipp-b-44379a-644486f9dd-jgqh6\" (UID: \"2978a571-5e62-4f52-ac09-e628c08425ae\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-44379a-644486f9dd-jgqh6" Oct 30 18:28:41.801937 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:41.801906 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2978a571-5e62-4f52-ac09-e628c08425ae-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-44379a-644486f9dd-jgqh6\" (UID: \"2978a571-5e62-4f52-ac09-e628c08425ae\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-44379a-644486f9dd-jgqh6" Oct 30 18:28:41.813369 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:41.813343 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-86bb6c4f5-q8xgr" Oct 30 18:28:41.821721 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:41.821695 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjzvk\" (UniqueName: \"kubernetes.io/projected/2978a571-5e62-4f52-ac09-e628c08425ae-kube-api-access-gjzvk\") pod \"payload-processing-e2e-ipp-b-44379a-644486f9dd-jgqh6\" (UID: \"2978a571-5e62-4f52-ac09-e628c08425ae\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-44379a-644486f9dd-jgqh6" Oct 30 18:28:41.947697 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:41.947409 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-44379a-644486f9dd-jgqh6" Oct 30 18:28:42.171224 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:42.171189 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-86bb6c4f5-q8xgr"] Oct 30 18:28:42.176792 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:28:42.176765 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f32ada3_60e5_44ce_92ad_cf17f0ce9184.slice/crio-649547318ecd064a563e615876a1ac8332837a043e3eedfe39f48b9256a47e1b WatchSource:0}: Error finding container 649547318ecd064a563e615876a1ac8332837a043e3eedfe39f48b9256a47e1b: Status 404 returned error can't find the container with id 649547318ecd064a563e615876a1ac8332837a043e3eedfe39f48b9256a47e1b Oct 30 18:28:42.362785 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:42.362735 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-44379a-644486f9dd-jgqh6"] Oct 30 18:28:42.368176 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:28:42.368110 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2978a571_5e62_4f52_ac09_e628c08425ae.slice/crio-1d6260ad97c2ae2c6f7df80bdae06e54a229f63e3fe0c454b9f65a31dbbe4d53 WatchSource:0}: Error finding container 1d6260ad97c2ae2c6f7df80bdae06e54a229f63e3fe0c454b9f65a31dbbe4d53: Status 404 returned error can't find the container with id 1d6260ad97c2ae2c6f7df80bdae06e54a229f63e3fe0c454b9f65a31dbbe4d53 Oct 30 18:28:42.557255 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:42.556744 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 18:28:42.557255 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:42.556808 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" podUID="ffc6ceba-80a2-4722-a064-d409fe90fde4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 18:28:43.156577 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:43.156537 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-86bb6c4f5-q8xgr" event={"ID":"3f32ada3-60e5-44ce-92ad-cf17f0ce9184","Type":"ContainerStarted","Data":"11740e104265d05649201e670adb383d4f0560281193131578139fb09f21c789"} Oct 30 18:28:43.157215 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:43.157191 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-86bb6c4f5-q8xgr" event={"ID":"3f32ada3-60e5-44ce-92ad-cf17f0ce9184","Type":"ContainerStarted","Data":"649547318ecd064a563e615876a1ac8332837a043e3eedfe39f48b9256a47e1b"} Oct 30 18:28:43.158423 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:43.158400 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-86bb6c4f5-q8xgr" Oct 30 18:28:43.165522 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:43.165485 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-44379a-644486f9dd-jgqh6" event={"ID":"2978a571-5e62-4f52-ac09-e628c08425ae","Type":"ContainerStarted","Data":"ee31a06f60774a208127ed3526d0f80fdfee1c46871efcc21254414bc62b8245"} Oct 30 18:28:43.165728 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:43.165710 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-44379a-644486f9dd-jgqh6" event={"ID":"2978a571-5e62-4f52-ac09-e628c08425ae","Type":"ContainerStarted","Data":"1d6260ad97c2ae2c6f7df80bdae06e54a229f63e3fe0c454b9f65a31dbbe4d53"} Oct 30 18:28:43.166675 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:43.166652 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-44379a-644486f9dd-jgqh6" Oct 30 18:28:43.210683 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:43.210622 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-86bb6c4f5-q8xgr" podStartSLOduration=2.210601028 podStartE2EDuration="2.210601028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:28:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:28:43.205480456 +0000 UTC m=+1820.295542795" watchObservedRunningTime="2026-10-30 18:28:43.210601028 +0000 UTC m=+1820.300663369" Oct 30 18:28:43.265934 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:43.265855 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-44379a-644486f9dd-jgqh6" podStartSLOduration=2.265827393 podStartE2EDuration="2.265827393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:28:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:28:43.249600367 +0000 UTC m=+1820.339662707" watchObservedRunningTime="2026-10-30 18:28:43.265827393 +0000 UTC m=+1820.355889732" Oct 30 18:28:43.556683 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:43.556592 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 18:28:43.556937 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:43.556911 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" podUID="ffc6ceba-80a2-4722-a064-d409fe90fde4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 18:28:44.556751 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:44.556707 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 18:28:44.557299 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:44.556770 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" podUID="ffc6ceba-80a2-4722-a064-d409fe90fde4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 18:28:45.557146 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:45.557094 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 18:28:45.557579 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:45.557169 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" podUID="ffc6ceba-80a2-4722-a064-d409fe90fde4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 18:28:46.557239 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:46.557196 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 18:28:46.557648 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:46.557265 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" podUID="ffc6ceba-80a2-4722-a064-d409fe90fde4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 18:28:47.609306 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:28:47.609267 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:28:48.604319 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:48.604270 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:28:48.604514 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:48.604340 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" podUID="ffc6ceba-80a2-4722-a064-d409fe90fde4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:49.216099 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:49.216066 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" Oct 30 18:28:49.216585 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:49.216415 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" Oct 30 18:28:49.217398 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:49.217375 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" Oct 30 18:28:49.370688 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:49.370636 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-a4f3f0/praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk" podUID="2174f0e1-8712-4655-86ff-48ea2c635a96" containerName="main" containerID="cri-o://5923385cfc4f6ab536aee2f88c74805f23ca51bb8f050248453337b31fd884fe" gracePeriod=45 Oct 30 18:28:49.637889 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:49.637856 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-a4f3f0/praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk" Oct 30 18:28:49.697415 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:49.697371 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2174f0e1-8712-4655-86ff-48ea2c635a96-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2174f0e1-8712-4655-86ff-48ea2c635a96-model-cache\") pod \"2174f0e1-8712-4655-86ff-48ea2c635a96\" (UID: \"2174f0e1-8712-4655-86ff-48ea2c635a96\") " Oct 30 18:28:49.697613 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:49.697455 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2174f0e1-8712-4655-86ff-48ea2c635a96-home\" (UniqueName: \"kubernetes.io/empty-dir/2174f0e1-8712-4655-86ff-48ea2c635a96-home\") pod \"2174f0e1-8712-4655-86ff-48ea2c635a96\" (UID: \"2174f0e1-8712-4655-86ff-48ea2c635a96\") " Oct 30 18:28:49.697613 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:49.697490 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2174f0e1-8712-4655-86ff-48ea2c635a96-tls-certs\" (UniqueName: \"kubernetes.io/secret/2174f0e1-8712-4655-86ff-48ea2c635a96-tls-certs\") pod \"2174f0e1-8712-4655-86ff-48ea2c635a96\" (UID: \"2174f0e1-8712-4655-86ff-48ea2c635a96\") " Oct 30 18:28:49.697613 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:49.697518 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2174f0e1-8712-4655-86ff-48ea2c635a96-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2174f0e1-8712-4655-86ff-48ea2c635a96-dshm\") pod \"2174f0e1-8712-4655-86ff-48ea2c635a96\" (UID: \"2174f0e1-8712-4655-86ff-48ea2c635a96\") " Oct 30 18:28:49.697813 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:49.697637 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2174f0e1-8712-4655-86ff-48ea2c635a96-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2174f0e1-8712-4655-86ff-48ea2c635a96-tmp-dir\") pod \"2174f0e1-8712-4655-86ff-48ea2c635a96\" (UID: \"2174f0e1-8712-4655-86ff-48ea2c635a96\") " Oct 30 18:28:49.697813 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:49.697663 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2174f0e1-8712-4655-86ff-48ea2c635a96-model-cache" pod "2174f0e1-8712-4655-86ff-48ea2c635a96" (UID: "2174f0e1-8712-4655-86ff-48ea2c635a96"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:49.697813 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:49.697689 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2174f0e1-8712-4655-86ff-48ea2c635a96-kube-api-access-nwmqt\" (UniqueName: \"kubernetes.io/projected/2174f0e1-8712-4655-86ff-48ea2c635a96-kube-api-access-nwmqt\") pod \"2174f0e1-8712-4655-86ff-48ea2c635a96\" (UID: \"2174f0e1-8712-4655-86ff-48ea2c635a96\") " Oct 30 18:28:49.697981 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:49.697924 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2174f0e1-8712-4655-86ff-48ea2c635a96-tmp-dir" pod "2174f0e1-8712-4655-86ff-48ea2c635a96" (UID: "2174f0e1-8712-4655-86ff-48ea2c635a96"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:49.698341 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:49.698316 4442 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2174f0e1-8712-4655-86ff-48ea2c635a96-model-cache\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:28:49.698477 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:49.698345 4442 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2174f0e1-8712-4655-86ff-48ea2c635a96-tmp-dir\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:28:49.698477 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:49.698426 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2174f0e1-8712-4655-86ff-48ea2c635a96-home" pod "2174f0e1-8712-4655-86ff-48ea2c635a96" (UID: "2174f0e1-8712-4655-86ff-48ea2c635a96"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:49.700709 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:49.700686 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2174f0e1-8712-4655-86ff-48ea2c635a96-tls-certs" pod "2174f0e1-8712-4655-86ff-48ea2c635a96" (UID: "2174f0e1-8712-4655-86ff-48ea2c635a96"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:28:49.702425 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:49.702404 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2174f0e1-8712-4655-86ff-48ea2c635a96-dshm" pod "2174f0e1-8712-4655-86ff-48ea2c635a96" (UID: "2174f0e1-8712-4655-86ff-48ea2c635a96"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:49.702605 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:49.702582 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2174f0e1-8712-4655-86ff-48ea2c635a96-kube-api-access-nwmqt" pod "2174f0e1-8712-4655-86ff-48ea2c635a96" (UID: "2174f0e1-8712-4655-86ff-48ea2c635a96"). InnerVolumeSpecName "kube-api-access-nwmqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:49.799857 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:49.799760 4442 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2174f0e1-8712-4655-86ff-48ea2c635a96-home\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:28:49.799857 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:49.799794 4442 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2174f0e1-8712-4655-86ff-48ea2c635a96-tls-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:28:49.799857 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:49.799809 4442 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2174f0e1-8712-4655-86ff-48ea2c635a96-dshm\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:28:49.799857 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:49.799819 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwmqt\" (UniqueName: \"kubernetes.io/projected/2174f0e1-8712-4655-86ff-48ea2c635a96-kube-api-access-nwmqt\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:28:50.227481 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:50.227437 4442 generic.go:356] "Generic (PLEG): container finished" podID="2174f0e1-8712-4655-86ff-48ea2c635a96" containerID="5923385cfc4f6ab536aee2f88c74805f23ca51bb8f050248453337b31fd884fe" exitCode=0 Oct 30 18:28:50.227967 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:50.227501 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-a4f3f0/praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk" Oct 30 18:28:50.227967 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:50.227528 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-a4f3f0/praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk" event={"ID":"2174f0e1-8712-4655-86ff-48ea2c635a96","Type":"ContainerDied","Data":"5923385cfc4f6ab536aee2f88c74805f23ca51bb8f050248453337b31fd884fe"} Oct 30 18:28:50.227967 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:50.227600 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-a4f3f0/praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk" event={"ID":"2174f0e1-8712-4655-86ff-48ea2c635a96","Type":"ContainerDied","Data":"d9525a6f9b4983beeb14e6e6c1072e0b674a37f562761478bd15a2b991ba3e91"} Oct 30 18:28:50.227967 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:50.227623 4442 scope.go:122] "RemoveContainer" containerID="5923385cfc4f6ab536aee2f88c74805f23ca51bb8f050248453337b31fd884fe" Oct 30 18:28:50.241394 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:50.241194 4442 scope.go:122] "RemoveContainer" containerID="5923385cfc4f6ab536aee2f88c74805f23ca51bb8f050248453337b31fd884fe" Oct 30 18:28:50.242931 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:28:50.241609 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5923385cfc4f6ab536aee2f88c74805f23ca51bb8f050248453337b31fd884fe\": container with ID starting with 5923385cfc4f6ab536aee2f88c74805f23ca51bb8f050248453337b31fd884fe not found: ID does not exist" containerID="5923385cfc4f6ab536aee2f88c74805f23ca51bb8f050248453337b31fd884fe" Oct 30 18:28:50.242931 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:50.241675 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5923385cfc4f6ab536aee2f88c74805f23ca51bb8f050248453337b31fd884fe"} err="failed to get container status \"5923385cfc4f6ab536aee2f88c74805f23ca51bb8f050248453337b31fd884fe\": rpc error: code = NotFound desc = could not find container \"5923385cfc4f6ab536aee2f88c74805f23ca51bb8f050248453337b31fd884fe\": container with ID starting with 5923385cfc4f6ab536aee2f88c74805f23ca51bb8f050248453337b31fd884fe not found: ID does not exist" Oct 30 18:28:50.281289 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:50.281255 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-a4f3f0/praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk"] Oct 30 18:28:50.284431 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:50.284406 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-a4f3f0/praxis-route-a4f3f0-kserve-6f8bfdd469-qlbvk"] Oct 30 18:28:51.068333 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:51.068297 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2174f0e1-8712-4655-86ff-48ea2c635a96" path="/var/lib/kubelet/pods/2174f0e1-8712-4655-86ff-48ea2c635a96/volumes" Oct 30 18:28:53.294620 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:53.294583 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74"] Oct 30 18:28:53.295215 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:53.295201 4442 cpu_manager.go:414] "containerMap: removing container" podUID="2174f0e1-8712-4655-86ff-48ea2c635a96" containerName="main" Oct 30 18:28:53.295283 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:53.295217 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2174f0e1-8712-4655-86ff-48ea2c635a96" containerName="main" Oct 30 18:28:53.295325 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:53.295314 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="2174f0e1-8712-4655-86ff-48ea2c635a96" containerName="main" Oct 30 18:28:53.299818 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:53.299790 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" Oct 30 18:28:53.327184 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:53.327106 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74"] Oct 30 18:28:53.360669 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:53.360633 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a99c9a23-d65d-4950-a153-71c8decd10aa-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-npw74\" (UID: \"a99c9a23-d65d-4950-a153-71c8decd10aa\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" Oct 30 18:28:53.360898 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:53.360710 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a99c9a23-d65d-4950-a153-71c8decd10aa-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-npw74\" (UID: \"a99c9a23-d65d-4950-a153-71c8decd10aa\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" Oct 30 18:28:53.360898 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:53.360835 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a99c9a23-d65d-4950-a153-71c8decd10aa-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-npw74\" (UID: \"a99c9a23-d65d-4950-a153-71c8decd10aa\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" Oct 30 18:28:53.360898 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:53.360881 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a99c9a23-d65d-4950-a153-71c8decd10aa-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-npw74\" (UID: \"a99c9a23-d65d-4950-a153-71c8decd10aa\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" Oct 30 18:28:53.361047 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:53.361005 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kx95\" (UniqueName: \"kubernetes.io/projected/a99c9a23-d65d-4950-a153-71c8decd10aa-kube-api-access-4kx95\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-npw74\" (UID: \"a99c9a23-d65d-4950-a153-71c8decd10aa\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" Oct 30 18:28:53.361047 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:53.361045 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a99c9a23-d65d-4950-a153-71c8decd10aa-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-npw74\" (UID: \"a99c9a23-d65d-4950-a153-71c8decd10aa\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" Oct 30 18:28:53.361280 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:53.361242 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a99c9a23-d65d-4950-a153-71c8decd10aa-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-npw74\" (UID: \"a99c9a23-d65d-4950-a153-71c8decd10aa\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" Oct 30 18:28:53.361348 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:53.361311 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a99c9a23-d65d-4950-a153-71c8decd10aa-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-npw74\" (UID: \"a99c9a23-d65d-4950-a153-71c8decd10aa\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" Oct 30 18:28:53.361404 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:53.361382 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a99c9a23-d65d-4950-a153-71c8decd10aa-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-npw74\" (UID: \"a99c9a23-d65d-4950-a153-71c8decd10aa\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" Oct 30 18:28:53.462068 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:53.462028 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kx95\" (UniqueName: \"kubernetes.io/projected/a99c9a23-d65d-4950-a153-71c8decd10aa-kube-api-access-4kx95\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-npw74\" (UID: \"a99c9a23-d65d-4950-a153-71c8decd10aa\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" Oct 30 18:28:53.462300 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:53.462083 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a99c9a23-d65d-4950-a153-71c8decd10aa-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-npw74\" (UID: \"a99c9a23-d65d-4950-a153-71c8decd10aa\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" Oct 30 18:28:53.462300 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:53.462167 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a99c9a23-d65d-4950-a153-71c8decd10aa-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-npw74\" (UID: \"a99c9a23-d65d-4950-a153-71c8decd10aa\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" Oct 30 18:28:53.462300 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:53.462205 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a99c9a23-d65d-4950-a153-71c8decd10aa-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-npw74\" (UID: \"a99c9a23-d65d-4950-a153-71c8decd10aa\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" Oct 30 18:28:53.462300 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:53.462269 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a99c9a23-d65d-4950-a153-71c8decd10aa-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-npw74\" (UID: \"a99c9a23-d65d-4950-a153-71c8decd10aa\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" Oct 30 18:28:53.462548 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:53.462311 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a99c9a23-d65d-4950-a153-71c8decd10aa-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-npw74\" (UID: \"a99c9a23-d65d-4950-a153-71c8decd10aa\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" Oct 30 18:28:53.462548 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:53.462373 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a99c9a23-d65d-4950-a153-71c8decd10aa-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-npw74\" (UID: \"a99c9a23-d65d-4950-a153-71c8decd10aa\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" Oct 30 18:28:53.462548 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:53.462453 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a99c9a23-d65d-4950-a153-71c8decd10aa-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-npw74\" (UID: \"a99c9a23-d65d-4950-a153-71c8decd10aa\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" Oct 30 18:28:53.462548 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:53.462488 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a99c9a23-d65d-4950-a153-71c8decd10aa-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-npw74\" (UID: \"a99c9a23-d65d-4950-a153-71c8decd10aa\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" Oct 30 18:28:53.462751 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:53.462597 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a99c9a23-d65d-4950-a153-71c8decd10aa-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-npw74\" (UID: \"a99c9a23-d65d-4950-a153-71c8decd10aa\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" Oct 30 18:28:53.462838 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:53.462813 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a99c9a23-d65d-4950-a153-71c8decd10aa-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-npw74\" (UID: \"a99c9a23-d65d-4950-a153-71c8decd10aa\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" Oct 30 18:28:53.463488 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:53.463460 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a99c9a23-d65d-4950-a153-71c8decd10aa-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-npw74\" (UID: \"a99c9a23-d65d-4950-a153-71c8decd10aa\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" Oct 30 18:28:53.463734 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:53.463713 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a99c9a23-d65d-4950-a153-71c8decd10aa-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-npw74\" (UID: \"a99c9a23-d65d-4950-a153-71c8decd10aa\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" Oct 30 18:28:53.463954 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:53.463933 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a99c9a23-d65d-4950-a153-71c8decd10aa-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-npw74\" (UID: \"a99c9a23-d65d-4950-a153-71c8decd10aa\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" Oct 30 18:28:53.465066 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:53.465038 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a99c9a23-d65d-4950-a153-71c8decd10aa-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-npw74\" (UID: \"a99c9a23-d65d-4950-a153-71c8decd10aa\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" Oct 30 18:28:53.465323 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:53.465304 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a99c9a23-d65d-4950-a153-71c8decd10aa-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-npw74\" (UID: \"a99c9a23-d65d-4950-a153-71c8decd10aa\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" Oct 30 18:28:53.482688 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:53.482653 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a99c9a23-d65d-4950-a153-71c8decd10aa-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-npw74\" (UID: \"a99c9a23-d65d-4950-a153-71c8decd10aa\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" Oct 30 18:28:53.482822 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:53.482783 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kx95\" (UniqueName: \"kubernetes.io/projected/a99c9a23-d65d-4950-a153-71c8decd10aa-kube-api-access-4kx95\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-npw74\" (UID: \"a99c9a23-d65d-4950-a153-71c8decd10aa\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" Oct 30 18:28:53.615529 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:53.615504 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" Oct 30 18:28:53.931070 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:53.931042 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74"] Oct 30 18:28:53.932432 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:28:53.932391 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda99c9a23_d65d_4950_a153_71c8decd10aa.slice/crio-50bc0a527445704fc6d26349e5b57e39cb7db470f6dc938111cfe70819cad4a7 WatchSource:0}: Error finding container 50bc0a527445704fc6d26349e5b57e39cb7db470f6dc938111cfe70819cad4a7: Status 404 returned error can't find the container with id 50bc0a527445704fc6d26349e5b57e39cb7db470f6dc938111cfe70819cad4a7 Oct 30 18:28:53.935886 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:53.935845 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:28:53.935975 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:53.935934 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:28:53.936014 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:53.935981 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:28:54.274041 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:54.273952 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" event={"ID":"a99c9a23-d65d-4950-a153-71c8decd10aa","Type":"ContainerStarted","Data":"b2dfe6d37d9b6174fec9511a906965354a94f74e5e90d431e1958e6971c118b5"} Oct 30 18:28:54.274324 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:54.274302 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" event={"ID":"a99c9a23-d65d-4950-a153-71c8decd10aa","Type":"ContainerStarted","Data":"50bc0a527445704fc6d26349e5b57e39cb7db470f6dc938111cfe70819cad4a7"} Oct 30 18:28:54.320446 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:54.320374 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" podStartSLOduration=1.320351625 podStartE2EDuration="1.320351625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:28:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:28:54.31672806 +0000 UTC m=+1831.406790424" watchObservedRunningTime="2026-10-30 18:28:54.320351625 +0000 UTC m=+1831.410413964" Oct 30 18:28:54.616460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:54.616436 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" Oct 30 18:28:55.185638 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:55.185612 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-86bb6c4f5-q8xgr" Oct 30 18:28:55.185860 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:55.185680 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-44379a-644486f9dd-jgqh6" Oct 30 18:28:55.617348 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:55.617314 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 18:28:55.617777 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:55.617366 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" podUID="a99c9a23-d65d-4950-a153-71c8decd10aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 18:28:56.615868 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:56.615835 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 18:28:56.616081 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:56.615894 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" podUID="a99c9a23-d65d-4950-a153-71c8decd10aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 18:28:57.616179 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:57.616109 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 18:28:57.616768 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:57.616734 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" podUID="a99c9a23-d65d-4950-a153-71c8decd10aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 18:28:58.616532 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:58.616488 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 18:28:58.616972 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:58.616562 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" podUID="a99c9a23-d65d-4950-a153-71c8decd10aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 18:28:59.615955 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:59.615920 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 18:28:59.616203 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:59.615975 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" podUID="a99c9a23-d65d-4950-a153-71c8decd10aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 18:28:59.775893 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:28:59.775861 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-44379a-644486f9dd-jgqh6_2978a571-5e62-4f52-ac09-e628c08425ae/payload-processing/0.log" Oct 30 18:29:00.616625 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:00.616591 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 18:29:00.616851 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:00.616639 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" podUID="a99c9a23-d65d-4950-a153-71c8decd10aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 18:29:01.663317 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:01.663271 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" Oct 30 18:29:01.663998 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:01.663945 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" Oct 30 18:29:01.664949 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:01.664929 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" Oct 30 18:29:17.568458 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:29:17.568374 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:29:21.707752 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:21.707718 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2"] Oct 30 18:29:21.708416 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:21.707986 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2" podUID="4ed27292-34ae-4a6a-86cb-dbf81d903a49" containerName="maas-api" containerID="cri-o://49eae0041851137c3ea92be92e86b5c3f4d066f21827ce2652b260ffb0900216" gracePeriod=30 Oct 30 18:29:22.015243 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:22.015215 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2" Oct 30 18:29:22.021737 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:22.021712 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ed27292-34ae-4a6a-86cb-dbf81d903a49-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ed27292-34ae-4a6a-86cb-dbf81d903a49-metrics-certs\") pod \"4ed27292-34ae-4a6a-86cb-dbf81d903a49\" (UID: \"4ed27292-34ae-4a6a-86cb-dbf81d903a49\") " Oct 30 18:29:22.021909 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:22.021767 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ed27292-34ae-4a6a-86cb-dbf81d903a49-kube-api-access-454v5\" (UniqueName: \"kubernetes.io/projected/4ed27292-34ae-4a6a-86cb-dbf81d903a49-kube-api-access-454v5\") pod \"4ed27292-34ae-4a6a-86cb-dbf81d903a49\" (UID: \"4ed27292-34ae-4a6a-86cb-dbf81d903a49\") " Oct 30 18:29:22.021909 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:22.021786 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ed27292-34ae-4a6a-86cb-dbf81d903a49-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ed27292-34ae-4a6a-86cb-dbf81d903a49-maas-api-tls\") pod \"4ed27292-34ae-4a6a-86cb-dbf81d903a49\" (UID: \"4ed27292-34ae-4a6a-86cb-dbf81d903a49\") " Oct 30 18:29:22.021909 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:22.021872 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ed27292-34ae-4a6a-86cb-dbf81d903a49-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ed27292-34ae-4a6a-86cb-dbf81d903a49-openshift-service-ca\") pod \"4ed27292-34ae-4a6a-86cb-dbf81d903a49\" (UID: \"4ed27292-34ae-4a6a-86cb-dbf81d903a49\") " Oct 30 18:29:22.022402 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:22.022382 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ed27292-34ae-4a6a-86cb-dbf81d903a49-openshift-service-ca" pod "4ed27292-34ae-4a6a-86cb-dbf81d903a49" (UID: "4ed27292-34ae-4a6a-86cb-dbf81d903a49"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:29:22.024257 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:22.024229 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ed27292-34ae-4a6a-86cb-dbf81d903a49-maas-api-tls" pod "4ed27292-34ae-4a6a-86cb-dbf81d903a49" (UID: "4ed27292-34ae-4a6a-86cb-dbf81d903a49"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:29:22.024952 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:22.024928 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ed27292-34ae-4a6a-86cb-dbf81d903a49-kube-api-access-454v5" pod "4ed27292-34ae-4a6a-86cb-dbf81d903a49" (UID: "4ed27292-34ae-4a6a-86cb-dbf81d903a49"). InnerVolumeSpecName "kube-api-access-454v5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:22.025085 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:22.024978 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ed27292-34ae-4a6a-86cb-dbf81d903a49-metrics-certs" pod "4ed27292-34ae-4a6a-86cb-dbf81d903a49" (UID: "4ed27292-34ae-4a6a-86cb-dbf81d903a49"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:29:22.123650 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:22.123617 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ed27292-34ae-4a6a-86cb-dbf81d903a49-openshift-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:29:22.123650 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:22.123648 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ed27292-34ae-4a6a-86cb-dbf81d903a49-metrics-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:29:22.123650 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:22.123658 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-454v5\" (UniqueName: \"kubernetes.io/projected/4ed27292-34ae-4a6a-86cb-dbf81d903a49-kube-api-access-454v5\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:29:22.123899 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:22.123667 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ed27292-34ae-4a6a-86cb-dbf81d903a49-maas-api-tls\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:29:22.553195 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:22.553145 4442 generic.go:356] "Generic (PLEG): container finished" podID="4ed27292-34ae-4a6a-86cb-dbf81d903a49" containerID="49eae0041851137c3ea92be92e86b5c3f4d066f21827ce2652b260ffb0900216" exitCode=0 Oct 30 18:29:22.553195 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:22.553190 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2" event={"ID":"4ed27292-34ae-4a6a-86cb-dbf81d903a49","Type":"ContainerDied","Data":"49eae0041851137c3ea92be92e86b5c3f4d066f21827ce2652b260ffb0900216"} Oct 30 18:29:22.553443 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:22.553214 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2" Oct 30 18:29:22.553443 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:22.553290 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2" event={"ID":"4ed27292-34ae-4a6a-86cb-dbf81d903a49","Type":"ContainerDied","Data":"6f7ef4c0250a9af4b63c60ead806781d58eca056b4ed9d2a9d72649c67edc301"} Oct 30 18:29:22.553443 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:22.553321 4442 scope.go:122] "RemoveContainer" containerID="49eae0041851137c3ea92be92e86b5c3f4d066f21827ce2652b260ffb0900216" Oct 30 18:29:22.571908 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:22.567916 4442 scope.go:122] "RemoveContainer" containerID="b7dd920b8a2155d4ee12d9c2677502e38f98dceda09eb8f0edc0c9ad6cd78368" Oct 30 18:29:22.582028 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:22.582004 4442 scope.go:122] "RemoveContainer" containerID="49eae0041851137c3ea92be92e86b5c3f4d066f21827ce2652b260ffb0900216" Oct 30 18:29:22.582504 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:29:22.582476 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49eae0041851137c3ea92be92e86b5c3f4d066f21827ce2652b260ffb0900216\": container with ID starting with 49eae0041851137c3ea92be92e86b5c3f4d066f21827ce2652b260ffb0900216 not found: ID does not exist" containerID="49eae0041851137c3ea92be92e86b5c3f4d066f21827ce2652b260ffb0900216" Oct 30 18:29:22.582616 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:22.582517 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49eae0041851137c3ea92be92e86b5c3f4d066f21827ce2652b260ffb0900216"} err="failed to get container status \"49eae0041851137c3ea92be92e86b5c3f4d066f21827ce2652b260ffb0900216\": rpc error: code = NotFound desc = could not find container \"49eae0041851137c3ea92be92e86b5c3f4d066f21827ce2652b260ffb0900216\": container with ID starting with 49eae0041851137c3ea92be92e86b5c3f4d066f21827ce2652b260ffb0900216 not found: ID does not exist" Oct 30 18:29:22.582616 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:22.582536 4442 scope.go:122] "RemoveContainer" containerID="b7dd920b8a2155d4ee12d9c2677502e38f98dceda09eb8f0edc0c9ad6cd78368" Oct 30 18:29:22.582895 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:29:22.582871 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7dd920b8a2155d4ee12d9c2677502e38f98dceda09eb8f0edc0c9ad6cd78368\": container with ID starting with b7dd920b8a2155d4ee12d9c2677502e38f98dceda09eb8f0edc0c9ad6cd78368 not found: ID does not exist" containerID="b7dd920b8a2155d4ee12d9c2677502e38f98dceda09eb8f0edc0c9ad6cd78368" Oct 30 18:29:22.582990 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:22.582905 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7dd920b8a2155d4ee12d9c2677502e38f98dceda09eb8f0edc0c9ad6cd78368"} err="failed to get container status \"b7dd920b8a2155d4ee12d9c2677502e38f98dceda09eb8f0edc0c9ad6cd78368\": rpc error: code = NotFound desc = could not find container \"b7dd920b8a2155d4ee12d9c2677502e38f98dceda09eb8f0edc0c9ad6cd78368\": container with ID starting with b7dd920b8a2155d4ee12d9c2677502e38f98dceda09eb8f0edc0c9ad6cd78368 not found: ID does not exist" Oct 30 18:29:22.610870 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:22.610845 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2"] Oct 30 18:29:22.622272 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:22.622244 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e9b146-689888db6b-b9nc2"] Oct 30 18:29:23.067478 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:23.067448 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ed27292-34ae-4a6a-86cb-dbf81d903a49" path="/var/lib/kubelet/pods/4ed27292-34ae-4a6a-86cb-dbf81d903a49/volumes" Oct 30 18:29:23.267605 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:23.267573 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d"] Oct 30 18:29:23.268164 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:23.268126 4442 cpu_manager.go:414] "containerMap: removing container" podUID="4ed27292-34ae-4a6a-86cb-dbf81d903a49" containerName="maas-api" Oct 30 18:29:23.268164 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:23.268159 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ed27292-34ae-4a6a-86cb-dbf81d903a49" containerName="maas-api" Oct 30 18:29:23.268252 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:23.268195 4442 cpu_manager.go:414] "containerMap: removing container" podUID="4ed27292-34ae-4a6a-86cb-dbf81d903a49" containerName="maas-api" Oct 30 18:29:23.268252 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:23.268200 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ed27292-34ae-4a6a-86cb-dbf81d903a49" containerName="maas-api" Oct 30 18:29:23.268317 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:23.268288 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ed27292-34ae-4a6a-86cb-dbf81d903a49" containerName="maas-api" Oct 30 18:29:23.273493 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:23.273465 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" Oct 30 18:29:23.300304 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:23.300270 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d"] Oct 30 18:29:23.348213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:23.347739 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4e00bfc-fc1d-46ce-946d-969438944b7e-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d\" (UID: \"f4e00bfc-fc1d-46ce-946d-969438944b7e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" Oct 30 18:29:23.348213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:23.347826 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4e00bfc-fc1d-46ce-946d-969438944b7e-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d\" (UID: \"f4e00bfc-fc1d-46ce-946d-969438944b7e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" Oct 30 18:29:23.348213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:23.347905 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4e00bfc-fc1d-46ce-946d-969438944b7e-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d\" (UID: \"f4e00bfc-fc1d-46ce-946d-969438944b7e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" Oct 30 18:29:23.348213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:23.347982 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4e00bfc-fc1d-46ce-946d-969438944b7e-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d\" (UID: \"f4e00bfc-fc1d-46ce-946d-969438944b7e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" Oct 30 18:29:23.348213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:23.348030 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4e00bfc-fc1d-46ce-946d-969438944b7e-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d\" (UID: \"f4e00bfc-fc1d-46ce-946d-969438944b7e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" Oct 30 18:29:23.348213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:23.348061 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4e00bfc-fc1d-46ce-946d-969438944b7e-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d\" (UID: \"f4e00bfc-fc1d-46ce-946d-969438944b7e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" Oct 30 18:29:23.348213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:23.348121 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4e00bfc-fc1d-46ce-946d-969438944b7e-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d\" (UID: \"f4e00bfc-fc1d-46ce-946d-969438944b7e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" Oct 30 18:29:23.348213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:23.348203 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4e00bfc-fc1d-46ce-946d-969438944b7e-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d\" (UID: \"f4e00bfc-fc1d-46ce-946d-969438944b7e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" Oct 30 18:29:23.348760 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:23.348241 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw4fw\" (UniqueName: \"kubernetes.io/projected/f4e00bfc-fc1d-46ce-946d-969438944b7e-kube-api-access-tw4fw\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d\" (UID: \"f4e00bfc-fc1d-46ce-946d-969438944b7e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" Oct 30 18:29:23.449250 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:23.449208 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4e00bfc-fc1d-46ce-946d-969438944b7e-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d\" (UID: \"f4e00bfc-fc1d-46ce-946d-969438944b7e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" Oct 30 18:29:23.449474 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:23.449267 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4e00bfc-fc1d-46ce-946d-969438944b7e-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d\" (UID: \"f4e00bfc-fc1d-46ce-946d-969438944b7e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" Oct 30 18:29:23.449474 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:23.449319 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4e00bfc-fc1d-46ce-946d-969438944b7e-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d\" (UID: \"f4e00bfc-fc1d-46ce-946d-969438944b7e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" Oct 30 18:29:23.449474 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:23.449374 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4e00bfc-fc1d-46ce-946d-969438944b7e-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d\" (UID: \"f4e00bfc-fc1d-46ce-946d-969438944b7e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" Oct 30 18:29:23.449746 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:23.449719 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4e00bfc-fc1d-46ce-946d-969438944b7e-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d\" (UID: \"f4e00bfc-fc1d-46ce-946d-969438944b7e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" Oct 30 18:29:23.449838 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:23.449765 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4e00bfc-fc1d-46ce-946d-969438944b7e-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d\" (UID: \"f4e00bfc-fc1d-46ce-946d-969438944b7e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" Oct 30 18:29:23.449838 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:23.449803 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4e00bfc-fc1d-46ce-946d-969438944b7e-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d\" (UID: \"f4e00bfc-fc1d-46ce-946d-969438944b7e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" Oct 30 18:29:23.449838 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:23.449807 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4e00bfc-fc1d-46ce-946d-969438944b7e-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d\" (UID: \"f4e00bfc-fc1d-46ce-946d-969438944b7e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" Oct 30 18:29:23.450009 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:23.449874 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4e00bfc-fc1d-46ce-946d-969438944b7e-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d\" (UID: \"f4e00bfc-fc1d-46ce-946d-969438944b7e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" Oct 30 18:29:23.450009 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:23.449933 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4e00bfc-fc1d-46ce-946d-969438944b7e-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d\" (UID: \"f4e00bfc-fc1d-46ce-946d-969438944b7e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" Oct 30 18:29:23.450009 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:23.449970 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw4fw\" (UniqueName: \"kubernetes.io/projected/f4e00bfc-fc1d-46ce-946d-969438944b7e-kube-api-access-tw4fw\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d\" (UID: \"f4e00bfc-fc1d-46ce-946d-969438944b7e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" Oct 30 18:29:23.450182 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:23.450116 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4e00bfc-fc1d-46ce-946d-969438944b7e-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d\" (UID: \"f4e00bfc-fc1d-46ce-946d-969438944b7e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" Oct 30 18:29:23.450182 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:23.450115 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4e00bfc-fc1d-46ce-946d-969438944b7e-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d\" (UID: \"f4e00bfc-fc1d-46ce-946d-969438944b7e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" Oct 30 18:29:23.450430 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:23.450400 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4e00bfc-fc1d-46ce-946d-969438944b7e-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d\" (UID: \"f4e00bfc-fc1d-46ce-946d-969438944b7e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" Oct 30 18:29:23.453315 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:23.453290 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4e00bfc-fc1d-46ce-946d-969438944b7e-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d\" (UID: \"f4e00bfc-fc1d-46ce-946d-969438944b7e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" Oct 30 18:29:23.453490 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:23.453472 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4e00bfc-fc1d-46ce-946d-969438944b7e-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d\" (UID: \"f4e00bfc-fc1d-46ce-946d-969438944b7e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" Oct 30 18:29:23.471342 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:23.471311 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4e00bfc-fc1d-46ce-946d-969438944b7e-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d\" (UID: \"f4e00bfc-fc1d-46ce-946d-969438944b7e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" Oct 30 18:29:23.472433 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:23.472404 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw4fw\" (UniqueName: \"kubernetes.io/projected/f4e00bfc-fc1d-46ce-946d-969438944b7e-kube-api-access-tw4fw\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d\" (UID: \"f4e00bfc-fc1d-46ce-946d-969438944b7e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" Oct 30 18:29:23.603072 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:23.602980 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" Oct 30 18:29:23.938340 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:29:23.938307 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4e00bfc_fc1d_46ce_946d_969438944b7e.slice/crio-788eabc9242cdb66560e32d9ddd75e66c4c71e3c1cfc8264fd5acedba13c0673 WatchSource:0}: Error finding container 788eabc9242cdb66560e32d9ddd75e66c4c71e3c1cfc8264fd5acedba13c0673: Status 404 returned error can't find the container with id 788eabc9242cdb66560e32d9ddd75e66c4c71e3c1cfc8264fd5acedba13c0673 Oct 30 18:29:23.939395 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:23.939284 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d"] Oct 30 18:29:23.941827 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:23.941783 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:29:23.941930 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:23.941875 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:29:23.941930 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:23.941923 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:29:24.598182 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:24.594397 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" event={"ID":"f4e00bfc-fc1d-46ce-946d-969438944b7e","Type":"ContainerStarted","Data":"e613121c7326daf440ea98122f0e0b43d85fb440d0946585ad4e7f3ea4b19e97"} Oct 30 18:29:24.598182 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:24.594471 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" event={"ID":"f4e00bfc-fc1d-46ce-946d-969438944b7e","Type":"ContainerStarted","Data":"788eabc9242cdb66560e32d9ddd75e66c4c71e3c1cfc8264fd5acedba13c0673"} Oct 30 18:29:24.604198 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:24.603569 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" Oct 30 18:29:24.650994 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:24.650937 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" podStartSLOduration=1.650920262 podStartE2EDuration="1.650920262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:29:24.647116583 +0000 UTC m=+1861.737178926" watchObservedRunningTime="2026-10-30 18:29:24.650920262 +0000 UTC m=+1861.740982601" Oct 30 18:29:25.183626 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:25.183598 4442 scope.go:122] "RemoveContainer" containerID="b253673626709c557f82295e22ec21f30949edbf3374f8606ddda1135c941b05" Oct 30 18:29:25.614983 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:25.614944 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 30 18:29:25.615393 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:25.615009 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" podUID="f4e00bfc-fc1d-46ce-946d-969438944b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 30 18:29:26.604187 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:26.604121 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 30 18:29:26.604409 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:26.604209 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" podUID="f4e00bfc-fc1d-46ce-946d-969438944b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 30 18:29:27.605612 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:27.605574 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 30 18:29:27.606267 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:27.606235 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" podUID="f4e00bfc-fc1d-46ce-946d-969438944b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 30 18:29:28.603574 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:28.603537 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 30 18:29:28.603782 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:28.603599 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" podUID="f4e00bfc-fc1d-46ce-946d-969438944b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 30 18:29:29.604324 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:29.604285 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 30 18:29:29.604762 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:29.604357 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" podUID="f4e00bfc-fc1d-46ce-946d-969438944b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 30 18:29:30.604396 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:30.604357 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 30 18:29:30.604809 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:30.604427 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" podUID="f4e00bfc-fc1d-46ce-946d-969438944b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 30 18:29:31.643951 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:31.643919 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" Oct 30 18:29:31.644489 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:31.644459 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" Oct 30 18:29:31.645246 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:31.645225 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" Oct 30 18:29:38.209944 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:38.209910 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn"] Oct 30 18:29:38.211045 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:38.211019 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ed27292-34ae-4a6a-86cb-dbf81d903a49" containerName="maas-api" Oct 30 18:29:38.216641 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:38.216622 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" Oct 30 18:29:38.239696 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:38.239665 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn"] Oct 30 18:29:38.410188 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:38.410108 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57w9x\" (UniqueName: \"kubernetes.io/projected/a122a97d-09d1-485a-902c-2f7832deab2c-kube-api-access-57w9x\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn\" (UID: \"a122a97d-09d1-485a-902c-2f7832deab2c\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" Oct 30 18:29:38.410434 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:38.410313 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a122a97d-09d1-485a-902c-2f7832deab2c-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn\" (UID: \"a122a97d-09d1-485a-902c-2f7832deab2c\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" Oct 30 18:29:38.410434 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:38.410350 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a122a97d-09d1-485a-902c-2f7832deab2c-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn\" (UID: \"a122a97d-09d1-485a-902c-2f7832deab2c\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" Oct 30 18:29:38.410556 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:38.410434 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a122a97d-09d1-485a-902c-2f7832deab2c-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn\" (UID: \"a122a97d-09d1-485a-902c-2f7832deab2c\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" Oct 30 18:29:38.410627 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:38.410576 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a122a97d-09d1-485a-902c-2f7832deab2c-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn\" (UID: \"a122a97d-09d1-485a-902c-2f7832deab2c\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" Oct 30 18:29:38.410627 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:38.410611 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a122a97d-09d1-485a-902c-2f7832deab2c-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn\" (UID: \"a122a97d-09d1-485a-902c-2f7832deab2c\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" Oct 30 18:29:38.410747 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:38.410640 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a122a97d-09d1-485a-902c-2f7832deab2c-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn\" (UID: \"a122a97d-09d1-485a-902c-2f7832deab2c\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" Oct 30 18:29:38.410806 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:38.410758 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a122a97d-09d1-485a-902c-2f7832deab2c-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn\" (UID: \"a122a97d-09d1-485a-902c-2f7832deab2c\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" Oct 30 18:29:38.410864 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:38.410824 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a122a97d-09d1-485a-902c-2f7832deab2c-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn\" (UID: \"a122a97d-09d1-485a-902c-2f7832deab2c\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" Oct 30 18:29:38.511803 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:38.511715 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a122a97d-09d1-485a-902c-2f7832deab2c-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn\" (UID: \"a122a97d-09d1-485a-902c-2f7832deab2c\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" Oct 30 18:29:38.511803 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:38.511759 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a122a97d-09d1-485a-902c-2f7832deab2c-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn\" (UID: \"a122a97d-09d1-485a-902c-2f7832deab2c\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" Oct 30 18:29:38.512039 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:38.511806 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a122a97d-09d1-485a-902c-2f7832deab2c-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn\" (UID: \"a122a97d-09d1-485a-902c-2f7832deab2c\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" Oct 30 18:29:38.512039 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:38.511888 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a122a97d-09d1-485a-902c-2f7832deab2c-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn\" (UID: \"a122a97d-09d1-485a-902c-2f7832deab2c\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" Oct 30 18:29:38.512039 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:38.511911 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a122a97d-09d1-485a-902c-2f7832deab2c-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn\" (UID: \"a122a97d-09d1-485a-902c-2f7832deab2c\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" Oct 30 18:29:38.512039 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:38.511939 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a122a97d-09d1-485a-902c-2f7832deab2c-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn\" (UID: \"a122a97d-09d1-485a-902c-2f7832deab2c\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" Oct 30 18:29:38.512039 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:38.511970 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a122a97d-09d1-485a-902c-2f7832deab2c-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn\" (UID: \"a122a97d-09d1-485a-902c-2f7832deab2c\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" Oct 30 18:29:38.512039 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:38.512015 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a122a97d-09d1-485a-902c-2f7832deab2c-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn\" (UID: \"a122a97d-09d1-485a-902c-2f7832deab2c\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" Oct 30 18:29:38.512381 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:38.512157 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57w9x\" (UniqueName: \"kubernetes.io/projected/a122a97d-09d1-485a-902c-2f7832deab2c-kube-api-access-57w9x\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn\" (UID: \"a122a97d-09d1-485a-902c-2f7832deab2c\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" Oct 30 18:29:38.512381 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:38.512256 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a122a97d-09d1-485a-902c-2f7832deab2c-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn\" (UID: \"a122a97d-09d1-485a-902c-2f7832deab2c\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" Oct 30 18:29:38.512381 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:38.512278 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a122a97d-09d1-485a-902c-2f7832deab2c-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn\" (UID: \"a122a97d-09d1-485a-902c-2f7832deab2c\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" Oct 30 18:29:38.512604 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:38.512580 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a122a97d-09d1-485a-902c-2f7832deab2c-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn\" (UID: \"a122a97d-09d1-485a-902c-2f7832deab2c\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" Oct 30 18:29:38.512754 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:38.512734 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a122a97d-09d1-485a-902c-2f7832deab2c-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn\" (UID: \"a122a97d-09d1-485a-902c-2f7832deab2c\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" Oct 30 18:29:38.512831 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:38.512769 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a122a97d-09d1-485a-902c-2f7832deab2c-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn\" (UID: \"a122a97d-09d1-485a-902c-2f7832deab2c\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" Oct 30 18:29:38.514251 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:38.514225 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a122a97d-09d1-485a-902c-2f7832deab2c-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn\" (UID: \"a122a97d-09d1-485a-902c-2f7832deab2c\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" Oct 30 18:29:38.514638 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:38.514618 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a122a97d-09d1-485a-902c-2f7832deab2c-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn\" (UID: \"a122a97d-09d1-485a-902c-2f7832deab2c\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" Oct 30 18:29:38.532048 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:38.532024 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a122a97d-09d1-485a-902c-2f7832deab2c-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn\" (UID: \"a122a97d-09d1-485a-902c-2f7832deab2c\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" Oct 30 18:29:38.532363 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:38.532339 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57w9x\" (UniqueName: \"kubernetes.io/projected/a122a97d-09d1-485a-902c-2f7832deab2c-kube-api-access-57w9x\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn\" (UID: \"a122a97d-09d1-485a-902c-2f7832deab2c\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" Oct 30 18:29:38.533943 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:38.533920 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" Oct 30 18:29:38.847146 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:29:38.847101 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda122a97d_09d1_485a_902c_2f7832deab2c.slice/crio-fefb3aafe0c28ba23f79f7f2f935a13a8e51a42788b4f0caf8c041fecffea0e4 WatchSource:0}: Error finding container fefb3aafe0c28ba23f79f7f2f935a13a8e51a42788b4f0caf8c041fecffea0e4: Status 404 returned error can't find the container with id fefb3aafe0c28ba23f79f7f2f935a13a8e51a42788b4f0caf8c041fecffea0e4 Oct 30 18:29:38.848540 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:38.848511 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn"] Oct 30 18:29:38.850076 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:38.850037 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:29:38.850198 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:38.850124 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:29:38.850198 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:38.850191 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:29:39.967159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:39.966227 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" event={"ID":"a122a97d-09d1-485a-902c-2f7832deab2c","Type":"ContainerStarted","Data":"49ce8ab2a48c1face88a013012ca4aa2d43b32be4f07aaa85b067d1eb2c11664"} Oct 30 18:29:39.967159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:39.966349 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" event={"ID":"a122a97d-09d1-485a-902c-2f7832deab2c","Type":"ContainerStarted","Data":"fefb3aafe0c28ba23f79f7f2f935a13a8e51a42788b4f0caf8c041fecffea0e4"} Oct 30 18:29:40.045363 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:40.045295 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" podStartSLOduration=2.045275657 podStartE2EDuration="2.045275657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:29:40.044939084 +0000 UTC m=+1877.135001423" watchObservedRunningTime="2026-10-30 18:29:40.045275657 +0000 UTC m=+1877.135337999" Oct 30 18:29:40.542161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:40.537997 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" Oct 30 18:29:40.542161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:40.538123 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 30 18:29:40.542161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:40.538191 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" podUID="a122a97d-09d1-485a-902c-2f7832deab2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 30 18:29:41.541161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:41.540062 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 30 18:29:41.541161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:41.540156 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" podUID="a122a97d-09d1-485a-902c-2f7832deab2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 30 18:29:42.544160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:42.541574 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 30 18:29:42.544160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:42.541654 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" podUID="a122a97d-09d1-485a-902c-2f7832deab2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 30 18:29:43.540157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:43.537531 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 30 18:29:43.540157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:43.537592 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" podUID="a122a97d-09d1-485a-902c-2f7832deab2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 30 18:29:44.534631 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:44.534596 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 30 18:29:44.535311 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:44.535266 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" podUID="a122a97d-09d1-485a-902c-2f7832deab2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 30 18:29:45.534588 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:45.534542 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 30 18:29:45.534788 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:45.534610 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" podUID="a122a97d-09d1-485a-902c-2f7832deab2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 30 18:29:47.567222 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:29:47.567172 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:29:47.576272 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:47.576244 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:29:47.576466 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:47.576296 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" podUID="a122a97d-09d1-485a-902c-2f7832deab2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:29:48.138447 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:48.138417 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" Oct 30 18:29:48.139383 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:48.139359 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" Oct 30 18:29:48.140615 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:48.140590 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" Oct 30 18:29:48.391104 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:48.391017 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-44379a-644486f9dd-jgqh6"] Oct 30 18:29:48.391306 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:48.391283 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-44379a-644486f9dd-jgqh6" podUID="2978a571-5e62-4f52-ac09-e628c08425ae" containerName="payload-processing" containerID="cri-o://ee31a06f60774a208127ed3526d0f80fdfee1c46871efcc21254414bc62b8245" gracePeriod=30 Oct 30 18:29:48.413402 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:48.410428 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-86bb6c4f5-q8xgr"] Oct 30 18:29:48.413402 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:48.411336 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-86bb6c4f5-q8xgr" podUID="3f32ada3-60e5-44ce-92ad-cf17f0ce9184" containerName="payload-pre-processing" containerID="cri-o://11740e104265d05649201e670adb383d4f0560281193131578139fb09f21c789" gracePeriod=30 Oct 30 18:29:48.726602 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:48.726567 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-44379a-644486f9dd-jgqh6" Oct 30 18:29:48.768662 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:48.768630 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-86bb6c4f5-q8xgr" Oct 30 18:29:48.965762 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:48.965609 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f32ada3-60e5-44ce-92ad-cf17f0ce9184-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f32ada3-60e5-44ce-92ad-cf17f0ce9184-plugins-config-volume\") pod \"3f32ada3-60e5-44ce-92ad-cf17f0ce9184\" (UID: \"3f32ada3-60e5-44ce-92ad-cf17f0ce9184\") " Oct 30 18:29:48.966061 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:48.966040 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2978a571-5e62-4f52-ac09-e628c08425ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2978a571-5e62-4f52-ac09-e628c08425ae-plugins-config-volume\") pod \"2978a571-5e62-4f52-ac09-e628c08425ae\" (UID: \"2978a571-5e62-4f52-ac09-e628c08425ae\") " Oct 30 18:29:48.966379 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:48.966353 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f32ada3-60e5-44ce-92ad-cf17f0ce9184-kube-api-access-4lmch\" (UniqueName: \"kubernetes.io/projected/3f32ada3-60e5-44ce-92ad-cf17f0ce9184-kube-api-access-4lmch\") pod \"3f32ada3-60e5-44ce-92ad-cf17f0ce9184\" (UID: \"3f32ada3-60e5-44ce-92ad-cf17f0ce9184\") " Oct 30 18:29:48.966562 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:48.966532 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2978a571-5e62-4f52-ac09-e628c08425ae-plugins-config-volume" pod "2978a571-5e62-4f52-ac09-e628c08425ae" (UID: "2978a571-5e62-4f52-ac09-e628c08425ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:29:48.966735 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:48.966561 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2978a571-5e62-4f52-ac09-e628c08425ae-kube-api-access-gjzvk\" (UniqueName: \"kubernetes.io/projected/2978a571-5e62-4f52-ac09-e628c08425ae-kube-api-access-gjzvk\") pod \"2978a571-5e62-4f52-ac09-e628c08425ae\" (UID: \"2978a571-5e62-4f52-ac09-e628c08425ae\") " Oct 30 18:29:48.967479 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:48.967453 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2978a571-5e62-4f52-ac09-e628c08425ae-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:29:48.967733 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:48.967713 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f32ada3-60e5-44ce-92ad-cf17f0ce9184-plugins-config-volume" pod "3f32ada3-60e5-44ce-92ad-cf17f0ce9184" (UID: "3f32ada3-60e5-44ce-92ad-cf17f0ce9184"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:29:48.971685 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:48.971660 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f32ada3-60e5-44ce-92ad-cf17f0ce9184-kube-api-access-4lmch" pod "3f32ada3-60e5-44ce-92ad-cf17f0ce9184" (UID: "3f32ada3-60e5-44ce-92ad-cf17f0ce9184"). InnerVolumeSpecName "kube-api-access-4lmch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:48.980846 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:48.980156 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2978a571-5e62-4f52-ac09-e628c08425ae-kube-api-access-gjzvk" pod "2978a571-5e62-4f52-ac09-e628c08425ae" (UID: "2978a571-5e62-4f52-ac09-e628c08425ae"). InnerVolumeSpecName "kube-api-access-gjzvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:49.072304 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:49.072261 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lmch\" (UniqueName: \"kubernetes.io/projected/3f32ada3-60e5-44ce-92ad-cf17f0ce9184-kube-api-access-4lmch\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:29:49.072304 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:49.072296 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjzvk\" (UniqueName: \"kubernetes.io/projected/2978a571-5e62-4f52-ac09-e628c08425ae-kube-api-access-gjzvk\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:29:49.072304 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:49.072309 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f32ada3-60e5-44ce-92ad-cf17f0ce9184-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:29:49.138815 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:49.138779 4442 generic.go:356] "Generic (PLEG): container finished" podID="3f32ada3-60e5-44ce-92ad-cf17f0ce9184" containerID="11740e104265d05649201e670adb383d4f0560281193131578139fb09f21c789" exitCode=0 Oct 30 18:29:49.138990 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:49.138845 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-86bb6c4f5-q8xgr" Oct 30 18:29:49.138990 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:49.138860 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-86bb6c4f5-q8xgr" event={"ID":"3f32ada3-60e5-44ce-92ad-cf17f0ce9184","Type":"ContainerDied","Data":"11740e104265d05649201e670adb383d4f0560281193131578139fb09f21c789"} Oct 30 18:29:49.138990 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:49.138922 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-86bb6c4f5-q8xgr" event={"ID":"3f32ada3-60e5-44ce-92ad-cf17f0ce9184","Type":"ContainerDied","Data":"649547318ecd064a563e615876a1ac8332837a043e3eedfe39f48b9256a47e1b"} Oct 30 18:29:49.138990 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:49.138947 4442 scope.go:122] "RemoveContainer" containerID="11740e104265d05649201e670adb383d4f0560281193131578139fb09f21c789" Oct 30 18:29:49.141183 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:49.141115 4442 generic.go:356] "Generic (PLEG): container finished" podID="2978a571-5e62-4f52-ac09-e628c08425ae" containerID="ee31a06f60774a208127ed3526d0f80fdfee1c46871efcc21254414bc62b8245" exitCode=0 Oct 30 18:29:49.141393 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:49.141368 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-44379a-644486f9dd-jgqh6" event={"ID":"2978a571-5e62-4f52-ac09-e628c08425ae","Type":"ContainerDied","Data":"ee31a06f60774a208127ed3526d0f80fdfee1c46871efcc21254414bc62b8245"} Oct 30 18:29:49.141511 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:49.141425 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-44379a-644486f9dd-jgqh6" event={"ID":"2978a571-5e62-4f52-ac09-e628c08425ae","Type":"ContainerDied","Data":"1d6260ad97c2ae2c6f7df80bdae06e54a229f63e3fe0c454b9f65a31dbbe4d53"} Oct 30 18:29:49.141511 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:49.141496 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-44379a-644486f9dd-jgqh6" Oct 30 18:29:49.150226 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:49.150210 4442 scope.go:122] "RemoveContainer" containerID="11740e104265d05649201e670adb383d4f0560281193131578139fb09f21c789" Oct 30 18:29:49.150508 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:29:49.150484 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11740e104265d05649201e670adb383d4f0560281193131578139fb09f21c789\": container with ID starting with 11740e104265d05649201e670adb383d4f0560281193131578139fb09f21c789 not found: ID does not exist" containerID="11740e104265d05649201e670adb383d4f0560281193131578139fb09f21c789" Oct 30 18:29:49.150605 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:49.150515 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11740e104265d05649201e670adb383d4f0560281193131578139fb09f21c789"} err="failed to get container status \"11740e104265d05649201e670adb383d4f0560281193131578139fb09f21c789\": rpc error: code = NotFound desc = could not find container \"11740e104265d05649201e670adb383d4f0560281193131578139fb09f21c789\": container with ID starting with 11740e104265d05649201e670adb383d4f0560281193131578139fb09f21c789 not found: ID does not exist" Oct 30 18:29:49.150605 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:49.150533 4442 scope.go:122] "RemoveContainer" containerID="ee31a06f60774a208127ed3526d0f80fdfee1c46871efcc21254414bc62b8245" Oct 30 18:29:49.160983 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:49.160966 4442 scope.go:122] "RemoveContainer" containerID="ee31a06f60774a208127ed3526d0f80fdfee1c46871efcc21254414bc62b8245" Oct 30 18:29:49.161311 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:29:49.161291 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee31a06f60774a208127ed3526d0f80fdfee1c46871efcc21254414bc62b8245\": container with ID starting with ee31a06f60774a208127ed3526d0f80fdfee1c46871efcc21254414bc62b8245 not found: ID does not exist" containerID="ee31a06f60774a208127ed3526d0f80fdfee1c46871efcc21254414bc62b8245" Oct 30 18:29:49.161368 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:49.161322 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee31a06f60774a208127ed3526d0f80fdfee1c46871efcc21254414bc62b8245"} err="failed to get container status \"ee31a06f60774a208127ed3526d0f80fdfee1c46871efcc21254414bc62b8245\": rpc error: code = NotFound desc = could not find container \"ee31a06f60774a208127ed3526d0f80fdfee1c46871efcc21254414bc62b8245\": container with ID starting with ee31a06f60774a208127ed3526d0f80fdfee1c46871efcc21254414bc62b8245 not found: ID does not exist" Oct 30 18:29:49.247188 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:49.245833 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-86bb6c4f5-q8xgr"] Oct 30 18:29:49.253153 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:49.251908 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-86bb6c4f5-q8xgr"] Oct 30 18:29:49.273357 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:49.273303 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-44379a-644486f9dd-jgqh6"] Oct 30 18:29:49.278053 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:49.278026 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-44379a-644486f9dd-jgqh6"] Oct 30 18:29:51.065793 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:51.065765 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2978a571-5e62-4f52-ac09-e628c08425ae" path="/var/lib/kubelet/pods/2978a571-5e62-4f52-ac09-e628c08425ae/volumes" Oct 30 18:29:51.066231 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:51.066213 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f32ada3-60e5-44ce-92ad-cf17f0ce9184" path="/var/lib/kubelet/pods/3f32ada3-60e5-44ce-92ad-cf17f0ce9184/volumes" Oct 30 18:29:53.223084 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:53.223044 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl"] Oct 30 18:29:53.223977 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:53.223950 4442 cpu_manager.go:414] "containerMap: removing container" podUID="3f32ada3-60e5-44ce-92ad-cf17f0ce9184" containerName="payload-pre-processing" Oct 30 18:29:53.223977 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:53.223977 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f32ada3-60e5-44ce-92ad-cf17f0ce9184" containerName="payload-pre-processing" Oct 30 18:29:53.224158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:53.224003 4442 cpu_manager.go:414] "containerMap: removing container" podUID="2978a571-5e62-4f52-ac09-e628c08425ae" containerName="payload-processing" Oct 30 18:29:53.224158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:53.224011 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2978a571-5e62-4f52-ac09-e628c08425ae" containerName="payload-processing" Oct 30 18:29:53.224234 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:53.224186 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f32ada3-60e5-44ce-92ad-cf17f0ce9184" containerName="payload-pre-processing" Oct 30 18:29:53.224234 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:53.224199 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="2978a571-5e62-4f52-ac09-e628c08425ae" containerName="payload-processing" Oct 30 18:29:53.228526 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:53.228503 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" Oct 30 18:29:53.263127 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:53.263091 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl"] Oct 30 18:29:53.274001 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:53.273933 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80927367-fb50-4945-86d8-9d22cf64f0ef-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl\" (UID: \"80927367-fb50-4945-86d8-9d22cf64f0ef\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" Oct 30 18:29:53.274369 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:53.274344 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80927367-fb50-4945-86d8-9d22cf64f0ef-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl\" (UID: \"80927367-fb50-4945-86d8-9d22cf64f0ef\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" Oct 30 18:29:53.274593 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:53.274565 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff68j\" (UniqueName: \"kubernetes.io/projected/80927367-fb50-4945-86d8-9d22cf64f0ef-kube-api-access-ff68j\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl\" (UID: \"80927367-fb50-4945-86d8-9d22cf64f0ef\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" Oct 30 18:29:53.274732 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:53.274608 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80927367-fb50-4945-86d8-9d22cf64f0ef-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl\" (UID: \"80927367-fb50-4945-86d8-9d22cf64f0ef\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" Oct 30 18:29:53.274732 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:53.274659 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80927367-fb50-4945-86d8-9d22cf64f0ef-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl\" (UID: \"80927367-fb50-4945-86d8-9d22cf64f0ef\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" Oct 30 18:29:53.274854 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:53.274761 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80927367-fb50-4945-86d8-9d22cf64f0ef-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl\" (UID: \"80927367-fb50-4945-86d8-9d22cf64f0ef\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" Oct 30 18:29:53.274854 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:53.274804 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80927367-fb50-4945-86d8-9d22cf64f0ef-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl\" (UID: \"80927367-fb50-4945-86d8-9d22cf64f0ef\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" Oct 30 18:29:53.274974 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:53.274855 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80927367-fb50-4945-86d8-9d22cf64f0ef-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl\" (UID: \"80927367-fb50-4945-86d8-9d22cf64f0ef\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" Oct 30 18:29:53.274974 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:53.274891 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80927367-fb50-4945-86d8-9d22cf64f0ef-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl\" (UID: \"80927367-fb50-4945-86d8-9d22cf64f0ef\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" Oct 30 18:29:53.376472 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:53.376431 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80927367-fb50-4945-86d8-9d22cf64f0ef-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl\" (UID: \"80927367-fb50-4945-86d8-9d22cf64f0ef\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" Oct 30 18:29:53.376624 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:53.376551 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80927367-fb50-4945-86d8-9d22cf64f0ef-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl\" (UID: \"80927367-fb50-4945-86d8-9d22cf64f0ef\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" Oct 30 18:29:53.376624 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:53.376579 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80927367-fb50-4945-86d8-9d22cf64f0ef-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl\" (UID: \"80927367-fb50-4945-86d8-9d22cf64f0ef\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" Oct 30 18:29:53.376727 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:53.376629 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff68j\" (UniqueName: \"kubernetes.io/projected/80927367-fb50-4945-86d8-9d22cf64f0ef-kube-api-access-ff68j\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl\" (UID: \"80927367-fb50-4945-86d8-9d22cf64f0ef\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" Oct 30 18:29:53.376727 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:53.376659 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80927367-fb50-4945-86d8-9d22cf64f0ef-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl\" (UID: \"80927367-fb50-4945-86d8-9d22cf64f0ef\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" Oct 30 18:29:53.376727 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:53.376705 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80927367-fb50-4945-86d8-9d22cf64f0ef-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl\" (UID: \"80927367-fb50-4945-86d8-9d22cf64f0ef\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" Oct 30 18:29:53.376841 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:53.376788 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80927367-fb50-4945-86d8-9d22cf64f0ef-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl\" (UID: \"80927367-fb50-4945-86d8-9d22cf64f0ef\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" Oct 30 18:29:53.376841 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:53.376831 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80927367-fb50-4945-86d8-9d22cf64f0ef-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl\" (UID: \"80927367-fb50-4945-86d8-9d22cf64f0ef\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" Oct 30 18:29:53.376942 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:53.376877 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80927367-fb50-4945-86d8-9d22cf64f0ef-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl\" (UID: \"80927367-fb50-4945-86d8-9d22cf64f0ef\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" Oct 30 18:29:53.377063 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:53.377037 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80927367-fb50-4945-86d8-9d22cf64f0ef-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl\" (UID: \"80927367-fb50-4945-86d8-9d22cf64f0ef\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" Oct 30 18:29:53.377179 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:53.377104 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80927367-fb50-4945-86d8-9d22cf64f0ef-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl\" (UID: \"80927367-fb50-4945-86d8-9d22cf64f0ef\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" Oct 30 18:29:53.377256 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:53.377217 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80927367-fb50-4945-86d8-9d22cf64f0ef-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl\" (UID: \"80927367-fb50-4945-86d8-9d22cf64f0ef\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" Oct 30 18:29:53.377396 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:53.377375 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80927367-fb50-4945-86d8-9d22cf64f0ef-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl\" (UID: \"80927367-fb50-4945-86d8-9d22cf64f0ef\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" Oct 30 18:29:53.377630 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:53.377613 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80927367-fb50-4945-86d8-9d22cf64f0ef-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl\" (UID: \"80927367-fb50-4945-86d8-9d22cf64f0ef\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" Oct 30 18:29:53.379382 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:53.379358 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80927367-fb50-4945-86d8-9d22cf64f0ef-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl\" (UID: \"80927367-fb50-4945-86d8-9d22cf64f0ef\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" Oct 30 18:29:53.379865 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:53.379849 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80927367-fb50-4945-86d8-9d22cf64f0ef-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl\" (UID: \"80927367-fb50-4945-86d8-9d22cf64f0ef\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" Oct 30 18:29:53.394786 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:53.394756 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80927367-fb50-4945-86d8-9d22cf64f0ef-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl\" (UID: \"80927367-fb50-4945-86d8-9d22cf64f0ef\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" Oct 30 18:29:53.395024 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:53.395003 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff68j\" (UniqueName: \"kubernetes.io/projected/80927367-fb50-4945-86d8-9d22cf64f0ef-kube-api-access-ff68j\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl\" (UID: \"80927367-fb50-4945-86d8-9d22cf64f0ef\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" Oct 30 18:29:53.546406 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:53.546312 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" Oct 30 18:29:53.866109 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:29:53.866060 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80927367_fb50_4945_86d8_9d22cf64f0ef.slice/crio-2f051281cefdfbc21768c904bbe0e116626c0fe5141d08c0d533c5d753169a4a WatchSource:0}: Error finding container 2f051281cefdfbc21768c904bbe0e116626c0fe5141d08c0d533c5d753169a4a: Status 404 returned error can't find the container with id 2f051281cefdfbc21768c904bbe0e116626c0fe5141d08c0d533c5d753169a4a Oct 30 18:29:53.867435 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:53.867407 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl"] Oct 30 18:29:53.868940 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:53.868904 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:29:53.869060 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:53.868981 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:29:53.869060 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:53.869026 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:29:54.191938 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:54.191897 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" event={"ID":"80927367-fb50-4945-86d8-9d22cf64f0ef","Type":"ContainerStarted","Data":"9b402b775fb6c429f5ac017e702b505cfa164329c0cef8e0c69685c9ce7cfbb4"} Oct 30 18:29:54.192113 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:54.191969 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" event={"ID":"80927367-fb50-4945-86d8-9d22cf64f0ef","Type":"ContainerStarted","Data":"2f051281cefdfbc21768c904bbe0e116626c0fe5141d08c0d533c5d753169a4a"} Oct 30 18:29:54.235919 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:54.235857 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" podStartSLOduration=1.235837778 podStartE2EDuration="1.235837778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:29:54.232580373 +0000 UTC m=+1891.322642713" watchObservedRunningTime="2026-10-30 18:29:54.235837778 +0000 UTC m=+1891.325900117" Oct 30 18:29:54.547224 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:54.547121 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" Oct 30 18:29:54.548779 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:54.548753 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 18:29:54.548915 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:54.548799 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" podUID="80927367-fb50-4945-86d8-9d22cf64f0ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 18:29:55.554163 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:55.548547 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 18:29:55.554163 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:55.548613 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" podUID="80927367-fb50-4945-86d8-9d22cf64f0ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 18:29:55.992942 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:55.992904 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z"] Oct 30 18:29:55.993456 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:55.993420 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" podUID="43c061bf-f41c-4bb8-93ca-fe9a3ee5e690" containerName="istio-proxy" containerID="cri-o://5f2d2b9fe82ddbf3708053bdb244c27c1b9f9c1b55e09646017b5cb6f7440a3e" gracePeriod=30 Oct 30 18:29:56.557377 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:56.557340 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 18:29:56.557848 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:56.557815 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" podUID="80927367-fb50-4945-86d8-9d22cf64f0ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 18:29:57.549158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:57.548488 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 18:29:57.549158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:57.548593 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" podUID="80927367-fb50-4945-86d8-9d22cf64f0ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 18:29:58.554157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:58.553540 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 18:29:58.554157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:58.553624 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" podUID="80927367-fb50-4945-86d8-9d22cf64f0ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 18:29:59.548582 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:59.548542 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 18:29:59.548918 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:29:59.548888 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" podUID="80927367-fb50-4945-86d8-9d22cf64f0ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 18:30:00.199158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:00.198210 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889750-xc8x9"] Oct 30 18:30:00.230160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:00.204590 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889750-lz4qt"] Oct 30 18:30:00.230160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:00.215078 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889750-xc8x9" Oct 30 18:30:00.265161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:00.264359 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889750-xc8x9"] Oct 30 18:30:00.276863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:00.276832 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889750-lz4qt" Oct 30 18:30:00.289265 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:00.289226 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889750-lz4qt"] Oct 30 18:30:00.300257 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:00.300232 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:30:00.411167 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:00.394018 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8szg6\" (UniqueName: \"kubernetes.io/projected/2af4a151-5f8a-41e4-acab-a5a09968c2e6-kube-api-access-8szg6\") pod \"maas-api-key-cleanup-e2e-worker-w0-921450-29889750-lz4qt\" (UID: \"2af4a151-5f8a-41e4-acab-a5a09968c2e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889750-lz4qt" Oct 30 18:30:00.411523 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:00.411495 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2mpd\" (UniqueName: \"kubernetes.io/projected/cc799fcb-604b-46c8-ac4d-5f71fc4aaec1-kube-api-access-v2mpd\") pod \"maas-api-key-cleanup-e2e-worker-w4-fe1374-29889750-xc8x9\" (UID: \"cc799fcb-604b-46c8-ac4d-5f71fc4aaec1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889750-xc8x9" Oct 30 18:30:00.447160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:00.425096 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2af4a151-5f8a-41e4-acab-a5a09968c2e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-921450-29889750-lz4qt\" (UID: \"2af4a151-5f8a-41e4-acab-a5a09968c2e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889750-lz4qt" Oct 30 18:30:00.535489 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:00.535397 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc799fcb-604b-46c8-ac4d-5f71fc4aaec1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-fe1374-29889750-xc8x9\" (UID: \"cc799fcb-604b-46c8-ac4d-5f71fc4aaec1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889750-xc8x9" Oct 30 18:30:00.565313 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:00.562309 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 18:30:00.565313 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:00.562375 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" podUID="80927367-fb50-4945-86d8-9d22cf64f0ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 18:30:00.692158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:00.675016 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc799fcb-604b-46c8-ac4d-5f71fc4aaec1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-fe1374-29889750-xc8x9\" (UID: \"cc799fcb-604b-46c8-ac4d-5f71fc4aaec1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889750-xc8x9" Oct 30 18:30:00.710258 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:00.692584 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8szg6\" (UniqueName: \"kubernetes.io/projected/2af4a151-5f8a-41e4-acab-a5a09968c2e6-kube-api-access-8szg6\") pod \"maas-api-key-cleanup-e2e-worker-w0-921450-29889750-lz4qt\" (UID: \"2af4a151-5f8a-41e4-acab-a5a09968c2e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889750-lz4qt" Oct 30 18:30:00.710954 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:00.710922 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2mpd\" (UniqueName: \"kubernetes.io/projected/cc799fcb-604b-46c8-ac4d-5f71fc4aaec1-kube-api-access-v2mpd\") pod \"maas-api-key-cleanup-e2e-worker-w4-fe1374-29889750-xc8x9\" (UID: \"cc799fcb-604b-46c8-ac4d-5f71fc4aaec1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889750-xc8x9" Oct 30 18:30:00.712254 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:00.712229 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2af4a151-5f8a-41e4-acab-a5a09968c2e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-921450-29889750-lz4qt\" (UID: \"2af4a151-5f8a-41e4-acab-a5a09968c2e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889750-lz4qt" Oct 30 18:30:00.737817 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:00.690622 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc799fcb-604b-46c8-ac4d-5f71fc4aaec1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-fe1374-29889750-xc8x9\" (UID: \"cc799fcb-604b-46c8-ac4d-5f71fc4aaec1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889750-xc8x9" Oct 30 18:30:00.737817 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:00.725976 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2af4a151-5f8a-41e4-acab-a5a09968c2e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-921450-29889750-lz4qt\" (UID: \"2af4a151-5f8a-41e4-acab-a5a09968c2e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889750-lz4qt" Oct 30 18:30:00.829217 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:00.827685 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8szg6\" (UniqueName: \"kubernetes.io/projected/2af4a151-5f8a-41e4-acab-a5a09968c2e6-kube-api-access-8szg6\") pod \"maas-api-key-cleanup-e2e-worker-w0-921450-29889750-lz4qt\" (UID: \"2af4a151-5f8a-41e4-acab-a5a09968c2e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889750-lz4qt" Oct 30 18:30:00.835985 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:00.835952 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889750-lz4qt" Oct 30 18:30:00.837424 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:00.837356 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2mpd\" (UniqueName: \"kubernetes.io/projected/cc799fcb-604b-46c8-ac4d-5f71fc4aaec1-kube-api-access-v2mpd\") pod \"maas-api-key-cleanup-e2e-worker-w4-fe1374-29889750-xc8x9\" (UID: \"cc799fcb-604b-46c8-ac4d-5f71fc4aaec1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889750-xc8x9" Oct 30 18:30:01.020169 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:01.019951 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889750-xc8x9" Oct 30 18:30:01.445213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:01.445170 4442 generic.go:356] "Generic (PLEG): container finished" podID="43c061bf-f41c-4bb8-93ca-fe9a3ee5e690" containerID="5f2d2b9fe82ddbf3708053bdb244c27c1b9f9c1b55e09646017b5cb6f7440a3e" exitCode=0 Oct 30 18:30:01.452653 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:01.452215 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" event={"ID":"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690","Type":"ContainerDied","Data":"5f2d2b9fe82ddbf3708053bdb244c27c1b9f9c1b55e09646017b5cb6f7440a3e"} Oct 30 18:30:01.453000 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:01.452980 4442 scope.go:122] "RemoveContainer" containerID="2a05cfd6b5ffdfdacede4a79d5df7c56a2863053ddba693efa47ea469237e376" Oct 30 18:30:01.589160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:01.588602 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 18:30:01.589160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:01.588668 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" podUID="80927367-fb50-4945-86d8-9d22cf64f0ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 18:30:02.245211 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:02.237070 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4" podUID="cf7d5f54-e091-4032-ab11-af2687e453c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:02.742203 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:02.733238 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:30:02.742203 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:02.733342 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" podUID="f4e00bfc-fc1d-46ce-946d-969438944b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:02.783183 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:02.771038 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:30:02.783183 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:02.771125 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" podUID="a99c9a23-d65d-4950-a153-71c8decd10aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:02.819339 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:02.819124 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8" podUID="1e2901e8-4919-4e5c-88c7-39126a7cca56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:02.866173 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:02.860800 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889750-lz4qt"] Oct 30 18:30:02.866173 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:02.860955 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:30:02.866173 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:02.861015 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" podUID="7084397f-85eb-43f9-8314-530af31c81c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:02.866173 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:02.861108 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 18:30:02.866173 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:02.861183 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" podUID="80927367-fb50-4945-86d8-9d22cf64f0ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 18:30:03.291195 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:03.278217 4442 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8" podUID="1e2901e8-4919-4e5c-88c7-39126a7cca56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:03.481246 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:03.481214 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889750-xc8x9"] Oct 30 18:30:03.775913 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:03.775880 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" Oct 30 18:30:03.911672 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:03.910165 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-workload-certs\") pod \"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690\" (UID: \"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690\") " Oct 30 18:30:03.911672 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:03.910220 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-istiod-ca-cert\") pod \"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690\" (UID: \"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690\") " Oct 30 18:30:03.911672 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:03.910263 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-workload-socket\") pod \"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690\" (UID: \"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690\") " Oct 30 18:30:03.911672 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:03.910332 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-credential-socket\") pod \"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690\" (UID: \"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690\") " Oct 30 18:30:03.911672 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:03.910367 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-istio-data\") pod \"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690\" (UID: \"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690\") " Oct 30 18:30:03.911672 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:03.910422 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-istio-podinfo\") pod \"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690\" (UID: \"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690\") " Oct 30 18:30:03.911672 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:03.910552 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-istio-envoy\") pod \"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690\" (UID: \"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690\") " Oct 30 18:30:03.911672 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:03.910602 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-kube-api-access-slv4c\" (UniqueName: \"kubernetes.io/projected/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-kube-api-access-slv4c\") pod \"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690\" (UID: \"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690\") " Oct 30 18:30:03.911672 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:03.910653 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-istio-token\" (UniqueName: \"kubernetes.io/projected/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-istio-token\") pod \"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690\" (UID: \"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690\") " Oct 30 18:30:03.917727 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:03.917537 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-credential-socket" pod "43c061bf-f41c-4bb8-93ca-fe9a3ee5e690" (UID: "43c061bf-f41c-4bb8-93ca-fe9a3ee5e690"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:30:03.918203 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:03.917941 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-workload-certs" pod "43c061bf-f41c-4bb8-93ca-fe9a3ee5e690" (UID: "43c061bf-f41c-4bb8-93ca-fe9a3ee5e690"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:30:03.918664 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:03.918452 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-istiod-ca-cert" pod "43c061bf-f41c-4bb8-93ca-fe9a3ee5e690" (UID: "43c061bf-f41c-4bb8-93ca-fe9a3ee5e690"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:03.919150 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:03.918815 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-workload-socket" pod "43c061bf-f41c-4bb8-93ca-fe9a3ee5e690" (UID: "43c061bf-f41c-4bb8-93ca-fe9a3ee5e690"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:30:03.920258 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:03.919933 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-istio-data" pod "43c061bf-f41c-4bb8-93ca-fe9a3ee5e690" (UID: "43c061bf-f41c-4bb8-93ca-fe9a3ee5e690"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:30:03.967599 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:03.967490 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889750-xc8x9" event={"ID":"cc799fcb-604b-46c8-ac4d-5f71fc4aaec1","Type":"ContainerStarted","Data":"3455f5cecd88d5ed9eccc1574ba787f585420aff2d4ee3329a0bf623d00ec73f"} Oct 30 18:30:03.971563 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:03.970880 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" event={"ID":"43c061bf-f41c-4bb8-93ca-fe9a3ee5e690","Type":"ContainerDied","Data":"ca4ba257793da66045da454b9256e8e381fbc64c40a6084f0e890c5f0b1b7e56"} Oct 30 18:30:03.971563 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:03.970947 4442 scope.go:122] "RemoveContainer" containerID="5f2d2b9fe82ddbf3708053bdb244c27c1b9f9c1b55e09646017b5cb6f7440a3e" Oct 30 18:30:03.971563 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:03.971070 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z" Oct 30 18:30:03.975286 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:03.975043 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889750-lz4qt" event={"ID":"2af4a151-5f8a-41e4-acab-a5a09968c2e6","Type":"ContainerStarted","Data":"45f27b1b9b822ca983226f9278b5200fde786978f7ccc10f380cb2cbd708a08e"} Oct 30 18:30:03.996821 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:03.990345 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-istio-envoy" pod "43c061bf-f41c-4bb8-93ca-fe9a3ee5e690" (UID: "43c061bf-f41c-4bb8-93ca-fe9a3ee5e690"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:30:03.996821 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:03.990445 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-istio-podinfo" pod "43c061bf-f41c-4bb8-93ca-fe9a3ee5e690" (UID: "43c061bf-f41c-4bb8-93ca-fe9a3ee5e690"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:30:04.021707 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:04.020693 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-istio-token" pod "43c061bf-f41c-4bb8-93ca-fe9a3ee5e690" (UID: "43c061bf-f41c-4bb8-93ca-fe9a3ee5e690"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:04.069159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:04.051964 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-kube-api-access-slv4c" pod "43c061bf-f41c-4bb8-93ca-fe9a3ee5e690" (UID: "43c061bf-f41c-4bb8-93ca-fe9a3ee5e690"). InnerVolumeSpecName "kube-api-access-slv4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:04.069159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:04.053659 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:30:04.069159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:04.053688 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:30:04.069159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:04.053703 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:30:04.069159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:04.053724 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:30:04.069159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:04.053739 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:30:04.069159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:04.053752 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:30:04.069159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:04.053764 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:30:04.069159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:04.053783 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slv4c\" (UniqueName: \"kubernetes.io/projected/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-kube-api-access-slv4c\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:30:04.069159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:04.053796 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:30:04.319417 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:04.319335 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z"] Oct 30 18:30:04.325190 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:04.324075 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-44379a-openshift-default-57d99754b9-pgg7z"] Oct 30 18:30:04.331008 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:04.330961 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" Oct 30 18:30:04.331950 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:04.331845 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" Oct 30 18:30:04.341067 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:04.341033 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" Oct 30 18:30:04.987912 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:04.987855 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889750-xc8x9" event={"ID":"cc799fcb-604b-46c8-ac4d-5f71fc4aaec1","Type":"ContainerStarted","Data":"05a07e7690a78e7d7992f6b6692a8ac504c47c1c22ffcf61e95c8d7b8fbc6eb8"} Oct 30 18:30:04.993507 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:04.993456 4442 generic.go:356] "Generic (PLEG): container finished" podID="2af4a151-5f8a-41e4-acab-a5a09968c2e6" containerID="3f0e15d121b7f7b09a1996cf67fd5ada9f8330c608f163708db17da0c94dd23b" exitCode=60 Oct 30 18:30:04.993752 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:04.993651 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889750-lz4qt" event={"ID":"2af4a151-5f8a-41e4-acab-a5a09968c2e6","Type":"ContainerDied","Data":"3f0e15d121b7f7b09a1996cf67fd5ada9f8330c608f163708db17da0c94dd23b"} Oct 30 18:30:04.994496 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:04.994460 4442 scope.go:122] "RemoveContainer" containerID="3f0e15d121b7f7b09a1996cf67fd5ada9f8330c608f163708db17da0c94dd23b" Oct 30 18:30:05.035337 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:05.035114 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889750-xc8x9" podStartSLOduration=5.035091542 podStartE2EDuration="5.035091542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:05.033561565 +0000 UTC m=+1902.123623906" watchObservedRunningTime="2026-10-30 18:30:05.035091542 +0000 UTC m=+1902.125153888" Oct 30 18:30:05.070752 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:05.070712 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43c061bf-f41c-4bb8-93ca-fe9a3ee5e690" path="/var/lib/kubelet/pods/43c061bf-f41c-4bb8-93ca-fe9a3ee5e690/volumes" Oct 30 18:30:06.003702 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:06.003661 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889750-lz4qt" event={"ID":"2af4a151-5f8a-41e4-acab-a5a09968c2e6","Type":"ContainerStarted","Data":"44328f38f7bcfb7770bece52355edaefb0aa341c8a78cb4b30cbdf21251ab7a4"} Oct 30 18:30:06.005418 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:06.005391 4442 generic.go:356] "Generic (PLEG): container finished" podID="cc799fcb-604b-46c8-ac4d-5f71fc4aaec1" containerID="05a07e7690a78e7d7992f6b6692a8ac504c47c1c22ffcf61e95c8d7b8fbc6eb8" exitCode=60 Oct 30 18:30:06.005551 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:06.005525 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889750-xc8x9" event={"ID":"cc799fcb-604b-46c8-ac4d-5f71fc4aaec1","Type":"ContainerDied","Data":"05a07e7690a78e7d7992f6b6692a8ac504c47c1c22ffcf61e95c8d7b8fbc6eb8"} Oct 30 18:30:06.005670 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:06.005657 4442 scope.go:122] "RemoveContainer" containerID="05a07e7690a78e7d7992f6b6692a8ac504c47c1c22ffcf61e95c8d7b8fbc6eb8" Oct 30 18:30:06.095956 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:06.095895 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889750-lz4qt" podStartSLOduration=6.09587898 podStartE2EDuration="6.09587898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:06.046520018 +0000 UTC m=+1903.136582358" watchObservedRunningTime="2026-10-30 18:30:06.09587898 +0000 UTC m=+1903.185941326" Oct 30 18:30:07.014768 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:07.014735 4442 generic.go:356] "Generic (PLEG): container finished" podID="2af4a151-5f8a-41e4-acab-a5a09968c2e6" containerID="44328f38f7bcfb7770bece52355edaefb0aa341c8a78cb4b30cbdf21251ab7a4" exitCode=60 Oct 30 18:30:07.015307 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:07.014813 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889750-lz4qt" event={"ID":"2af4a151-5f8a-41e4-acab-a5a09968c2e6","Type":"ContainerDied","Data":"44328f38f7bcfb7770bece52355edaefb0aa341c8a78cb4b30cbdf21251ab7a4"} Oct 30 18:30:07.015307 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:07.014885 4442 scope.go:122] "RemoveContainer" containerID="3f0e15d121b7f7b09a1996cf67fd5ada9f8330c608f163708db17da0c94dd23b" Oct 30 18:30:07.015307 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:07.015228 4442 scope.go:122] "RemoveContainer" containerID="44328f38f7bcfb7770bece52355edaefb0aa341c8a78cb4b30cbdf21251ab7a4" Oct 30 18:30:07.015634 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:30:07.015609 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-921450-29889750-lz4qt_odh-ai-gateway-infra(2af4a151-5f8a-41e4-acab-a5a09968c2e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889750-lz4qt" podUID="2af4a151-5f8a-41e4-acab-a5a09968c2e6" Oct 30 18:30:07.017535 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:07.017510 4442 generic.go:356] "Generic (PLEG): container finished" podID="cc799fcb-604b-46c8-ac4d-5f71fc4aaec1" containerID="4ee6702f720ad0d1a90c3dac49eb19a744ce33e5e1b12c0a18ffc121909e1889" exitCode=60 Oct 30 18:30:07.017648 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:07.017621 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889750-xc8x9" event={"ID":"cc799fcb-604b-46c8-ac4d-5f71fc4aaec1","Type":"ContainerDied","Data":"4ee6702f720ad0d1a90c3dac49eb19a744ce33e5e1b12c0a18ffc121909e1889"} Oct 30 18:30:07.017898 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:07.017882 4442 scope.go:122] "RemoveContainer" containerID="4ee6702f720ad0d1a90c3dac49eb19a744ce33e5e1b12c0a18ffc121909e1889" Oct 30 18:30:07.018235 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:30:07.018211 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-fe1374-29889750-xc8x9_odh-ai-gateway-infra(cc799fcb-604b-46c8-ac4d-5f71fc4aaec1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889750-xc8x9" podUID="cc799fcb-604b-46c8-ac4d-5f71fc4aaec1" Oct 30 18:30:07.031766 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:07.031740 4442 scope.go:122] "RemoveContainer" containerID="05a07e7690a78e7d7992f6b6692a8ac504c47c1c22ffcf61e95c8d7b8fbc6eb8" Oct 30 18:30:08.028071 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:08.028040 4442 scope.go:122] "RemoveContainer" containerID="4ee6702f720ad0d1a90c3dac49eb19a744ce33e5e1b12c0a18ffc121909e1889" Oct 30 18:30:08.028626 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:30:08.028385 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-fe1374-29889750-xc8x9_odh-ai-gateway-infra(cc799fcb-604b-46c8-ac4d-5f71fc4aaec1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889750-xc8x9" podUID="cc799fcb-604b-46c8-ac4d-5f71fc4aaec1" Oct 30 18:30:08.030216 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:08.030193 4442 scope.go:122] "RemoveContainer" containerID="44328f38f7bcfb7770bece52355edaefb0aa341c8a78cb4b30cbdf21251ab7a4" Oct 30 18:30:08.030470 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:30:08.030450 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-921450-29889750-lz4qt_odh-ai-gateway-infra(2af4a151-5f8a-41e4-acab-a5a09968c2e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889750-lz4qt" podUID="2af4a151-5f8a-41e4-acab-a5a09968c2e6" Oct 30 18:30:13.936548 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:13.936499 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-200c58-7c6bfb8c8b-ppvqf"] Oct 30 18:30:13.974172 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:13.962007 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-200c58-7d66656f4c-8k98q"] Oct 30 18:30:17.563003 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:30:17.562959 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:30:18.057561 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:18.057529 4442 scope.go:122] "RemoveContainer" containerID="44328f38f7bcfb7770bece52355edaefb0aa341c8a78cb4b30cbdf21251ab7a4" Oct 30 18:30:18.556214 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:18.556181 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889750-lz4qt" event={"ID":"2af4a151-5f8a-41e4-acab-a5a09968c2e6","Type":"ContainerStarted","Data":"df33c761c20801fbe357c638588b1f2f62f4fe839b7eea7065ae51ac569675cd"} Oct 30 18:30:18.937804 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:18.937754 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-200c58-7c6bfb8c8b-ppvqf" podUID="0f1c999a-d8e4-4ee1-8cca-f53f93eed216" containerName="payload-processing" containerID="cri-o://ede9700fe29c1e0fc3c58e85aa0652e83a5894d6bb337c65ff700f6cc78cb578" gracePeriod=25 Oct 30 18:30:18.974990 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:18.974952 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-200c58-7d66656f4c-8k98q" podUID="9e4d3b1c-e383-4db5-aeec-8a6a66d32808" containerName="payload-pre-processing" containerID="cri-o://0f31c7b74d23e79f7419b3c59e646d981db8b56cbd42ead3426ca08aac1dd4b6" gracePeriod=25 Oct 30 18:30:19.057386 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:19.057348 4442 scope.go:122] "RemoveContainer" containerID="4ee6702f720ad0d1a90c3dac49eb19a744ce33e5e1b12c0a18ffc121909e1889" Oct 30 18:30:19.325894 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:19.325173 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-200c58-7c6bfb8c8b-ppvqf" Oct 30 18:30:19.345613 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:19.345581 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-200c58-7d66656f4c-8k98q" Oct 30 18:30:19.447744 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:19.447718 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e4d3b1c-e383-4db5-aeec-8a6a66d32808-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e4d3b1c-e383-4db5-aeec-8a6a66d32808-plugins-config-volume\") pod \"9e4d3b1c-e383-4db5-aeec-8a6a66d32808\" (UID: \"9e4d3b1c-e383-4db5-aeec-8a6a66d32808\") " Oct 30 18:30:19.447893 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:19.447856 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e4d3b1c-e383-4db5-aeec-8a6a66d32808-kube-api-access-5r9wz\" (UniqueName: \"kubernetes.io/projected/9e4d3b1c-e383-4db5-aeec-8a6a66d32808-kube-api-access-5r9wz\") pod \"9e4d3b1c-e383-4db5-aeec-8a6a66d32808\" (UID: \"9e4d3b1c-e383-4db5-aeec-8a6a66d32808\") " Oct 30 18:30:19.447948 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:19.447892 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f1c999a-d8e4-4ee1-8cca-f53f93eed216-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f1c999a-d8e4-4ee1-8cca-f53f93eed216-plugins-config-volume\") pod \"0f1c999a-d8e4-4ee1-8cca-f53f93eed216\" (UID: \"0f1c999a-d8e4-4ee1-8cca-f53f93eed216\") " Oct 30 18:30:19.447948 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:19.447924 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f1c999a-d8e4-4ee1-8cca-f53f93eed216-kube-api-access-9g4fc\" (UniqueName: \"kubernetes.io/projected/0f1c999a-d8e4-4ee1-8cca-f53f93eed216-kube-api-access-9g4fc\") pod \"0f1c999a-d8e4-4ee1-8cca-f53f93eed216\" (UID: \"0f1c999a-d8e4-4ee1-8cca-f53f93eed216\") " Oct 30 18:30:19.448245 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:19.448217 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e4d3b1c-e383-4db5-aeec-8a6a66d32808-plugins-config-volume" pod "9e4d3b1c-e383-4db5-aeec-8a6a66d32808" (UID: "9e4d3b1c-e383-4db5-aeec-8a6a66d32808"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:19.448310 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:19.448294 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f1c999a-d8e4-4ee1-8cca-f53f93eed216-plugins-config-volume" pod "0f1c999a-d8e4-4ee1-8cca-f53f93eed216" (UID: "0f1c999a-d8e4-4ee1-8cca-f53f93eed216"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:19.448611 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:19.448538 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f1c999a-d8e4-4ee1-8cca-f53f93eed216-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:30:19.448611 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:19.448565 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e4d3b1c-e383-4db5-aeec-8a6a66d32808-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:30:19.450372 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:19.450345 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e4d3b1c-e383-4db5-aeec-8a6a66d32808-kube-api-access-5r9wz" pod "9e4d3b1c-e383-4db5-aeec-8a6a66d32808" (UID: "9e4d3b1c-e383-4db5-aeec-8a6a66d32808"). InnerVolumeSpecName "kube-api-access-5r9wz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:19.450464 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:19.450377 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f1c999a-d8e4-4ee1-8cca-f53f93eed216-kube-api-access-9g4fc" pod "0f1c999a-d8e4-4ee1-8cca-f53f93eed216" (UID: "0f1c999a-d8e4-4ee1-8cca-f53f93eed216"). InnerVolumeSpecName "kube-api-access-9g4fc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:19.550304 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:19.550273 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r9wz\" (UniqueName: \"kubernetes.io/projected/9e4d3b1c-e383-4db5-aeec-8a6a66d32808-kube-api-access-5r9wz\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:30:19.550421 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:19.550309 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g4fc\" (UniqueName: \"kubernetes.io/projected/0f1c999a-d8e4-4ee1-8cca-f53f93eed216-kube-api-access-9g4fc\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:30:19.565995 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:19.565959 4442 generic.go:356] "Generic (PLEG): container finished" podID="2af4a151-5f8a-41e4-acab-a5a09968c2e6" containerID="df33c761c20801fbe357c638588b1f2f62f4fe839b7eea7065ae51ac569675cd" exitCode=60 Oct 30 18:30:19.566117 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:19.566038 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889750-lz4qt" event={"ID":"2af4a151-5f8a-41e4-acab-a5a09968c2e6","Type":"ContainerDied","Data":"df33c761c20801fbe357c638588b1f2f62f4fe839b7eea7065ae51ac569675cd"} Oct 30 18:30:19.566186 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:19.566107 4442 scope.go:122] "RemoveContainer" containerID="44328f38f7bcfb7770bece52355edaefb0aa341c8a78cb4b30cbdf21251ab7a4" Oct 30 18:30:19.566566 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:19.566548 4442 scope.go:122] "RemoveContainer" containerID="df33c761c20801fbe357c638588b1f2f62f4fe839b7eea7065ae51ac569675cd" Oct 30 18:30:19.566912 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:30:19.566890 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-921450-29889750-lz4qt_odh-ai-gateway-infra(2af4a151-5f8a-41e4-acab-a5a09968c2e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889750-lz4qt" podUID="2af4a151-5f8a-41e4-acab-a5a09968c2e6" Oct 30 18:30:19.568495 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:19.568472 4442 generic.go:356] "Generic (PLEG): container finished" podID="9e4d3b1c-e383-4db5-aeec-8a6a66d32808" containerID="0f31c7b74d23e79f7419b3c59e646d981db8b56cbd42ead3426ca08aac1dd4b6" exitCode=0 Oct 30 18:30:19.568582 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:19.568554 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-200c58-7d66656f4c-8k98q" event={"ID":"9e4d3b1c-e383-4db5-aeec-8a6a66d32808","Type":"ContainerDied","Data":"0f31c7b74d23e79f7419b3c59e646d981db8b56cbd42ead3426ca08aac1dd4b6"} Oct 30 18:30:19.568648 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:19.568596 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-200c58-7d66656f4c-8k98q" event={"ID":"9e4d3b1c-e383-4db5-aeec-8a6a66d32808","Type":"ContainerDied","Data":"337eddc122710c71ce758be9ebb79d943742cca8a88ab75d58cf50e3ada22863"} Oct 30 18:30:19.568648 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:19.568643 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-200c58-7d66656f4c-8k98q" Oct 30 18:30:19.570034 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:19.570010 4442 generic.go:356] "Generic (PLEG): container finished" podID="0f1c999a-d8e4-4ee1-8cca-f53f93eed216" containerID="ede9700fe29c1e0fc3c58e85aa0652e83a5894d6bb337c65ff700f6cc78cb578" exitCode=0 Oct 30 18:30:19.570250 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:19.570052 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-200c58-7c6bfb8c8b-ppvqf" Oct 30 18:30:19.570250 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:19.570195 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-200c58-7c6bfb8c8b-ppvqf" event={"ID":"0f1c999a-d8e4-4ee1-8cca-f53f93eed216","Type":"ContainerDied","Data":"ede9700fe29c1e0fc3c58e85aa0652e83a5894d6bb337c65ff700f6cc78cb578"} Oct 30 18:30:19.570250 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:19.570243 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-200c58-7c6bfb8c8b-ppvqf" event={"ID":"0f1c999a-d8e4-4ee1-8cca-f53f93eed216","Type":"ContainerDied","Data":"ecd4fc5d689fb0a5193d5dc4aea5ed497d67ee350bf4dfc3304907b804838f89"} Oct 30 18:30:19.573882 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:19.573860 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889750-xc8x9" event={"ID":"cc799fcb-604b-46c8-ac4d-5f71fc4aaec1","Type":"ContainerStarted","Data":"2645f1476fb1f21bea2b9472858ab8b6f943e20fdf5c11918382d3c0de18c16f"} Oct 30 18:30:19.582291 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:19.582261 4442 scope.go:122] "RemoveContainer" containerID="0f31c7b74d23e79f7419b3c59e646d981db8b56cbd42ead3426ca08aac1dd4b6" Oct 30 18:30:19.592056 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:19.592039 4442 scope.go:122] "RemoveContainer" containerID="0f31c7b74d23e79f7419b3c59e646d981db8b56cbd42ead3426ca08aac1dd4b6" Oct 30 18:30:19.592363 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:30:19.592343 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f31c7b74d23e79f7419b3c59e646d981db8b56cbd42ead3426ca08aac1dd4b6\": container with ID starting with 0f31c7b74d23e79f7419b3c59e646d981db8b56cbd42ead3426ca08aac1dd4b6 not found: ID does not exist" containerID="0f31c7b74d23e79f7419b3c59e646d981db8b56cbd42ead3426ca08aac1dd4b6" Oct 30 18:30:19.592427 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:19.592371 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f31c7b74d23e79f7419b3c59e646d981db8b56cbd42ead3426ca08aac1dd4b6"} err="failed to get container status \"0f31c7b74d23e79f7419b3c59e646d981db8b56cbd42ead3426ca08aac1dd4b6\": rpc error: code = NotFound desc = could not find container \"0f31c7b74d23e79f7419b3c59e646d981db8b56cbd42ead3426ca08aac1dd4b6\": container with ID starting with 0f31c7b74d23e79f7419b3c59e646d981db8b56cbd42ead3426ca08aac1dd4b6 not found: ID does not exist" Oct 30 18:30:19.592427 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:19.592391 4442 scope.go:122] "RemoveContainer" containerID="ede9700fe29c1e0fc3c58e85aa0652e83a5894d6bb337c65ff700f6cc78cb578" Oct 30 18:30:19.602119 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:19.602093 4442 scope.go:122] "RemoveContainer" containerID="ede9700fe29c1e0fc3c58e85aa0652e83a5894d6bb337c65ff700f6cc78cb578" Oct 30 18:30:19.602497 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:30:19.602465 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ede9700fe29c1e0fc3c58e85aa0652e83a5894d6bb337c65ff700f6cc78cb578\": container with ID starting with ede9700fe29c1e0fc3c58e85aa0652e83a5894d6bb337c65ff700f6cc78cb578 not found: ID does not exist" containerID="ede9700fe29c1e0fc3c58e85aa0652e83a5894d6bb337c65ff700f6cc78cb578" Oct 30 18:30:19.602587 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:19.602504 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ede9700fe29c1e0fc3c58e85aa0652e83a5894d6bb337c65ff700f6cc78cb578"} err="failed to get container status \"ede9700fe29c1e0fc3c58e85aa0652e83a5894d6bb337c65ff700f6cc78cb578\": rpc error: code = NotFound desc = could not find container \"ede9700fe29c1e0fc3c58e85aa0652e83a5894d6bb337c65ff700f6cc78cb578\": container with ID starting with ede9700fe29c1e0fc3c58e85aa0652e83a5894d6bb337c65ff700f6cc78cb578 not found: ID does not exist" Oct 30 18:30:19.616335 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:19.616309 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889750-lz4qt"] Oct 30 18:30:19.646060 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:19.646020 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-200c58-7c6bfb8c8b-ppvqf"] Oct 30 18:30:19.649322 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:19.649296 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-200c58-7c6bfb8c8b-ppvqf"] Oct 30 18:30:19.679908 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:19.679878 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-200c58-7d66656f4c-8k98q"] Oct 30 18:30:19.685364 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:19.685340 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-200c58-7d66656f4c-8k98q"] Oct 30 18:30:20.587806 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:20.587780 4442 generic.go:356] "Generic (PLEG): container finished" podID="cc799fcb-604b-46c8-ac4d-5f71fc4aaec1" containerID="2645f1476fb1f21bea2b9472858ab8b6f943e20fdf5c11918382d3c0de18c16f" exitCode=60 Oct 30 18:30:20.588241 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:20.587853 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889750-xc8x9" event={"ID":"cc799fcb-604b-46c8-ac4d-5f71fc4aaec1","Type":"ContainerDied","Data":"2645f1476fb1f21bea2b9472858ab8b6f943e20fdf5c11918382d3c0de18c16f"} Oct 30 18:30:20.588241 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:20.587930 4442 scope.go:122] "RemoveContainer" containerID="4ee6702f720ad0d1a90c3dac49eb19a744ce33e5e1b12c0a18ffc121909e1889" Oct 30 18:30:20.588436 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:20.588420 4442 scope.go:122] "RemoveContainer" containerID="2645f1476fb1f21bea2b9472858ab8b6f943e20fdf5c11918382d3c0de18c16f" Oct 30 18:30:20.588779 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:30:20.588759 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-fe1374-29889750-xc8x9_odh-ai-gateway-infra(cc799fcb-604b-46c8-ac4d-5f71fc4aaec1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889750-xc8x9" podUID="cc799fcb-604b-46c8-ac4d-5f71fc4aaec1" Oct 30 18:30:20.733945 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:20.733907 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889750-xc8x9"] Oct 30 18:30:20.759306 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:20.759278 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889750-lz4qt" Oct 30 18:30:20.866924 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:20.866890 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2af4a151-5f8a-41e4-acab-a5a09968c2e6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2af4a151-5f8a-41e4-acab-a5a09968c2e6-maas-api-service-ca\") pod \"2af4a151-5f8a-41e4-acab-a5a09968c2e6\" (UID: \"2af4a151-5f8a-41e4-acab-a5a09968c2e6\") " Oct 30 18:30:20.867366 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:20.867331 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2af4a151-5f8a-41e4-acab-a5a09968c2e6-maas-api-service-ca" pod "2af4a151-5f8a-41e4-acab-a5a09968c2e6" (UID: "2af4a151-5f8a-41e4-acab-a5a09968c2e6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:20.867504 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:20.867398 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2af4a151-5f8a-41e4-acab-a5a09968c2e6-kube-api-access-8szg6\" (UniqueName: \"kubernetes.io/projected/2af4a151-5f8a-41e4-acab-a5a09968c2e6-kube-api-access-8szg6\") pod \"2af4a151-5f8a-41e4-acab-a5a09968c2e6\" (UID: \"2af4a151-5f8a-41e4-acab-a5a09968c2e6\") " Oct 30 18:30:20.868091 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:20.868027 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2af4a151-5f8a-41e4-acab-a5a09968c2e6-maas-api-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:30:20.870163 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:20.870103 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2af4a151-5f8a-41e4-acab-a5a09968c2e6-kube-api-access-8szg6" pod "2af4a151-5f8a-41e4-acab-a5a09968c2e6" (UID: "2af4a151-5f8a-41e4-acab-a5a09968c2e6"). InnerVolumeSpecName "kube-api-access-8szg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:20.969229 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:20.969193 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8szg6\" (UniqueName: \"kubernetes.io/projected/2af4a151-5f8a-41e4-acab-a5a09968c2e6-kube-api-access-8szg6\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:30:21.067545 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:21.067511 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f1c999a-d8e4-4ee1-8cca-f53f93eed216" path="/var/lib/kubelet/pods/0f1c999a-d8e4-4ee1-8cca-f53f93eed216/volumes" Oct 30 18:30:21.068041 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:21.068020 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e4d3b1c-e383-4db5-aeec-8a6a66d32808" path="/var/lib/kubelet/pods/9e4d3b1c-e383-4db5-aeec-8a6a66d32808/volumes" Oct 30 18:30:21.604919 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:21.604889 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889750-lz4qt" Oct 30 18:30:21.605400 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:21.604922 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889750-lz4qt" event={"ID":"2af4a151-5f8a-41e4-acab-a5a09968c2e6","Type":"ContainerDied","Data":"45f27b1b9b822ca983226f9278b5200fde786978f7ccc10f380cb2cbd708a08e"} Oct 30 18:30:21.605400 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:21.605005 4442 scope.go:122] "RemoveContainer" containerID="df33c761c20801fbe357c638588b1f2f62f4fe839b7eea7065ae51ac569675cd" Oct 30 18:30:21.698744 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:21.698707 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889750-lz4qt"] Oct 30 18:30:21.702825 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:21.702795 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889750-lz4qt"] Oct 30 18:30:21.778678 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:21.778653 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889750-xc8x9" Oct 30 18:30:21.880670 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:21.880640 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc799fcb-604b-46c8-ac4d-5f71fc4aaec1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc799fcb-604b-46c8-ac4d-5f71fc4aaec1-maas-api-service-ca\") pod \"cc799fcb-604b-46c8-ac4d-5f71fc4aaec1\" (UID: \"cc799fcb-604b-46c8-ac4d-5f71fc4aaec1\") " Oct 30 18:30:21.880870 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:21.880725 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc799fcb-604b-46c8-ac4d-5f71fc4aaec1-kube-api-access-v2mpd\" (UniqueName: \"kubernetes.io/projected/cc799fcb-604b-46c8-ac4d-5f71fc4aaec1-kube-api-access-v2mpd\") pod \"cc799fcb-604b-46c8-ac4d-5f71fc4aaec1\" (UID: \"cc799fcb-604b-46c8-ac4d-5f71fc4aaec1\") " Oct 30 18:30:21.881069 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:21.881045 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc799fcb-604b-46c8-ac4d-5f71fc4aaec1-maas-api-service-ca" pod "cc799fcb-604b-46c8-ac4d-5f71fc4aaec1" (UID: "cc799fcb-604b-46c8-ac4d-5f71fc4aaec1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:21.883045 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:21.883018 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc799fcb-604b-46c8-ac4d-5f71fc4aaec1-kube-api-access-v2mpd" pod "cc799fcb-604b-46c8-ac4d-5f71fc4aaec1" (UID: "cc799fcb-604b-46c8-ac4d-5f71fc4aaec1"). InnerVolumeSpecName "kube-api-access-v2mpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:21.982783 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:21.982740 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc799fcb-604b-46c8-ac4d-5f71fc4aaec1-maas-api-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:30:21.982783 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:21.982780 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2mpd\" (UniqueName: \"kubernetes.io/projected/cc799fcb-604b-46c8-ac4d-5f71fc4aaec1-kube-api-access-v2mpd\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:30:22.616469 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:22.616435 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889750-xc8x9" event={"ID":"cc799fcb-604b-46c8-ac4d-5f71fc4aaec1","Type":"ContainerDied","Data":"3455f5cecd88d5ed9eccc1574ba787f585420aff2d4ee3329a0bf623d00ec73f"} Oct 30 18:30:22.616915 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:22.616474 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889750-xc8x9" Oct 30 18:30:22.616915 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:22.616496 4442 scope.go:122] "RemoveContainer" containerID="2645f1476fb1f21bea2b9472858ab8b6f943e20fdf5c11918382d3c0de18c16f" Oct 30 18:30:22.678452 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:22.678420 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889750-xc8x9"] Oct 30 18:30:22.681744 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:22.681722 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889750-xc8x9"] Oct 30 18:30:22.687871 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:30:22.687837 4442 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc799fcb_604b_46c8_ac4d_5f71fc4aaec1.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc799fcb_604b_46c8_ac4d_5f71fc4aaec1.slice/crio-3455f5cecd88d5ed9eccc1574ba787f585420aff2d4ee3329a0bf623d00ec73f\": RecentStats: unable to find data in memory cache]" Oct 30 18:30:23.097405 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.096560 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2af4a151-5f8a-41e4-acab-a5a09968c2e6" path="/var/lib/kubelet/pods/2af4a151-5f8a-41e4-acab-a5a09968c2e6/volumes" Oct 30 18:30:23.098166 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.097619 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc799fcb-604b-46c8-ac4d-5f71fc4aaec1" path="/var/lib/kubelet/pods/cc799fcb-604b-46c8-ac4d-5f71fc4aaec1/volumes" Oct 30 18:30:23.251432 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.251398 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj"] Oct 30 18:30:23.252059 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.252043 4442 cpu_manager.go:414] "containerMap: removing container" podUID="2af4a151-5f8a-41e4-acab-a5a09968c2e6" containerName="cleanup" Oct 30 18:30:23.252059 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.252061 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2af4a151-5f8a-41e4-acab-a5a09968c2e6" containerName="cleanup" Oct 30 18:30:23.252242 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.252072 4442 cpu_manager.go:414] "containerMap: removing container" podUID="cc799fcb-604b-46c8-ac4d-5f71fc4aaec1" containerName="cleanup" Oct 30 18:30:23.252242 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.252077 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc799fcb-604b-46c8-ac4d-5f71fc4aaec1" containerName="cleanup" Oct 30 18:30:23.252242 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.252090 4442 cpu_manager.go:414] "containerMap: removing container" podUID="cc799fcb-604b-46c8-ac4d-5f71fc4aaec1" containerName="cleanup" Oct 30 18:30:23.252242 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.252095 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc799fcb-604b-46c8-ac4d-5f71fc4aaec1" containerName="cleanup" Oct 30 18:30:23.252242 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.252109 4442 cpu_manager.go:414] "containerMap: removing container" podUID="43c061bf-f41c-4bb8-93ca-fe9a3ee5e690" containerName="istio-proxy" Oct 30 18:30:23.252242 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.252113 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43c061bf-f41c-4bb8-93ca-fe9a3ee5e690" containerName="istio-proxy" Oct 30 18:30:23.252242 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.252125 4442 cpu_manager.go:414] "containerMap: removing container" podUID="2af4a151-5f8a-41e4-acab-a5a09968c2e6" containerName="cleanup" Oct 30 18:30:23.252242 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.252143 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2af4a151-5f8a-41e4-acab-a5a09968c2e6" containerName="cleanup" Oct 30 18:30:23.252242 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.252150 4442 cpu_manager.go:414] "containerMap: removing container" podUID="9e4d3b1c-e383-4db5-aeec-8a6a66d32808" containerName="payload-pre-processing" Oct 30 18:30:23.252242 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.252155 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e4d3b1c-e383-4db5-aeec-8a6a66d32808" containerName="payload-pre-processing" Oct 30 18:30:23.252242 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.252172 4442 cpu_manager.go:414] "containerMap: removing container" podUID="0f1c999a-d8e4-4ee1-8cca-f53f93eed216" containerName="payload-processing" Oct 30 18:30:23.252242 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.252179 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f1c999a-d8e4-4ee1-8cca-f53f93eed216" containerName="payload-processing" Oct 30 18:30:23.252242 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.252200 4442 cpu_manager.go:414] "containerMap: removing container" podUID="43c061bf-f41c-4bb8-93ca-fe9a3ee5e690" containerName="istio-proxy" Oct 30 18:30:23.252242 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.252205 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43c061bf-f41c-4bb8-93ca-fe9a3ee5e690" containerName="istio-proxy" Oct 30 18:30:23.252679 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.252283 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="2af4a151-5f8a-41e4-acab-a5a09968c2e6" containerName="cleanup" Oct 30 18:30:23.252679 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.252292 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e4d3b1c-e383-4db5-aeec-8a6a66d32808" containerName="payload-pre-processing" Oct 30 18:30:23.252679 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.252299 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f1c999a-d8e4-4ee1-8cca-f53f93eed216" containerName="payload-processing" Oct 30 18:30:23.252679 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.252305 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc799fcb-604b-46c8-ac4d-5f71fc4aaec1" containerName="cleanup" Oct 30 18:30:23.252679 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.252312 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="43c061bf-f41c-4bb8-93ca-fe9a3ee5e690" containerName="istio-proxy" Oct 30 18:30:23.252679 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.252318 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="2af4a151-5f8a-41e4-acab-a5a09968c2e6" containerName="cleanup" Oct 30 18:30:23.252679 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.252325 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc799fcb-604b-46c8-ac4d-5f71fc4aaec1" containerName="cleanup" Oct 30 18:30:23.257043 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.257021 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" Oct 30 18:30:23.278006 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.277977 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj"] Oct 30 18:30:23.419159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.417615 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9slt\" (UniqueName: \"kubernetes.io/projected/9760b73c-522d-4255-b72b-c0786527ded6-kube-api-access-w9slt\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj\" (UID: \"9760b73c-522d-4255-b72b-c0786527ded6\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" Oct 30 18:30:23.419159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.417690 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9760b73c-522d-4255-b72b-c0786527ded6-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj\" (UID: \"9760b73c-522d-4255-b72b-c0786527ded6\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" Oct 30 18:30:23.419159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.417754 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9760b73c-522d-4255-b72b-c0786527ded6-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj\" (UID: \"9760b73c-522d-4255-b72b-c0786527ded6\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" Oct 30 18:30:23.419159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.417783 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9760b73c-522d-4255-b72b-c0786527ded6-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj\" (UID: \"9760b73c-522d-4255-b72b-c0786527ded6\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" Oct 30 18:30:23.419159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.417852 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9760b73c-522d-4255-b72b-c0786527ded6-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj\" (UID: \"9760b73c-522d-4255-b72b-c0786527ded6\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" Oct 30 18:30:23.419159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.417904 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9760b73c-522d-4255-b72b-c0786527ded6-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj\" (UID: \"9760b73c-522d-4255-b72b-c0786527ded6\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" Oct 30 18:30:23.419159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.418034 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9760b73c-522d-4255-b72b-c0786527ded6-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj\" (UID: \"9760b73c-522d-4255-b72b-c0786527ded6\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" Oct 30 18:30:23.419159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.418075 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9760b73c-522d-4255-b72b-c0786527ded6-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj\" (UID: \"9760b73c-522d-4255-b72b-c0786527ded6\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" Oct 30 18:30:23.419159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.418103 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9760b73c-522d-4255-b72b-c0786527ded6-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj\" (UID: \"9760b73c-522d-4255-b72b-c0786527ded6\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" Oct 30 18:30:23.521158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.519164 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9760b73c-522d-4255-b72b-c0786527ded6-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj\" (UID: \"9760b73c-522d-4255-b72b-c0786527ded6\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" Oct 30 18:30:23.521158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.519229 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9760b73c-522d-4255-b72b-c0786527ded6-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj\" (UID: \"9760b73c-522d-4255-b72b-c0786527ded6\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" Oct 30 18:30:23.521158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.519261 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9760b73c-522d-4255-b72b-c0786527ded6-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj\" (UID: \"9760b73c-522d-4255-b72b-c0786527ded6\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" Oct 30 18:30:23.521158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.519312 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9slt\" (UniqueName: \"kubernetes.io/projected/9760b73c-522d-4255-b72b-c0786527ded6-kube-api-access-w9slt\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj\" (UID: \"9760b73c-522d-4255-b72b-c0786527ded6\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" Oct 30 18:30:23.521158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.519340 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9760b73c-522d-4255-b72b-c0786527ded6-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj\" (UID: \"9760b73c-522d-4255-b72b-c0786527ded6\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" Oct 30 18:30:23.521158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.519400 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9760b73c-522d-4255-b72b-c0786527ded6-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj\" (UID: \"9760b73c-522d-4255-b72b-c0786527ded6\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" Oct 30 18:30:23.521158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.519427 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9760b73c-522d-4255-b72b-c0786527ded6-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj\" (UID: \"9760b73c-522d-4255-b72b-c0786527ded6\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" Oct 30 18:30:23.521158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.519491 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9760b73c-522d-4255-b72b-c0786527ded6-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj\" (UID: \"9760b73c-522d-4255-b72b-c0786527ded6\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" Oct 30 18:30:23.521158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.519535 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9760b73c-522d-4255-b72b-c0786527ded6-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj\" (UID: \"9760b73c-522d-4255-b72b-c0786527ded6\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" Oct 30 18:30:23.521158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.520342 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9760b73c-522d-4255-b72b-c0786527ded6-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj\" (UID: \"9760b73c-522d-4255-b72b-c0786527ded6\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" Oct 30 18:30:23.521158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.520965 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9760b73c-522d-4255-b72b-c0786527ded6-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj\" (UID: \"9760b73c-522d-4255-b72b-c0786527ded6\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" Oct 30 18:30:23.521886 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.521197 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9760b73c-522d-4255-b72b-c0786527ded6-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj\" (UID: \"9760b73c-522d-4255-b72b-c0786527ded6\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" Oct 30 18:30:23.542162 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.542031 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9760b73c-522d-4255-b72b-c0786527ded6-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj\" (UID: \"9760b73c-522d-4255-b72b-c0786527ded6\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" Oct 30 18:30:23.551295 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.547441 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9760b73c-522d-4255-b72b-c0786527ded6-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj\" (UID: \"9760b73c-522d-4255-b72b-c0786527ded6\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" Oct 30 18:30:23.553686 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.553651 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9slt\" (UniqueName: \"kubernetes.io/projected/9760b73c-522d-4255-b72b-c0786527ded6-kube-api-access-w9slt\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj\" (UID: \"9760b73c-522d-4255-b72b-c0786527ded6\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" Oct 30 18:30:23.554377 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.554352 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9760b73c-522d-4255-b72b-c0786527ded6-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj\" (UID: \"9760b73c-522d-4255-b72b-c0786527ded6\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" Oct 30 18:30:23.559155 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.555301 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9760b73c-522d-4255-b72b-c0786527ded6-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj\" (UID: \"9760b73c-522d-4255-b72b-c0786527ded6\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" Oct 30 18:30:23.576157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.569667 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9760b73c-522d-4255-b72b-c0786527ded6-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj\" (UID: \"9760b73c-522d-4255-b72b-c0786527ded6\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" Oct 30 18:30:23.576440 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.576359 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" Oct 30 18:30:24.002207 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:23.999020 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj"] Oct 30 18:30:24.061483 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:30:24.061430 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9760b73c_522d_4255_b72b_c0786527ded6.slice/crio-03ae4610beaf1d47531aba2b01a9776530008eadb824db1493fbce562f11f983 WatchSource:0}: Error finding container 03ae4610beaf1d47531aba2b01a9776530008eadb824db1493fbce562f11f983: Status 404 returned error can't find the container with id 03ae4610beaf1d47531aba2b01a9776530008eadb824db1493fbce562f11f983 Oct 30 18:30:24.066426 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:24.065656 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:30:24.066426 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:24.065745 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:30:24.066426 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:24.065794 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:30:24.689767 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:24.689726 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" event={"ID":"9760b73c-522d-4255-b72b-c0786527ded6","Type":"ContainerStarted","Data":"efb047674294cc3b828900791f861bb68cc61db6bcb6ca5da05d8bc7469bfad5"} Oct 30 18:30:24.690100 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:24.690077 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" event={"ID":"9760b73c-522d-4255-b72b-c0786527ded6","Type":"ContainerStarted","Data":"03ae4610beaf1d47531aba2b01a9776530008eadb824db1493fbce562f11f983"} Oct 30 18:30:24.778418 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:24.778348 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" podStartSLOduration=1.7783251020000002 podStartE2EDuration="1.778325102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:24.754897525 +0000 UTC m=+1921.844959876" watchObservedRunningTime="2026-10-30 18:30:24.778325102 +0000 UTC m=+1921.868387442" Oct 30 18:30:25.580562 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:25.580472 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" Oct 30 18:30:25.582087 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:25.582059 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 18:30:25.582363 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:25.582333 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" podUID="9760b73c-522d-4255-b72b-c0786527ded6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 18:30:26.582161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:26.581458 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 18:30:26.582161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:26.581519 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" podUID="9760b73c-522d-4255-b72b-c0786527ded6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 18:30:27.577082 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:27.577048 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 18:30:27.577323 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:27.577110 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" podUID="9760b73c-522d-4255-b72b-c0786527ded6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 18:30:28.577557 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:28.577511 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 18:30:28.578065 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:28.577589 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" podUID="9760b73c-522d-4255-b72b-c0786527ded6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 18:30:28.818055 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:28.818025 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-200c58-844699bd54-xg4s6"] Oct 30 18:30:28.818277 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:28.818255 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-200c58-844699bd54-xg4s6" podUID="ddc07105-e8f9-446d-b932-ab76d5471d87" containerName="maas-api" containerID="cri-o://c6641aaaffbfcca844d2de0a231a8c04b8cb94797afbba026a44645cc4dc1dae" gracePeriod=30 Oct 30 18:30:29.499371 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:29.499343 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-200c58-844699bd54-xg4s6" Oct 30 18:30:29.521340 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:29.521302 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddc07105-e8f9-446d-b932-ab76d5471d87-kube-api-access-jtdkr\" (UniqueName: \"kubernetes.io/projected/ddc07105-e8f9-446d-b932-ab76d5471d87-kube-api-access-jtdkr\") pod \"ddc07105-e8f9-446d-b932-ab76d5471d87\" (UID: \"ddc07105-e8f9-446d-b932-ab76d5471d87\") " Oct 30 18:30:29.521525 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:29.521468 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddc07105-e8f9-446d-b932-ab76d5471d87-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddc07105-e8f9-446d-b932-ab76d5471d87-metrics-certs\") pod \"ddc07105-e8f9-446d-b932-ab76d5471d87\" (UID: \"ddc07105-e8f9-446d-b932-ab76d5471d87\") " Oct 30 18:30:29.521525 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:29.521514 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddc07105-e8f9-446d-b932-ab76d5471d87-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddc07105-e8f9-446d-b932-ab76d5471d87-openshift-service-ca\") pod \"ddc07105-e8f9-446d-b932-ab76d5471d87\" (UID: \"ddc07105-e8f9-446d-b932-ab76d5471d87\") " Oct 30 18:30:29.521650 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:29.521541 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddc07105-e8f9-446d-b932-ab76d5471d87-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddc07105-e8f9-446d-b932-ab76d5471d87-maas-api-tls\") pod \"ddc07105-e8f9-446d-b932-ab76d5471d87\" (UID: \"ddc07105-e8f9-446d-b932-ab76d5471d87\") " Oct 30 18:30:29.521940 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:29.521908 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddc07105-e8f9-446d-b932-ab76d5471d87-openshift-service-ca" pod "ddc07105-e8f9-446d-b932-ab76d5471d87" (UID: "ddc07105-e8f9-446d-b932-ab76d5471d87"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:29.543159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:29.531256 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddc07105-e8f9-446d-b932-ab76d5471d87-metrics-certs" pod "ddc07105-e8f9-446d-b932-ab76d5471d87" (UID: "ddc07105-e8f9-446d-b932-ab76d5471d87"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:30:29.543159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:29.531615 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddc07105-e8f9-446d-b932-ab76d5471d87-maas-api-tls" pod "ddc07105-e8f9-446d-b932-ab76d5471d87" (UID: "ddc07105-e8f9-446d-b932-ab76d5471d87"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:30:29.543159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:29.537274 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddc07105-e8f9-446d-b932-ab76d5471d87-kube-api-access-jtdkr" pod "ddc07105-e8f9-446d-b932-ab76d5471d87" (UID: "ddc07105-e8f9-446d-b932-ab76d5471d87"). InnerVolumeSpecName "kube-api-access-jtdkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:29.578869 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:29.578524 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 18:30:29.578869 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:29.578573 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" podUID="9760b73c-522d-4255-b72b-c0786527ded6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 18:30:29.646159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:29.623497 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtdkr\" (UniqueName: \"kubernetes.io/projected/ddc07105-e8f9-446d-b932-ab76d5471d87-kube-api-access-jtdkr\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:30:29.646159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:29.623535 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddc07105-e8f9-446d-b932-ab76d5471d87-metrics-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:30:29.646159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:29.623551 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddc07105-e8f9-446d-b932-ab76d5471d87-openshift-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:30:29.646159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:29.623564 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddc07105-e8f9-446d-b932-ab76d5471d87-maas-api-tls\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:30:29.800213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:29.799709 4442 generic.go:356] "Generic (PLEG): container finished" podID="ddc07105-e8f9-446d-b932-ab76d5471d87" containerID="c6641aaaffbfcca844d2de0a231a8c04b8cb94797afbba026a44645cc4dc1dae" exitCode=0 Oct 30 18:30:29.800213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:29.799916 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-200c58-844699bd54-xg4s6" event={"ID":"ddc07105-e8f9-446d-b932-ab76d5471d87","Type":"ContainerDied","Data":"c6641aaaffbfcca844d2de0a231a8c04b8cb94797afbba026a44645cc4dc1dae"} Oct 30 18:30:29.800213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:29.799969 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-200c58-844699bd54-xg4s6" event={"ID":"ddc07105-e8f9-446d-b932-ab76d5471d87","Type":"ContainerDied","Data":"3057fa0eaa2ceb9d261723336829dc85d804a4c8d33a87b6848eaa52aabd9885"} Oct 30 18:30:29.800213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:29.799997 4442 scope.go:122] "RemoveContainer" containerID="c6641aaaffbfcca844d2de0a231a8c04b8cb94797afbba026a44645cc4dc1dae" Oct 30 18:30:29.800598 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:29.800355 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-200c58-844699bd54-xg4s6" Oct 30 18:30:29.853687 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:29.841740 4442 scope.go:122] "RemoveContainer" containerID="c6641aaaffbfcca844d2de0a231a8c04b8cb94797afbba026a44645cc4dc1dae" Oct 30 18:30:29.853687 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:30:29.842210 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6641aaaffbfcca844d2de0a231a8c04b8cb94797afbba026a44645cc4dc1dae\": container with ID starting with c6641aaaffbfcca844d2de0a231a8c04b8cb94797afbba026a44645cc4dc1dae not found: ID does not exist" containerID="c6641aaaffbfcca844d2de0a231a8c04b8cb94797afbba026a44645cc4dc1dae" Oct 30 18:30:29.853687 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:29.842244 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6641aaaffbfcca844d2de0a231a8c04b8cb94797afbba026a44645cc4dc1dae"} err="failed to get container status \"c6641aaaffbfcca844d2de0a231a8c04b8cb94797afbba026a44645cc4dc1dae\": rpc error: code = NotFound desc = could not find container \"c6641aaaffbfcca844d2de0a231a8c04b8cb94797afbba026a44645cc4dc1dae\": container with ID starting with c6641aaaffbfcca844d2de0a231a8c04b8cb94797afbba026a44645cc4dc1dae not found: ID does not exist" Oct 30 18:30:29.864082 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:29.864051 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-200c58-844699bd54-xg4s6"] Oct 30 18:30:29.867362 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:29.867342 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-200c58-844699bd54-xg4s6"] Oct 30 18:30:30.578079 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:30.578042 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 18:30:30.578309 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:30.578099 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" podUID="9760b73c-522d-4255-b72b-c0786527ded6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 18:30:31.065697 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:31.065662 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddc07105-e8f9-446d-b932-ab76d5471d87" path="/var/lib/kubelet/pods/ddc07105-e8f9-446d-b932-ab76d5471d87/volumes" Oct 30 18:30:31.577121 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:31.577083 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 18:30:31.577314 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:31.577181 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" podUID="9760b73c-522d-4255-b72b-c0786527ded6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 18:30:32.577781 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:32.577743 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 18:30:32.578197 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:32.577796 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" podUID="9760b73c-522d-4255-b72b-c0786527ded6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 18:30:33.646625 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:33.646593 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" Oct 30 18:30:33.648536 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:33.648512 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" Oct 30 18:30:33.653338 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:33.653308 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" Oct 30 18:30:38.257968 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.257939 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw"] Oct 30 18:30:38.259193 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.259169 4442 cpu_manager.go:414] "containerMap: removing container" podUID="ddc07105-e8f9-446d-b932-ab76d5471d87" containerName="maas-api" Oct 30 18:30:38.259193 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.259193 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddc07105-e8f9-446d-b932-ab76d5471d87" containerName="maas-api" Oct 30 18:30:38.259372 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.259218 4442 cpu_manager.go:414] "containerMap: removing container" podUID="cc799fcb-604b-46c8-ac4d-5f71fc4aaec1" containerName="cleanup" Oct 30 18:30:38.259372 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.259223 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc799fcb-604b-46c8-ac4d-5f71fc4aaec1" containerName="cleanup" Oct 30 18:30:38.259372 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.259242 4442 cpu_manager.go:414] "containerMap: removing container" podUID="2af4a151-5f8a-41e4-acab-a5a09968c2e6" containerName="cleanup" Oct 30 18:30:38.259372 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.259247 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2af4a151-5f8a-41e4-acab-a5a09968c2e6" containerName="cleanup" Oct 30 18:30:38.259498 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.259395 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc799fcb-604b-46c8-ac4d-5f71fc4aaec1" containerName="cleanup" Oct 30 18:30:38.259498 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.259411 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddc07105-e8f9-446d-b932-ab76d5471d87" containerName="maas-api" Oct 30 18:30:38.259498 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.259422 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="43c061bf-f41c-4bb8-93ca-fe9a3ee5e690" containerName="istio-proxy" Oct 30 18:30:38.259498 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.259433 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="2af4a151-5f8a-41e4-acab-a5a09968c2e6" containerName="cleanup" Oct 30 18:30:38.266780 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.265458 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" Oct 30 18:30:38.270003 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.269973 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn"] Oct 30 18:30:38.274378 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.274357 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" Oct 30 18:30:38.292954 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.292917 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn"] Oct 30 18:30:38.302196 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.302000 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw"] Oct 30 18:30:38.410970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.406867 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83443c95-ada0-4803-beec-b66c915f5e50-workload-certs\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw\" (UID: \"83443c95-ada0-4803-beec-b66c915f5e50\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" Oct 30 18:30:38.410970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.407026 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd9c7\" (UniqueName: \"kubernetes.io/projected/e2065cdb-c364-4e80-addd-1055ed222931-kube-api-access-cd9c7\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn\" (UID: \"e2065cdb-c364-4e80-addd-1055ed222931\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" Oct 30 18:30:38.410970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.407059 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83443c95-ada0-4803-beec-b66c915f5e50-credential-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw\" (UID: \"83443c95-ada0-4803-beec-b66c915f5e50\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" Oct 30 18:30:38.410970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.407192 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2065cdb-c364-4e80-addd-1055ed222931-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn\" (UID: \"e2065cdb-c364-4e80-addd-1055ed222931\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" Oct 30 18:30:38.410970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.407334 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83443c95-ada0-4803-beec-b66c915f5e50-istiod-ca-cert\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw\" (UID: \"83443c95-ada0-4803-beec-b66c915f5e50\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" Oct 30 18:30:38.410970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.407417 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2065cdb-c364-4e80-addd-1055ed222931-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn\" (UID: \"e2065cdb-c364-4e80-addd-1055ed222931\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" Oct 30 18:30:38.410970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.407514 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2065cdb-c364-4e80-addd-1055ed222931-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn\" (UID: \"e2065cdb-c364-4e80-addd-1055ed222931\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" Oct 30 18:30:38.410970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.407595 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2065cdb-c364-4e80-addd-1055ed222931-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn\" (UID: \"e2065cdb-c364-4e80-addd-1055ed222931\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" Oct 30 18:30:38.410970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.407648 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83443c95-ada0-4803-beec-b66c915f5e50-istio-data\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw\" (UID: \"83443c95-ada0-4803-beec-b66c915f5e50\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" Oct 30 18:30:38.410970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.407741 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2065cdb-c364-4e80-addd-1055ed222931-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn\" (UID: \"e2065cdb-c364-4e80-addd-1055ed222931\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" Oct 30 18:30:38.410970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.407776 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83443c95-ada0-4803-beec-b66c915f5e50-workload-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw\" (UID: \"83443c95-ada0-4803-beec-b66c915f5e50\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" Oct 30 18:30:38.410970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.407824 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2065cdb-c364-4e80-addd-1055ed222931-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn\" (UID: \"e2065cdb-c364-4e80-addd-1055ed222931\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" Oct 30 18:30:38.410970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.407863 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83443c95-ada0-4803-beec-b66c915f5e50-istio-podinfo\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw\" (UID: \"83443c95-ada0-4803-beec-b66c915f5e50\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" Oct 30 18:30:38.410970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.407888 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83443c95-ada0-4803-beec-b66c915f5e50-istio-token\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw\" (UID: \"83443c95-ada0-4803-beec-b66c915f5e50\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" Oct 30 18:30:38.410970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.408003 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2065cdb-c364-4e80-addd-1055ed222931-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn\" (UID: \"e2065cdb-c364-4e80-addd-1055ed222931\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" Oct 30 18:30:38.410970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.408057 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2065cdb-c364-4e80-addd-1055ed222931-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn\" (UID: \"e2065cdb-c364-4e80-addd-1055ed222931\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" Oct 30 18:30:38.410970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.408098 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdrw5\" (UniqueName: \"kubernetes.io/projected/83443c95-ada0-4803-beec-b66c915f5e50-kube-api-access-gdrw5\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw\" (UID: \"83443c95-ada0-4803-beec-b66c915f5e50\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" Oct 30 18:30:38.410970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.408150 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83443c95-ada0-4803-beec-b66c915f5e50-istio-envoy\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw\" (UID: \"83443c95-ada0-4803-beec-b66c915f5e50\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" Oct 30 18:30:38.509899 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.509805 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83443c95-ada0-4803-beec-b66c915f5e50-istiod-ca-cert\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw\" (UID: \"83443c95-ada0-4803-beec-b66c915f5e50\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" Oct 30 18:30:38.510163 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.510144 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2065cdb-c364-4e80-addd-1055ed222931-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn\" (UID: \"e2065cdb-c364-4e80-addd-1055ed222931\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" Oct 30 18:30:38.510320 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.510306 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2065cdb-c364-4e80-addd-1055ed222931-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn\" (UID: \"e2065cdb-c364-4e80-addd-1055ed222931\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" Oct 30 18:30:38.510879 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.510858 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2065cdb-c364-4e80-addd-1055ed222931-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn\" (UID: \"e2065cdb-c364-4e80-addd-1055ed222931\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" Oct 30 18:30:38.511062 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.511047 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83443c95-ada0-4803-beec-b66c915f5e50-istio-data\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw\" (UID: \"83443c95-ada0-4803-beec-b66c915f5e50\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" Oct 30 18:30:38.511195 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.511181 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2065cdb-c364-4e80-addd-1055ed222931-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn\" (UID: \"e2065cdb-c364-4e80-addd-1055ed222931\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" Oct 30 18:30:38.511310 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.511295 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83443c95-ada0-4803-beec-b66c915f5e50-workload-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw\" (UID: \"83443c95-ada0-4803-beec-b66c915f5e50\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" Oct 30 18:30:38.511413 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.511387 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2065cdb-c364-4e80-addd-1055ed222931-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn\" (UID: \"e2065cdb-c364-4e80-addd-1055ed222931\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" Oct 30 18:30:38.511494 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.511408 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2065cdb-c364-4e80-addd-1055ed222931-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn\" (UID: \"e2065cdb-c364-4e80-addd-1055ed222931\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" Oct 30 18:30:38.511604 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.511592 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83443c95-ada0-4803-beec-b66c915f5e50-istio-podinfo\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw\" (UID: \"83443c95-ada0-4803-beec-b66c915f5e50\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" Oct 30 18:30:38.511698 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.511686 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83443c95-ada0-4803-beec-b66c915f5e50-istio-token\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw\" (UID: \"83443c95-ada0-4803-beec-b66c915f5e50\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" Oct 30 18:30:38.511834 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.511784 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2065cdb-c364-4e80-addd-1055ed222931-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn\" (UID: \"e2065cdb-c364-4e80-addd-1055ed222931\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" Oct 30 18:30:38.511928 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.511905 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83443c95-ada0-4803-beec-b66c915f5e50-workload-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw\" (UID: \"83443c95-ada0-4803-beec-b66c915f5e50\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" Oct 30 18:30:38.511993 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.511686 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83443c95-ada0-4803-beec-b66c915f5e50-istio-data\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw\" (UID: \"83443c95-ada0-4803-beec-b66c915f5e50\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" Oct 30 18:30:38.512080 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.510538 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83443c95-ada0-4803-beec-b66c915f5e50-istiod-ca-cert\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw\" (UID: \"83443c95-ada0-4803-beec-b66c915f5e50\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" Oct 30 18:30:38.512493 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.512475 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2065cdb-c364-4e80-addd-1055ed222931-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn\" (UID: \"e2065cdb-c364-4e80-addd-1055ed222931\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" Oct 30 18:30:38.512783 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.512755 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2065cdb-c364-4e80-addd-1055ed222931-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn\" (UID: \"e2065cdb-c364-4e80-addd-1055ed222931\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" Oct 30 18:30:38.512868 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.512815 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdrw5\" (UniqueName: \"kubernetes.io/projected/83443c95-ada0-4803-beec-b66c915f5e50-kube-api-access-gdrw5\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw\" (UID: \"83443c95-ada0-4803-beec-b66c915f5e50\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" Oct 30 18:30:38.512868 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.512854 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83443c95-ada0-4803-beec-b66c915f5e50-istio-envoy\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw\" (UID: \"83443c95-ada0-4803-beec-b66c915f5e50\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" Oct 30 18:30:38.512991 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.512926 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83443c95-ada0-4803-beec-b66c915f5e50-workload-certs\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw\" (UID: \"83443c95-ada0-4803-beec-b66c915f5e50\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" Oct 30 18:30:38.512991 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.512961 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd9c7\" (UniqueName: \"kubernetes.io/projected/e2065cdb-c364-4e80-addd-1055ed222931-kube-api-access-cd9c7\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn\" (UID: \"e2065cdb-c364-4e80-addd-1055ed222931\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" Oct 30 18:30:38.513172 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.512985 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83443c95-ada0-4803-beec-b66c915f5e50-credential-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw\" (UID: \"83443c95-ada0-4803-beec-b66c915f5e50\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" Oct 30 18:30:38.513172 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.513039 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2065cdb-c364-4e80-addd-1055ed222931-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn\" (UID: \"e2065cdb-c364-4e80-addd-1055ed222931\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" Oct 30 18:30:38.513378 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.513360 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2065cdb-c364-4e80-addd-1055ed222931-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn\" (UID: \"e2065cdb-c364-4e80-addd-1055ed222931\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" Oct 30 18:30:38.523079 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.516181 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2065cdb-c364-4e80-addd-1055ed222931-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn\" (UID: \"e2065cdb-c364-4e80-addd-1055ed222931\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" Oct 30 18:30:38.523079 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.516421 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2065cdb-c364-4e80-addd-1055ed222931-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn\" (UID: \"e2065cdb-c364-4e80-addd-1055ed222931\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" Oct 30 18:30:38.523079 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.517476 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83443c95-ada0-4803-beec-b66c915f5e50-workload-certs\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw\" (UID: \"83443c95-ada0-4803-beec-b66c915f5e50\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" Oct 30 18:30:38.523079 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.517714 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83443c95-ada0-4803-beec-b66c915f5e50-credential-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw\" (UID: \"83443c95-ada0-4803-beec-b66c915f5e50\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" Oct 30 18:30:38.523079 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.517928 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2065cdb-c364-4e80-addd-1055ed222931-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn\" (UID: \"e2065cdb-c364-4e80-addd-1055ed222931\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" Oct 30 18:30:38.523079 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.521160 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2065cdb-c364-4e80-addd-1055ed222931-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn\" (UID: \"e2065cdb-c364-4e80-addd-1055ed222931\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" Oct 30 18:30:38.523079 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.522025 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83443c95-ada0-4803-beec-b66c915f5e50-istio-envoy\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw\" (UID: \"83443c95-ada0-4803-beec-b66c915f5e50\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" Oct 30 18:30:38.523079 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.522762 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83443c95-ada0-4803-beec-b66c915f5e50-istio-podinfo\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw\" (UID: \"83443c95-ada0-4803-beec-b66c915f5e50\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" Oct 30 18:30:38.529691 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.529665 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2065cdb-c364-4e80-addd-1055ed222931-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn\" (UID: \"e2065cdb-c364-4e80-addd-1055ed222931\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" Oct 30 18:30:38.532629 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.532596 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83443c95-ada0-4803-beec-b66c915f5e50-istio-token\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw\" (UID: \"83443c95-ada0-4803-beec-b66c915f5e50\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" Oct 30 18:30:38.536798 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.536764 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd9c7\" (UniqueName: \"kubernetes.io/projected/e2065cdb-c364-4e80-addd-1055ed222931-kube-api-access-cd9c7\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn\" (UID: \"e2065cdb-c364-4e80-addd-1055ed222931\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" Oct 30 18:30:38.537635 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.537549 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdrw5\" (UniqueName: \"kubernetes.io/projected/83443c95-ada0-4803-beec-b66c915f5e50-kube-api-access-gdrw5\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw\" (UID: \"83443c95-ada0-4803-beec-b66c915f5e50\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" Oct 30 18:30:38.584395 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.584227 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" Oct 30 18:30:38.600890 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:38.600691 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" Oct 30 18:30:39.403648 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:39.373478 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn"] Oct 30 18:30:39.403648 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:39.394526 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:30:39.403648 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:39.394629 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:30:39.403648 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:39.394671 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:30:39.985159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:39.960778 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw"] Oct 30 18:30:40.055304 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:30:39.998540 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83443c95_ada0_4803_beec_b66c915f5e50.slice/crio-79f77941264e6712acd889f1bb83ae571a9c0a40a24bc23edb36803b2955a6c1 WatchSource:0}: Error finding container 79f77941264e6712acd889f1bb83ae571a9c0a40a24bc23edb36803b2955a6c1: Status 404 returned error can't find the container with id 79f77941264e6712acd889f1bb83ae571a9c0a40a24bc23edb36803b2955a6c1 Oct 30 18:30:40.055304 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:40.005242 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:30:40.055304 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:40.007283 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" podUID="e8ac1217-0b8a-4d72-907f-c70814b60a55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:40.055304 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:40.027279 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:30:40.055304 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:40.027376 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:30:40.055304 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:40.027423 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:30:40.294631 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:40.294576 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" event={"ID":"83443c95-ada0-4803-beec-b66c915f5e50","Type":"ContainerStarted","Data":"c4fdb8c7bbdcb0df1592046e6b68a79ef6cb462bf943db092005658b54eaa50d"} Oct 30 18:30:40.294888 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:40.294646 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" event={"ID":"83443c95-ada0-4803-beec-b66c915f5e50","Type":"ContainerStarted","Data":"79f77941264e6712acd889f1bb83ae571a9c0a40a24bc23edb36803b2955a6c1"} Oct 30 18:30:40.297850 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:40.297819 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" event={"ID":"e2065cdb-c364-4e80-addd-1055ed222931","Type":"ContainerStarted","Data":"ee8ebd7c9d53ab796aaf6e03c26bb9dd23913ba8495b8b9d0f766971d875b8ab"} Oct 30 18:30:40.297990 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:40.297879 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" event={"ID":"e2065cdb-c364-4e80-addd-1055ed222931","Type":"ContainerStarted","Data":"102f688314cf4c3d64f0f676c08f49bd9a98e69074ce5e4e1cb3d6fd77366c3e"} Oct 30 18:30:40.341178 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:40.341093 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" podStartSLOduration=2.341071108 podStartE2EDuration="2.341071108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:40.337319261 +0000 UTC m=+1937.427381598" watchObservedRunningTime="2026-10-30 18:30:40.341071108 +0000 UTC m=+1937.431133448" Oct 30 18:30:40.377596 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:40.377547 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" podStartSLOduration=2.377531 podStartE2EDuration="2.377531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:40.376827458 +0000 UTC m=+1937.466889810" watchObservedRunningTime="2026-10-30 18:30:40.377531 +0000 UTC m=+1937.467593340" Oct 30 18:30:40.585446 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:40.585358 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" Oct 30 18:30:40.601730 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:40.601698 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" Oct 30 18:30:40.603366 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:40.603342 4442 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 18:30:40.603492 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:40.603388 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" podUID="e2065cdb-c364-4e80-addd-1055ed222931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 18:30:41.609345 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:41.609308 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 30 18:30:41.611385 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:41.611347 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" podUID="83443c95-ada0-4803-beec-b66c915f5e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 30 18:30:41.611822 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:41.610246 4442 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 18:30:41.612019 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:41.611997 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" podUID="e2065cdb-c364-4e80-addd-1055ed222931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 18:30:42.585548 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:42.585504 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 30 18:30:42.585734 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:42.585561 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" podUID="83443c95-ada0-4803-beec-b66c915f5e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 30 18:30:42.601837 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:42.601806 4442 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 18:30:42.601999 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:42.601863 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" podUID="e2065cdb-c364-4e80-addd-1055ed222931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 18:30:43.596160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:43.587647 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 30 18:30:43.596160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:43.587731 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" podUID="83443c95-ada0-4803-beec-b66c915f5e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 30 18:30:43.606159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:43.601719 4442 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 18:30:43.606159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:43.601863 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" podUID="e2065cdb-c364-4e80-addd-1055ed222931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 18:30:44.586854 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:44.586818 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 30 18:30:44.587106 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:44.587062 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" podUID="83443c95-ada0-4803-beec-b66c915f5e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 30 18:30:44.601275 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:44.601247 4442 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 18:30:44.601711 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:44.601300 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" podUID="e2065cdb-c364-4e80-addd-1055ed222931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 18:30:45.587160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:45.586566 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 30 18:30:45.587160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:45.586628 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" podUID="83443c95-ada0-4803-beec-b66c915f5e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 30 18:30:45.601445 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:45.601409 4442 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 18:30:45.601910 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:45.601473 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" podUID="e2065cdb-c364-4e80-addd-1055ed222931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 18:30:46.585082 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:46.585048 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 30 18:30:46.585273 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:46.585098 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" podUID="83443c95-ada0-4803-beec-b66c915f5e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 30 18:30:46.601320 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:46.601283 4442 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 18:30:46.601499 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:46.601344 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" podUID="e2065cdb-c364-4e80-addd-1055ed222931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 18:30:47.572268 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:30:47.572220 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:30:47.584957 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:47.584922 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 30 18:30:47.585195 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:47.584983 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" podUID="83443c95-ada0-4803-beec-b66c915f5e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 30 18:30:47.910673 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:47.910630 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" Oct 30 18:30:47.911192 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:47.911050 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" Oct 30 18:30:47.912637 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:47.912614 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" Oct 30 18:30:48.258186 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:48.258069 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm"] Oct 30 18:30:48.266651 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:48.266481 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" Oct 30 18:30:48.283965 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:48.283922 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-ab244c-openshift-default-dockercfg-k9mvg\"" Oct 30 18:30:48.286981 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:48.286956 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm"] Oct 30 18:30:48.330709 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:48.330668 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd55e4f9-4d9d-4fba-a53e-fd3403134906-credential-socket\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm\" (UID: \"cd55e4f9-4d9d-4fba-a53e-fd3403134906\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" Oct 30 18:30:48.330893 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:48.330743 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd55e4f9-4d9d-4fba-a53e-fd3403134906-workload-certs\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm\" (UID: \"cd55e4f9-4d9d-4fba-a53e-fd3403134906\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" Oct 30 18:30:48.330893 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:48.330845 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd55e4f9-4d9d-4fba-a53e-fd3403134906-istio-podinfo\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm\" (UID: \"cd55e4f9-4d9d-4fba-a53e-fd3403134906\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" Oct 30 18:30:48.331032 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:48.330968 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgck5\" (UniqueName: \"kubernetes.io/projected/cd55e4f9-4d9d-4fba-a53e-fd3403134906-kube-api-access-vgck5\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm\" (UID: \"cd55e4f9-4d9d-4fba-a53e-fd3403134906\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" Oct 30 18:30:48.331091 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:48.331056 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd55e4f9-4d9d-4fba-a53e-fd3403134906-istiod-ca-cert\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm\" (UID: \"cd55e4f9-4d9d-4fba-a53e-fd3403134906\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" Oct 30 18:30:48.331211 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:48.331188 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd55e4f9-4d9d-4fba-a53e-fd3403134906-istio-token\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm\" (UID: \"cd55e4f9-4d9d-4fba-a53e-fd3403134906\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" Oct 30 18:30:48.331291 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:48.331261 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd55e4f9-4d9d-4fba-a53e-fd3403134906-workload-socket\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm\" (UID: \"cd55e4f9-4d9d-4fba-a53e-fd3403134906\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" Oct 30 18:30:48.331354 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:48.331301 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd55e4f9-4d9d-4fba-a53e-fd3403134906-istio-data\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm\" (UID: \"cd55e4f9-4d9d-4fba-a53e-fd3403134906\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" Oct 30 18:30:48.331417 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:48.331358 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd55e4f9-4d9d-4fba-a53e-fd3403134906-istio-envoy\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm\" (UID: \"cd55e4f9-4d9d-4fba-a53e-fd3403134906\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" Oct 30 18:30:48.454171 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:48.451870 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd55e4f9-4d9d-4fba-a53e-fd3403134906-workload-certs\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm\" (UID: \"cd55e4f9-4d9d-4fba-a53e-fd3403134906\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" Oct 30 18:30:48.454171 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:48.451924 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd55e4f9-4d9d-4fba-a53e-fd3403134906-istio-podinfo\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm\" (UID: \"cd55e4f9-4d9d-4fba-a53e-fd3403134906\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" Oct 30 18:30:48.454171 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:48.451989 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgck5\" (UniqueName: \"kubernetes.io/projected/cd55e4f9-4d9d-4fba-a53e-fd3403134906-kube-api-access-vgck5\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm\" (UID: \"cd55e4f9-4d9d-4fba-a53e-fd3403134906\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" Oct 30 18:30:48.454171 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:48.452072 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd55e4f9-4d9d-4fba-a53e-fd3403134906-istiod-ca-cert\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm\" (UID: \"cd55e4f9-4d9d-4fba-a53e-fd3403134906\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" Oct 30 18:30:48.454171 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:48.452176 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd55e4f9-4d9d-4fba-a53e-fd3403134906-istio-token\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm\" (UID: \"cd55e4f9-4d9d-4fba-a53e-fd3403134906\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" Oct 30 18:30:48.454171 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:48.452236 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd55e4f9-4d9d-4fba-a53e-fd3403134906-workload-socket\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm\" (UID: \"cd55e4f9-4d9d-4fba-a53e-fd3403134906\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" Oct 30 18:30:48.454171 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:48.452268 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd55e4f9-4d9d-4fba-a53e-fd3403134906-istio-data\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm\" (UID: \"cd55e4f9-4d9d-4fba-a53e-fd3403134906\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" Oct 30 18:30:48.454171 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:48.452330 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd55e4f9-4d9d-4fba-a53e-fd3403134906-istio-envoy\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm\" (UID: \"cd55e4f9-4d9d-4fba-a53e-fd3403134906\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" Oct 30 18:30:48.454171 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:48.452370 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd55e4f9-4d9d-4fba-a53e-fd3403134906-credential-socket\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm\" (UID: \"cd55e4f9-4d9d-4fba-a53e-fd3403134906\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" Oct 30 18:30:48.454171 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:48.452846 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd55e4f9-4d9d-4fba-a53e-fd3403134906-credential-socket\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm\" (UID: \"cd55e4f9-4d9d-4fba-a53e-fd3403134906\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" Oct 30 18:30:48.454171 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:48.453110 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd55e4f9-4d9d-4fba-a53e-fd3403134906-workload-certs\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm\" (UID: \"cd55e4f9-4d9d-4fba-a53e-fd3403134906\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" Oct 30 18:30:48.473332 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:48.463327 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd55e4f9-4d9d-4fba-a53e-fd3403134906-istio-data\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm\" (UID: \"cd55e4f9-4d9d-4fba-a53e-fd3403134906\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" Oct 30 18:30:48.473332 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:48.463596 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd55e4f9-4d9d-4fba-a53e-fd3403134906-workload-socket\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm\" (UID: \"cd55e4f9-4d9d-4fba-a53e-fd3403134906\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" Oct 30 18:30:48.473332 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:48.463598 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd55e4f9-4d9d-4fba-a53e-fd3403134906-istiod-ca-cert\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm\" (UID: \"cd55e4f9-4d9d-4fba-a53e-fd3403134906\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" Oct 30 18:30:48.485217 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:48.474337 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd55e4f9-4d9d-4fba-a53e-fd3403134906-istio-podinfo\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm\" (UID: \"cd55e4f9-4d9d-4fba-a53e-fd3403134906\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" Oct 30 18:30:48.514436 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:48.514327 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd55e4f9-4d9d-4fba-a53e-fd3403134906-istio-envoy\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm\" (UID: \"cd55e4f9-4d9d-4fba-a53e-fd3403134906\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" Oct 30 18:30:48.537525 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:48.535847 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd55e4f9-4d9d-4fba-a53e-fd3403134906-istio-token\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm\" (UID: \"cd55e4f9-4d9d-4fba-a53e-fd3403134906\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" Oct 30 18:30:48.571179 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:48.570623 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgck5\" (UniqueName: \"kubernetes.io/projected/cd55e4f9-4d9d-4fba-a53e-fd3403134906-kube-api-access-vgck5\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm\" (UID: \"cd55e4f9-4d9d-4fba-a53e-fd3403134906\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" Oct 30 18:30:48.600159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:48.592914 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" Oct 30 18:30:48.650588 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:48.650323 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" Oct 30 18:30:48.651976 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:48.651908 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" Oct 30 18:30:48.653056 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:48.653021 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" Oct 30 18:30:48.961922 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:30:48.961891 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd55e4f9_4d9d_4fba_a53e_fd3403134906.slice/crio-0f7749592ef66f80457589258fc9491b6527fb22177c2d718ce7e2aec24d2a3e WatchSource:0}: Error finding container 0f7749592ef66f80457589258fc9491b6527fb22177c2d718ce7e2aec24d2a3e: Status 404 returned error can't find the container with id 0f7749592ef66f80457589258fc9491b6527fb22177c2d718ce7e2aec24d2a3e Oct 30 18:30:48.962434 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:48.962402 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm"] Oct 30 18:30:48.964381 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:48.964350 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:30:48.964456 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:48.964413 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:30:48.964456 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:48.964442 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:30:49.538930 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:49.538887 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" event={"ID":"cd55e4f9-4d9d-4fba-a53e-fd3403134906","Type":"ContainerStarted","Data":"4b6fff8d24713f5365a2e0787679ac6779d2151c989a6708a3b1b6ce309d68bb"} Oct 30 18:30:49.539183 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:49.538952 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" event={"ID":"cd55e4f9-4d9d-4fba-a53e-fd3403134906","Type":"ContainerStarted","Data":"0f7749592ef66f80457589258fc9491b6527fb22177c2d718ce7e2aec24d2a3e"} Oct 30 18:30:49.594418 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:49.594389 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" Oct 30 18:30:49.604084 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:49.604036 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" podStartSLOduration=1.604021066 podStartE2EDuration="1.604021066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:49.590893436 +0000 UTC m=+1946.680955775" watchObservedRunningTime="2026-10-30 18:30:49.604021066 +0000 UTC m=+1946.694083404" Oct 30 18:30:50.599904 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:50.599873 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" Oct 30 18:30:51.577510 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:51.577433 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" Oct 30 18:30:51.578790 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:51.578764 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" Oct 30 18:30:53.822680 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:53.822647 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab244c-7659db958c-jbckm"] Oct 30 18:30:53.827024 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:53.827002 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab244c-7659db958c-jbckm" Oct 30 18:30:53.845755 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:53.845726 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab244c-7659db958c-jbckm"] Oct 30 18:30:53.852818 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:53.852795 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-ab244c\"" Oct 30 18:30:54.021157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:54.012889 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-ab244c-dockercfg-h5z7n\"" Oct 30 18:30:54.078912 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:54.078516 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr7m4\" (UniqueName: \"kubernetes.io/projected/a1352e66-acc1-499e-8344-4cab5734f78a-kube-api-access-tr7m4\") pod \"payload-pre-processing-e2e-shared-a-w3-ab244c-7659db958c-jbckm\" (UID: \"a1352e66-acc1-499e-8344-4cab5734f78a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab244c-7659db958c-jbckm" Oct 30 18:30:54.089053 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:54.089019 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1352e66-acc1-499e-8344-4cab5734f78a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-ab244c-7659db958c-jbckm\" (UID: \"a1352e66-acc1-499e-8344-4cab5734f78a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab244c-7659db958c-jbckm" Oct 30 18:30:54.191023 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:54.190986 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1352e66-acc1-499e-8344-4cab5734f78a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-ab244c-7659db958c-jbckm\" (UID: \"a1352e66-acc1-499e-8344-4cab5734f78a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab244c-7659db958c-jbckm" Oct 30 18:30:54.191245 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:54.191170 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr7m4\" (UniqueName: \"kubernetes.io/projected/a1352e66-acc1-499e-8344-4cab5734f78a-kube-api-access-tr7m4\") pod \"payload-pre-processing-e2e-shared-a-w3-ab244c-7659db958c-jbckm\" (UID: \"a1352e66-acc1-499e-8344-4cab5734f78a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab244c-7659db958c-jbckm" Oct 30 18:30:54.191930 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:54.191908 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1352e66-acc1-499e-8344-4cab5734f78a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-ab244c-7659db958c-jbckm\" (UID: \"a1352e66-acc1-499e-8344-4cab5734f78a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab244c-7659db958c-jbckm" Oct 30 18:30:54.212582 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:54.212530 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr7m4\" (UniqueName: \"kubernetes.io/projected/a1352e66-acc1-499e-8344-4cab5734f78a-kube-api-access-tr7m4\") pod \"payload-pre-processing-e2e-shared-a-w3-ab244c-7659db958c-jbckm\" (UID: \"a1352e66-acc1-499e-8344-4cab5734f78a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab244c-7659db958c-jbckm" Oct 30 18:30:54.442800 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:54.442764 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab244c-7659db958c-jbckm" Oct 30 18:30:54.760012 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:30:54.759981 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1352e66_acc1_499e_8344_4cab5734f78a.slice/crio-f2318c14e16bb8f6ca10815d326258d939ba5100976968933a4ff21c2444abd3 WatchSource:0}: Error finding container f2318c14e16bb8f6ca10815d326258d939ba5100976968933a4ff21c2444abd3: Status 404 returned error can't find the container with id f2318c14e16bb8f6ca10815d326258d939ba5100976968933a4ff21c2444abd3 Oct 30 18:30:54.760260 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:54.760239 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab244c-7659db958c-jbckm"] Oct 30 18:30:55.608495 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:55.608459 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab244c-7659db958c-jbckm" event={"ID":"a1352e66-acc1-499e-8344-4cab5734f78a","Type":"ContainerStarted","Data":"c9e6bda211e4c3e7ae4b6d0e8d2cf75080afa9d1c0858c5baa595dfaae447bf9"} Oct 30 18:30:55.608974 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:55.608522 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab244c-7659db958c-jbckm" event={"ID":"a1352e66-acc1-499e-8344-4cab5734f78a","Type":"ContainerStarted","Data":"f2318c14e16bb8f6ca10815d326258d939ba5100976968933a4ff21c2444abd3"} Oct 30 18:30:55.608974 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:55.608551 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab244c-7659db958c-jbckm" Oct 30 18:30:55.647899 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:30:55.647841 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab244c-7659db958c-jbckm" podStartSLOduration=2.6478272179999998 podStartE2EDuration="2.647827218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:55.646400812 +0000 UTC m=+1952.736463152" watchObservedRunningTime="2026-10-30 18:30:55.647827218 +0000 UTC m=+1952.737889556" Oct 30 18:31:02.589158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:02.588230 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:31:02.589158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:02.588296 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" podUID="7084397f-85eb-43f9-8314-530af31c81c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:06.617976 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:06.617945 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab244c-7659db958c-jbckm" Oct 30 18:31:07.643456 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:07.643420 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4"] Oct 30 18:31:07.650715 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:07.650686 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" Oct 30 18:31:07.670362 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:07.670331 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-istio-data\") pod \"e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4\" (UID: \"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54\") " pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" Oct 30 18:31:07.670514 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:07.670370 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-workload-socket\") pod \"e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4\" (UID: \"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54\") " pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" Oct 30 18:31:07.670560 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:07.670539 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-workload-certs\") pod \"e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4\" (UID: \"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54\") " pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" Oct 30 18:31:07.670628 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:07.670612 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-credential-socket\") pod \"e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4\" (UID: \"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54\") " pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" Oct 30 18:31:07.670669 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:07.670646 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-istiod-ca-cert\") pod \"e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4\" (UID: \"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54\") " pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" Oct 30 18:31:07.670766 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:07.670744 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv4gk\" (UniqueName: \"kubernetes.io/projected/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-kube-api-access-qv4gk\") pod \"e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4\" (UID: \"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54\") " pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" Oct 30 18:31:07.670804 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:07.670784 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-istio-token\") pod \"e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4\" (UID: \"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54\") " pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" Oct 30 18:31:07.670842 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:07.670821 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-istio-podinfo\") pod \"e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4\" (UID: \"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54\") " pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" Oct 30 18:31:07.670944 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:07.670928 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-istio-envoy\") pod \"e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4\" (UID: \"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54\") " pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" Oct 30 18:31:07.680749 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:07.680716 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4"] Oct 30 18:31:07.686830 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:07.686797 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-675070-openshift-default-dockercfg-nvjvv\"" Oct 30 18:31:07.777279 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:07.777228 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-istio-data\") pod \"e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4\" (UID: \"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54\") " pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" Oct 30 18:31:07.806399 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:07.806367 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-workload-socket\") pod \"e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4\" (UID: \"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54\") " pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" Oct 30 18:31:07.816192 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:07.810596 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-workload-certs\") pod \"e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4\" (UID: \"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54\") " pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" Oct 30 18:31:07.816192 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:07.810666 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-credential-socket\") pod \"e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4\" (UID: \"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54\") " pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" Oct 30 18:31:07.816192 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:07.810711 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-istiod-ca-cert\") pod \"e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4\" (UID: \"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54\") " pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" Oct 30 18:31:07.816192 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:07.810844 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv4gk\" (UniqueName: \"kubernetes.io/projected/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-kube-api-access-qv4gk\") pod \"e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4\" (UID: \"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54\") " pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" Oct 30 18:31:07.816192 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:07.810892 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-istio-token\") pod \"e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4\" (UID: \"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54\") " pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" Oct 30 18:31:07.816192 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:07.810920 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-istio-podinfo\") pod \"e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4\" (UID: \"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54\") " pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" Oct 30 18:31:07.816192 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:07.811059 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-istio-envoy\") pod \"e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4\" (UID: \"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54\") " pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" Oct 30 18:31:07.817868 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:07.817838 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-credential-socket\") pod \"e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4\" (UID: \"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54\") " pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" Oct 30 18:31:07.833220 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:07.783630 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-istio-data\") pod \"e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4\" (UID: \"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54\") " pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" Oct 30 18:31:07.833461 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:07.832742 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-workload-certs\") pod \"e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4\" (UID: \"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54\") " pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" Oct 30 18:31:07.833561 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:07.807010 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-workload-socket\") pod \"e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4\" (UID: \"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54\") " pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" Oct 30 18:31:07.834702 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:07.834673 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-istiod-ca-cert\") pod \"e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4\" (UID: \"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54\") " pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" Oct 30 18:31:07.866171 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:07.846836 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-istio-podinfo\") pod \"e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4\" (UID: \"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54\") " pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" Oct 30 18:31:07.912211 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:07.898956 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-istio-token\") pod \"e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4\" (UID: \"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54\") " pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" Oct 30 18:31:07.912211 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:07.899957 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv4gk\" (UniqueName: \"kubernetes.io/projected/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-kube-api-access-qv4gk\") pod \"e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4\" (UID: \"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54\") " pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" Oct 30 18:31:07.943209 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:07.914227 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-istio-envoy\") pod \"e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4\" (UID: \"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54\") " pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" Oct 30 18:31:08.004937 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:08.004613 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" Oct 30 18:31:08.208111 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:08.208018 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c"] Oct 30 18:31:08.208784 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:08.208723 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" podUID="dab58a66-301a-4e74-9375-95bd4964b644" containerName="istio-proxy" containerID="cri-o://9ed8fd43e99ee5f67663526a917e7077855e8895ca1e022354be284f255da881" gracePeriod=30 Oct 30 18:31:08.378450 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:08.378419 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4"] Oct 30 18:31:08.379443 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:31:08.379416 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bf25ed4_7f2c_4b1b_ad43_e38b37d9ab54.slice/crio-d10f6dcfe04e75e9d975292928cee936a1e3abe930acc4c85cc311e5e9613eef WatchSource:0}: Error finding container d10f6dcfe04e75e9d975292928cee936a1e3abe930acc4c85cc311e5e9613eef: Status 404 returned error can't find the container with id d10f6dcfe04e75e9d975292928cee936a1e3abe930acc4c85cc311e5e9613eef Oct 30 18:31:08.382179 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:08.382116 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:31:08.382275 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:08.382237 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:31:08.382275 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:08.382270 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:31:08.961706 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:08.961664 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" event={"ID":"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54","Type":"ContainerStarted","Data":"428227e6c5a7d68f2d063fd873665ee979ec1ec52df3f48dd50ed5b36d64cd24"} Oct 30 18:31:08.962209 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:08.961734 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" event={"ID":"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54","Type":"ContainerStarted","Data":"d10f6dcfe04e75e9d975292928cee936a1e3abe930acc4c85cc311e5e9613eef"} Oct 30 18:31:09.006229 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:09.006181 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" Oct 30 18:31:09.007006 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:09.006961 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" podStartSLOduration=2.006946838 podStartE2EDuration="2.006946838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:31:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:31:09.003837759 +0000 UTC m=+1966.093900100" watchObservedRunningTime="2026-10-30 18:31:09.006946838 +0000 UTC m=+1966.097009176" Oct 30 18:31:09.011993 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:09.011971 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" Oct 30 18:31:09.969538 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:09.969507 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" Oct 30 18:31:09.970487 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:09.970469 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" Oct 30 18:31:13.460224 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.460181 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c"] Oct 30 18:31:13.467314 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.467275 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" Oct 30 18:31:13.496569 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.496493 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c"] Oct 30 18:31:13.497752 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.497727 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-1a2436-openshift-default-dockercfg-qrht6\"" Oct 30 18:31:13.538191 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.531931 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-istio-envoy\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c\" (UID: \"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" Oct 30 18:31:13.538191 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.531981 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-istio-podinfo\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c\" (UID: \"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" Oct 30 18:31:13.538191 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.532118 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-credential-socket\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c\" (UID: \"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" Oct 30 18:31:13.538191 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.532220 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt9lb\" (UniqueName: \"kubernetes.io/projected/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-kube-api-access-gt9lb\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c\" (UID: \"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" Oct 30 18:31:13.538191 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.532291 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-istio-token\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c\" (UID: \"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" Oct 30 18:31:13.538191 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.532383 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-workload-socket\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c\" (UID: \"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" Oct 30 18:31:13.538191 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.532431 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-istio-data\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c\" (UID: \"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" Oct 30 18:31:13.538191 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.532551 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-workload-certs\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c\" (UID: \"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" Oct 30 18:31:13.538191 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.532768 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-istiod-ca-cert\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c\" (UID: \"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" Oct 30 18:31:13.547422 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.547391 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" Oct 30 18:31:13.653169 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.651107 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-istio-envoy\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c\" (UID: \"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" Oct 30 18:31:13.653169 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.651177 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-istio-podinfo\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c\" (UID: \"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" Oct 30 18:31:13.653169 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.651218 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-credential-socket\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c\" (UID: \"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" Oct 30 18:31:13.653169 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.651255 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt9lb\" (UniqueName: \"kubernetes.io/projected/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-kube-api-access-gt9lb\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c\" (UID: \"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" Oct 30 18:31:13.653169 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.651282 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-istio-token\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c\" (UID: \"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" Oct 30 18:31:13.653169 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.651306 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-workload-socket\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c\" (UID: \"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" Oct 30 18:31:13.653169 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.651345 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-istio-data\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c\" (UID: \"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" Oct 30 18:31:13.653169 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.651396 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-workload-certs\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c\" (UID: \"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" Oct 30 18:31:13.653169 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.651574 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-istiod-ca-cert\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c\" (UID: \"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" Oct 30 18:31:13.653169 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.652416 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-istiod-ca-cert\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c\" (UID: \"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" Oct 30 18:31:13.663398 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.662333 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-credential-socket\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c\" (UID: \"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" Oct 30 18:31:13.663398 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.662580 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-istio-data\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c\" (UID: \"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" Oct 30 18:31:13.663398 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.662832 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-workload-socket\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c\" (UID: \"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" Oct 30 18:31:13.663398 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.663041 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-workload-certs\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c\" (UID: \"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" Oct 30 18:31:13.694037 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.693461 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-istio-envoy\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c\" (UID: \"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" Oct 30 18:31:13.716169 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.715370 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-istio-podinfo\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c\" (UID: \"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" Oct 30 18:31:13.717203 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.717170 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-istio-token\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c\" (UID: \"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" Oct 30 18:31:13.718908 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.718882 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt9lb\" (UniqueName: \"kubernetes.io/projected/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-kube-api-access-gt9lb\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c\" (UID: \"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" Oct 30 18:31:13.756210 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.753919 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dab58a66-301a-4e74-9375-95bd4964b644-kube-api-access-8hgnf\" (UniqueName: \"kubernetes.io/projected/dab58a66-301a-4e74-9375-95bd4964b644-kube-api-access-8hgnf\") pod \"dab58a66-301a-4e74-9375-95bd4964b644\" (UID: \"dab58a66-301a-4e74-9375-95bd4964b644\") " Oct 30 18:31:13.756210 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.754028 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dab58a66-301a-4e74-9375-95bd4964b644-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab58a66-301a-4e74-9375-95bd4964b644-workload-socket\") pod \"dab58a66-301a-4e74-9375-95bd4964b644\" (UID: \"dab58a66-301a-4e74-9375-95bd4964b644\") " Oct 30 18:31:13.756210 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.754178 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dab58a66-301a-4e74-9375-95bd4964b644-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab58a66-301a-4e74-9375-95bd4964b644-credential-socket\") pod \"dab58a66-301a-4e74-9375-95bd4964b644\" (UID: \"dab58a66-301a-4e74-9375-95bd4964b644\") " Oct 30 18:31:13.756210 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.754253 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dab58a66-301a-4e74-9375-95bd4964b644-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dab58a66-301a-4e74-9375-95bd4964b644-istio-data\") pod \"dab58a66-301a-4e74-9375-95bd4964b644\" (UID: \"dab58a66-301a-4e74-9375-95bd4964b644\") " Oct 30 18:31:13.756210 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.754294 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dab58a66-301a-4e74-9375-95bd4964b644-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dab58a66-301a-4e74-9375-95bd4964b644-istio-envoy\") pod \"dab58a66-301a-4e74-9375-95bd4964b644\" (UID: \"dab58a66-301a-4e74-9375-95bd4964b644\") " Oct 30 18:31:13.756210 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.754364 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dab58a66-301a-4e74-9375-95bd4964b644-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dab58a66-301a-4e74-9375-95bd4964b644-istiod-ca-cert\") pod \"dab58a66-301a-4e74-9375-95bd4964b644\" (UID: \"dab58a66-301a-4e74-9375-95bd4964b644\") " Oct 30 18:31:13.756210 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.754431 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dab58a66-301a-4e74-9375-95bd4964b644-istio-token\" (UniqueName: \"kubernetes.io/projected/dab58a66-301a-4e74-9375-95bd4964b644-istio-token\") pod \"dab58a66-301a-4e74-9375-95bd4964b644\" (UID: \"dab58a66-301a-4e74-9375-95bd4964b644\") " Oct 30 18:31:13.756210 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.754475 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dab58a66-301a-4e74-9375-95bd4964b644-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dab58a66-301a-4e74-9375-95bd4964b644-workload-certs\") pod \"dab58a66-301a-4e74-9375-95bd4964b644\" (UID: \"dab58a66-301a-4e74-9375-95bd4964b644\") " Oct 30 18:31:13.756210 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.754552 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dab58a66-301a-4e74-9375-95bd4964b644-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dab58a66-301a-4e74-9375-95bd4964b644-istio-podinfo\") pod \"dab58a66-301a-4e74-9375-95bd4964b644\" (UID: \"dab58a66-301a-4e74-9375-95bd4964b644\") " Oct 30 18:31:13.777725 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.777378 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dab58a66-301a-4e74-9375-95bd4964b644-istiod-ca-cert" pod "dab58a66-301a-4e74-9375-95bd4964b644" (UID: "dab58a66-301a-4e74-9375-95bd4964b644"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:31:13.793421 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.793366 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dab58a66-301a-4e74-9375-95bd4964b644-istio-data" pod "dab58a66-301a-4e74-9375-95bd4964b644" (UID: "dab58a66-301a-4e74-9375-95bd4964b644"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:31:13.795826 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.795545 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" Oct 30 18:31:13.797367 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.797328 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dab58a66-301a-4e74-9375-95bd4964b644-kube-api-access-8hgnf" pod "dab58a66-301a-4e74-9375-95bd4964b644" (UID: "dab58a66-301a-4e74-9375-95bd4964b644"). InnerVolumeSpecName "kube-api-access-8hgnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:31:13.797517 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.797445 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dab58a66-301a-4e74-9375-95bd4964b644-istio-token" pod "dab58a66-301a-4e74-9375-95bd4964b644" (UID: "dab58a66-301a-4e74-9375-95bd4964b644"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:31:13.868496 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.801273 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dab58a66-301a-4e74-9375-95bd4964b644-credential-socket" pod "dab58a66-301a-4e74-9375-95bd4964b644" (UID: "dab58a66-301a-4e74-9375-95bd4964b644"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:31:13.868496 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.807379 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dab58a66-301a-4e74-9375-95bd4964b644-workload-socket" pod "dab58a66-301a-4e74-9375-95bd4964b644" (UID: "dab58a66-301a-4e74-9375-95bd4964b644"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:31:13.868496 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.807461 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dab58a66-301a-4e74-9375-95bd4964b644-workload-certs" pod "dab58a66-301a-4e74-9375-95bd4964b644" (UID: "dab58a66-301a-4e74-9375-95bd4964b644"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:31:13.868496 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.807877 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dab58a66-301a-4e74-9375-95bd4964b644-istio-podinfo" pod "dab58a66-301a-4e74-9375-95bd4964b644" (UID: "dab58a66-301a-4e74-9375-95bd4964b644"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:31:13.868496 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.846356 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dab58a66-301a-4e74-9375-95bd4964b644-istio-envoy" pod "dab58a66-301a-4e74-9375-95bd4964b644" (UID: "dab58a66-301a-4e74-9375-95bd4964b644"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:31:13.868496 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.857192 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dab58a66-301a-4e74-9375-95bd4964b644-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:31:13.868496 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.857219 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hgnf\" (UniqueName: \"kubernetes.io/projected/dab58a66-301a-4e74-9375-95bd4964b644-kube-api-access-8hgnf\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:31:13.868496 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.857234 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab58a66-301a-4e74-9375-95bd4964b644-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:31:13.868496 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.857248 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab58a66-301a-4e74-9375-95bd4964b644-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:31:13.868496 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.857261 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dab58a66-301a-4e74-9375-95bd4964b644-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:31:13.868496 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.857274 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dab58a66-301a-4e74-9375-95bd4964b644-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:31:13.868496 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.857285 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dab58a66-301a-4e74-9375-95bd4964b644-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:31:13.868496 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.857296 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dab58a66-301a-4e74-9375-95bd4964b644-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:31:13.868496 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:13.857308 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dab58a66-301a-4e74-9375-95bd4964b644-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:31:14.003572 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:14.003413 4442 generic.go:356] "Generic (PLEG): container finished" podID="dab58a66-301a-4e74-9375-95bd4964b644" containerID="9ed8fd43e99ee5f67663526a917e7077855e8895ca1e022354be284f255da881" exitCode=0 Oct 30 18:31:14.003572 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:14.003463 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" event={"ID":"dab58a66-301a-4e74-9375-95bd4964b644","Type":"ContainerDied","Data":"9ed8fd43e99ee5f67663526a917e7077855e8895ca1e022354be284f255da881"} Oct 30 18:31:14.003572 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:14.003506 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" event={"ID":"dab58a66-301a-4e74-9375-95bd4964b644","Type":"ContainerDied","Data":"353a89117e65871c736bf6ddddea1fca29ea6e06a00020b571695a6846d5e300"} Oct 30 18:31:14.003572 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:14.003527 4442 scope.go:122] "RemoveContainer" containerID="9ed8fd43e99ee5f67663526a917e7077855e8895ca1e022354be284f255da881" Oct 30 18:31:14.003856 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:14.003653 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c" Oct 30 18:31:14.029759 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:14.029737 4442 scope.go:122] "RemoveContainer" containerID="9ed8fd43e99ee5f67663526a917e7077855e8895ca1e022354be284f255da881" Oct 30 18:31:14.030562 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:31:14.030117 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ed8fd43e99ee5f67663526a917e7077855e8895ca1e022354be284f255da881\": container with ID starting with 9ed8fd43e99ee5f67663526a917e7077855e8895ca1e022354be284f255da881 not found: ID does not exist" containerID="9ed8fd43e99ee5f67663526a917e7077855e8895ca1e022354be284f255da881" Oct 30 18:31:14.030562 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:14.030185 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ed8fd43e99ee5f67663526a917e7077855e8895ca1e022354be284f255da881"} err="failed to get container status \"9ed8fd43e99ee5f67663526a917e7077855e8895ca1e022354be284f255da881\": rpc error: code = NotFound desc = could not find container \"9ed8fd43e99ee5f67663526a917e7077855e8895ca1e022354be284f255da881\": container with ID starting with 9ed8fd43e99ee5f67663526a917e7077855e8895ca1e022354be284f255da881 not found: ID does not exist" Oct 30 18:31:14.075398 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:14.075369 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c"] Oct 30 18:31:14.079240 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:14.079210 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vlm9c"] Oct 30 18:31:14.253928 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:14.253799 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c"] Oct 30 18:31:14.253928 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:31:14.253908 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb9a48e1_6d20_4f44_8dc6_eb0e6125276d.slice/crio-dc579ab93cffb95cd2697122cf4b4861b4b0b343f1e28ea4665f04e06dd42bd5 WatchSource:0}: Error finding container dc579ab93cffb95cd2697122cf4b4861b4b0b343f1e28ea4665f04e06dd42bd5: Status 404 returned error can't find the container with id dc579ab93cffb95cd2697122cf4b4861b4b0b343f1e28ea4665f04e06dd42bd5 Oct 30 18:31:14.258465 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:14.258434 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:31:14.258556 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:14.258513 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:31:14.258617 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:14.258560 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:31:15.013025 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:15.012979 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" event={"ID":"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d","Type":"ContainerStarted","Data":"6d4813a4d8e1241c4cecbf9c43bbf423127bd8b4f6fc0bff157e1e5dddc1868b"} Oct 30 18:31:15.013471 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:15.013041 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" event={"ID":"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d","Type":"ContainerStarted","Data":"dc579ab93cffb95cd2697122cf4b4861b4b0b343f1e28ea4665f04e06dd42bd5"} Oct 30 18:31:15.059157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:15.058595 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" podStartSLOduration=2.058575996 podStartE2EDuration="2.058575996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:31:15.0562026 +0000 UTC m=+1972.146264940" watchObservedRunningTime="2026-10-30 18:31:15.058575996 +0000 UTC m=+1972.148638338" Oct 30 18:31:15.070658 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:15.070192 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dab58a66-301a-4e74-9375-95bd4964b644" path="/var/lib/kubelet/pods/dab58a66-301a-4e74-9375-95bd4964b644/volumes" Oct 30 18:31:15.798612 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:15.798579 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" Oct 30 18:31:15.804459 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:15.804433 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" Oct 30 18:31:16.020437 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:16.020409 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" Oct 30 18:31:16.021506 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:16.021481 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" Oct 30 18:31:17.571569 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:31:17.571530 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:31:20.013172 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:20.013112 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq"] Oct 30 18:31:20.014871 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:20.014723 4442 cpu_manager.go:414] "containerMap: removing container" podUID="dab58a66-301a-4e74-9375-95bd4964b644" containerName="istio-proxy" Oct 30 18:31:20.015058 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:20.015038 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dab58a66-301a-4e74-9375-95bd4964b644" containerName="istio-proxy" Oct 30 18:31:20.015351 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:20.015333 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="dab58a66-301a-4e74-9375-95bd4964b644" containerName="istio-proxy" Oct 30 18:31:20.029466 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:20.029440 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" Oct 30 18:31:20.080205 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:20.067721 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-e95c51-openshift-default-dockercfg-t2rck\"" Oct 30 18:31:20.080417 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:20.080239 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-credential-socket\") pod \"e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq\" (UID: \"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0\") " pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" Oct 30 18:31:20.080417 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:20.080327 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-istio-podinfo\") pod \"e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq\" (UID: \"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0\") " pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" Oct 30 18:31:20.080574 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:20.080417 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-istio-envoy\") pod \"e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq\" (UID: \"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0\") " pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" Oct 30 18:31:20.080574 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:20.080513 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-istio-data\") pod \"e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq\" (UID: \"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0\") " pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" Oct 30 18:31:20.080574 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:20.080553 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfl4f\" (UniqueName: \"kubernetes.io/projected/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-kube-api-access-cfl4f\") pod \"e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq\" (UID: \"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0\") " pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" Oct 30 18:31:20.080755 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:20.080579 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-workload-certs\") pod \"e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq\" (UID: \"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0\") " pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" Oct 30 18:31:20.080755 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:20.080620 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-workload-socket\") pod \"e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq\" (UID: \"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0\") " pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" Oct 30 18:31:20.080755 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:20.080666 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-istiod-ca-cert\") pod \"e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq\" (UID: \"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0\") " pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" Oct 30 18:31:20.080755 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:20.080751 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-istio-token\") pod \"e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq\" (UID: \"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0\") " pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" Oct 30 18:31:20.118957 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:20.118900 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq"] Oct 30 18:31:20.185164 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:20.182527 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-istiod-ca-cert\") pod \"e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq\" (UID: \"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0\") " pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" Oct 30 18:31:20.185164 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:20.182627 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-istio-token\") pod \"e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq\" (UID: \"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0\") " pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" Oct 30 18:31:20.185164 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:20.182679 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-credential-socket\") pod \"e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq\" (UID: \"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0\") " pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" Oct 30 18:31:20.185164 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:20.182763 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-istio-podinfo\") pod \"e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq\" (UID: \"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0\") " pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" Oct 30 18:31:20.185164 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:20.182847 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-istio-envoy\") pod \"e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq\" (UID: \"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0\") " pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" Oct 30 18:31:20.185164 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:20.182933 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-istio-data\") pod \"e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq\" (UID: \"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0\") " pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" Oct 30 18:31:20.185164 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:20.182975 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfl4f\" (UniqueName: \"kubernetes.io/projected/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-kube-api-access-cfl4f\") pod \"e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq\" (UID: \"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0\") " pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" Oct 30 18:31:20.185164 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:20.183002 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-workload-certs\") pod \"e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq\" (UID: \"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0\") " pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" Oct 30 18:31:20.185164 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:20.183042 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-workload-socket\") pod \"e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq\" (UID: \"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0\") " pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" Oct 30 18:31:20.185164 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:20.183632 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-workload-socket\") pod \"e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq\" (UID: \"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0\") " pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" Oct 30 18:31:20.185164 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:20.184355 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-istiod-ca-cert\") pod \"e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq\" (UID: \"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0\") " pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" Oct 30 18:31:20.185164 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:20.184972 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-credential-socket\") pod \"e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq\" (UID: \"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0\") " pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" Oct 30 18:31:20.185961 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:20.185937 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-istio-data\") pod \"e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq\" (UID: \"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0\") " pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" Oct 30 18:31:20.270814 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:20.209598 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-workload-certs\") pod \"e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq\" (UID: \"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0\") " pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" Oct 30 18:31:20.408701 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:20.408665 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-istio-token\") pod \"e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq\" (UID: \"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0\") " pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" Oct 30 18:31:20.436225 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:20.436113 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-istio-podinfo\") pod \"e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq\" (UID: \"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0\") " pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" Oct 30 18:31:20.473540 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:20.469108 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfl4f\" (UniqueName: \"kubernetes.io/projected/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-kube-api-access-cfl4f\") pod \"e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq\" (UID: \"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0\") " pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" Oct 30 18:31:20.473540 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:20.469384 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-istio-envoy\") pod \"e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq\" (UID: \"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0\") " pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" Oct 30 18:31:20.716450 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:20.716412 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" Oct 30 18:31:20.840179 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:20.810233 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1a2436-7676957859-jxlmq"] Oct 30 18:31:20.890188 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:20.883424 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1a2436-7676957859-jxlmq" Oct 30 18:31:20.929837 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:20.927060 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-1a2436-dockercfg-dmtgd\"" Oct 30 18:31:20.989219 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:20.984349 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-1a2436\"" Oct 30 18:31:21.012436 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:21.006206 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1a2436-7676957859-jxlmq"] Oct 30 18:31:21.066187 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:21.022604 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0249c5c5-6032-425a-9107-2526fd89627a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-1a2436-7676957859-jxlmq\" (UID: \"0249c5c5-6032-425a-9107-2526fd89627a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1a2436-7676957859-jxlmq" Oct 30 18:31:21.066187 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:21.022658 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdc8z\" (UniqueName: \"kubernetes.io/projected/0249c5c5-6032-425a-9107-2526fd89627a-kube-api-access-fdc8z\") pod \"payload-pre-processing-e2e-shared-a-w4-1a2436-7676957859-jxlmq\" (UID: \"0249c5c5-6032-425a-9107-2526fd89627a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1a2436-7676957859-jxlmq" Oct 30 18:31:21.321209 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:21.320484 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0249c5c5-6032-425a-9107-2526fd89627a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-1a2436-7676957859-jxlmq\" (UID: \"0249c5c5-6032-425a-9107-2526fd89627a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1a2436-7676957859-jxlmq" Oct 30 18:31:21.321209 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:21.320645 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdc8z\" (UniqueName: \"kubernetes.io/projected/0249c5c5-6032-425a-9107-2526fd89627a-kube-api-access-fdc8z\") pod \"payload-pre-processing-e2e-shared-a-w4-1a2436-7676957859-jxlmq\" (UID: \"0249c5c5-6032-425a-9107-2526fd89627a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1a2436-7676957859-jxlmq" Oct 30 18:31:21.322400 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:21.322344 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0249c5c5-6032-425a-9107-2526fd89627a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-1a2436-7676957859-jxlmq\" (UID: \"0249c5c5-6032-425a-9107-2526fd89627a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1a2436-7676957859-jxlmq" Oct 30 18:31:21.439422 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:21.439390 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdc8z\" (UniqueName: \"kubernetes.io/projected/0249c5c5-6032-425a-9107-2526fd89627a-kube-api-access-fdc8z\") pod \"payload-pre-processing-e2e-shared-a-w4-1a2436-7676957859-jxlmq\" (UID: \"0249c5c5-6032-425a-9107-2526fd89627a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1a2436-7676957859-jxlmq" Oct 30 18:31:21.467158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:21.462950 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1a2436-7676957859-jxlmq" Oct 30 18:31:22.220489 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:22.220442 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq"] Oct 30 18:31:22.311490 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:22.311446 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:31:22.311642 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:22.311535 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:31:22.311642 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:22.311569 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:31:22.429633 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:22.429574 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" event={"ID":"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0","Type":"ContainerStarted","Data":"203ad1dabde5cdacd87c19d825fdd58ef77fffd02038b0d264cc6ab1bab58f2f"} Oct 30 18:31:22.564051 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:22.563974 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1a2436-7676957859-jxlmq"] Oct 30 18:31:23.207479 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.207450 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2"] Oct 30 18:31:23.211764 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.211743 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" Oct 30 18:31:23.251465 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.236111 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-45dc34-openshift-default-dockercfg-w86md\"" Oct 30 18:31:23.297545 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.297510 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn"] Oct 30 18:31:23.306581 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.306547 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2"] Oct 30 18:31:23.306732 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.306675 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" Oct 30 18:31:23.308441 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.308408 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn"] Oct 30 18:31:23.385654 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.385617 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7df5e06-8248-4312-8abc-72deee20f840-istio-envoy\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2\" (UID: \"b7df5e06-8248-4312-8abc-72deee20f840\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" Oct 30 18:31:23.385851 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.385752 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7df5e06-8248-4312-8abc-72deee20f840-credential-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2\" (UID: \"b7df5e06-8248-4312-8abc-72deee20f840\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" Oct 30 18:31:23.385851 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.385793 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7df5e06-8248-4312-8abc-72deee20f840-istio-podinfo\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2\" (UID: \"b7df5e06-8248-4312-8abc-72deee20f840\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" Oct 30 18:31:23.385953 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.385921 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7df5e06-8248-4312-8abc-72deee20f840-istiod-ca-cert\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2\" (UID: \"b7df5e06-8248-4312-8abc-72deee20f840\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" Oct 30 18:31:23.386128 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.386107 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpg2g\" (UniqueName: \"kubernetes.io/projected/b7df5e06-8248-4312-8abc-72deee20f840-kube-api-access-gpg2g\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2\" (UID: \"b7df5e06-8248-4312-8abc-72deee20f840\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" Oct 30 18:31:23.386128 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.386156 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7df5e06-8248-4312-8abc-72deee20f840-workload-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2\" (UID: \"b7df5e06-8248-4312-8abc-72deee20f840\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" Oct 30 18:31:23.386300 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.386253 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7df5e06-8248-4312-8abc-72deee20f840-workload-certs\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2\" (UID: \"b7df5e06-8248-4312-8abc-72deee20f840\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" Oct 30 18:31:23.386582 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.386569 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7df5e06-8248-4312-8abc-72deee20f840-istio-token\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2\" (UID: \"b7df5e06-8248-4312-8abc-72deee20f840\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" Oct 30 18:31:23.386619 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.386611 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7df5e06-8248-4312-8abc-72deee20f840-istio-data\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2\" (UID: \"b7df5e06-8248-4312-8abc-72deee20f840\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" Oct 30 18:31:23.465585 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.465502 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1a2436-7676957859-jxlmq" event={"ID":"0249c5c5-6032-425a-9107-2526fd89627a","Type":"ContainerStarted","Data":"5b6d407866c8208622812345b02af5516e616c559b0fce5ccf46501bf7ee5d1f"} Oct 30 18:31:23.465585 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.465563 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1a2436-7676957859-jxlmq" event={"ID":"0249c5c5-6032-425a-9107-2526fd89627a","Type":"ContainerStarted","Data":"607e2f4eca7aaf4974e3607e9b458afc6fb516a375da6467ce57fe058c1d92ee"} Oct 30 18:31:23.465958 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.465932 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1a2436-7676957859-jxlmq" Oct 30 18:31:23.467748 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.467725 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" event={"ID":"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0","Type":"ContainerStarted","Data":"bda4e615fb2dbd2e4c3d093e91f3fcc329e846cc637cecd17211dbb72f73e2de"} Oct 30 18:31:23.488411 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.488387 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15bc30e5-adb7-45f8-abd5-d5fea229bda7-istio-podinfo\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn\" (UID: \"15bc30e5-adb7-45f8-abd5-d5fea229bda7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" Oct 30 18:31:23.488504 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.488439 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7df5e06-8248-4312-8abc-72deee20f840-istio-token\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2\" (UID: \"b7df5e06-8248-4312-8abc-72deee20f840\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" Oct 30 18:31:23.488555 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.488511 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7df5e06-8248-4312-8abc-72deee20f840-istio-data\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2\" (UID: \"b7df5e06-8248-4312-8abc-72deee20f840\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" Oct 30 18:31:23.488555 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.488537 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15bc30e5-adb7-45f8-abd5-d5fea229bda7-workload-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn\" (UID: \"15bc30e5-adb7-45f8-abd5-d5fea229bda7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" Oct 30 18:31:23.488666 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.488645 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7df5e06-8248-4312-8abc-72deee20f840-istio-envoy\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2\" (UID: \"b7df5e06-8248-4312-8abc-72deee20f840\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" Oct 30 18:31:23.488736 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.488715 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15bc30e5-adb7-45f8-abd5-d5fea229bda7-credential-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn\" (UID: \"15bc30e5-adb7-45f8-abd5-d5fea229bda7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" Oct 30 18:31:23.488812 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.488760 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7df5e06-8248-4312-8abc-72deee20f840-credential-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2\" (UID: \"b7df5e06-8248-4312-8abc-72deee20f840\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" Oct 30 18:31:23.488812 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.488794 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7df5e06-8248-4312-8abc-72deee20f840-istio-podinfo\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2\" (UID: \"b7df5e06-8248-4312-8abc-72deee20f840\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" Oct 30 18:31:23.488892 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.488845 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skqh8\" (UniqueName: \"kubernetes.io/projected/15bc30e5-adb7-45f8-abd5-d5fea229bda7-kube-api-access-skqh8\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn\" (UID: \"15bc30e5-adb7-45f8-abd5-d5fea229bda7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" Oct 30 18:31:23.488892 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.488879 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15bc30e5-adb7-45f8-abd5-d5fea229bda7-istio-data\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn\" (UID: \"15bc30e5-adb7-45f8-abd5-d5fea229bda7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" Oct 30 18:31:23.488967 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.488912 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7df5e06-8248-4312-8abc-72deee20f840-istiod-ca-cert\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2\" (UID: \"b7df5e06-8248-4312-8abc-72deee20f840\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" Oct 30 18:31:23.488967 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.488942 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7df5e06-8248-4312-8abc-72deee20f840-istio-data\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2\" (UID: \"b7df5e06-8248-4312-8abc-72deee20f840\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" Oct 30 18:31:23.489047 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.488983 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpg2g\" (UniqueName: \"kubernetes.io/projected/b7df5e06-8248-4312-8abc-72deee20f840-kube-api-access-gpg2g\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2\" (UID: \"b7df5e06-8248-4312-8abc-72deee20f840\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" Oct 30 18:31:23.489114 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.489051 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15bc30e5-adb7-45f8-abd5-d5fea229bda7-istio-token\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn\" (UID: \"15bc30e5-adb7-45f8-abd5-d5fea229bda7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" Oct 30 18:31:23.489234 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.489200 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7df5e06-8248-4312-8abc-72deee20f840-workload-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2\" (UID: \"b7df5e06-8248-4312-8abc-72deee20f840\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" Oct 30 18:31:23.489504 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.489449 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7df5e06-8248-4312-8abc-72deee20f840-workload-certs\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2\" (UID: \"b7df5e06-8248-4312-8abc-72deee20f840\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" Oct 30 18:31:23.489504 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.489391 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7df5e06-8248-4312-8abc-72deee20f840-credential-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2\" (UID: \"b7df5e06-8248-4312-8abc-72deee20f840\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" Oct 30 18:31:23.489504 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.489490 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15bc30e5-adb7-45f8-abd5-d5fea229bda7-istiod-ca-cert\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn\" (UID: \"15bc30e5-adb7-45f8-abd5-d5fea229bda7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" Oct 30 18:31:23.489836 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.489586 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7df5e06-8248-4312-8abc-72deee20f840-workload-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2\" (UID: \"b7df5e06-8248-4312-8abc-72deee20f840\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" Oct 30 18:31:23.489836 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.489587 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15bc30e5-adb7-45f8-abd5-d5fea229bda7-workload-certs\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn\" (UID: \"15bc30e5-adb7-45f8-abd5-d5fea229bda7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" Oct 30 18:31:23.489836 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.489754 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15bc30e5-adb7-45f8-abd5-d5fea229bda7-istio-envoy\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn\" (UID: \"15bc30e5-adb7-45f8-abd5-d5fea229bda7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" Oct 30 18:31:23.489836 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.489755 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7df5e06-8248-4312-8abc-72deee20f840-workload-certs\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2\" (UID: \"b7df5e06-8248-4312-8abc-72deee20f840\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" Oct 30 18:31:23.490065 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.490010 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7df5e06-8248-4312-8abc-72deee20f840-istiod-ca-cert\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2\" (UID: \"b7df5e06-8248-4312-8abc-72deee20f840\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" Oct 30 18:31:23.491454 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.491431 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7df5e06-8248-4312-8abc-72deee20f840-istio-envoy\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2\" (UID: \"b7df5e06-8248-4312-8abc-72deee20f840\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" Oct 30 18:31:23.492419 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.492396 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7df5e06-8248-4312-8abc-72deee20f840-istio-podinfo\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2\" (UID: \"b7df5e06-8248-4312-8abc-72deee20f840\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" Oct 30 18:31:23.506100 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.506043 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1a2436-7676957859-jxlmq" podStartSLOduration=3.5060310059999997 podStartE2EDuration="3.506031006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:31:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:31:23.50292086 +0000 UTC m=+1980.592983203" watchObservedRunningTime="2026-10-30 18:31:23.506031006 +0000 UTC m=+1980.596093345" Oct 30 18:31:23.508394 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.508363 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7df5e06-8248-4312-8abc-72deee20f840-istio-token\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2\" (UID: \"b7df5e06-8248-4312-8abc-72deee20f840\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" Oct 30 18:31:23.508643 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.508622 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpg2g\" (UniqueName: \"kubernetes.io/projected/b7df5e06-8248-4312-8abc-72deee20f840-kube-api-access-gpg2g\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2\" (UID: \"b7df5e06-8248-4312-8abc-72deee20f840\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" Oct 30 18:31:23.560284 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.559515 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" podStartSLOduration=4.559492162 podStartE2EDuration="4.559492162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:31:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:31:23.545743122 +0000 UTC m=+1980.635805462" watchObservedRunningTime="2026-10-30 18:31:23.559492162 +0000 UTC m=+1980.649554502" Oct 30 18:31:23.591479 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.591445 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15bc30e5-adb7-45f8-abd5-d5fea229bda7-istio-data\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn\" (UID: \"15bc30e5-adb7-45f8-abd5-d5fea229bda7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" Oct 30 18:31:23.591660 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.591525 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15bc30e5-adb7-45f8-abd5-d5fea229bda7-istio-token\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn\" (UID: \"15bc30e5-adb7-45f8-abd5-d5fea229bda7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" Oct 30 18:31:23.591660 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.591557 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15bc30e5-adb7-45f8-abd5-d5fea229bda7-istiod-ca-cert\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn\" (UID: \"15bc30e5-adb7-45f8-abd5-d5fea229bda7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" Oct 30 18:31:23.591660 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.591611 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15bc30e5-adb7-45f8-abd5-d5fea229bda7-workload-certs\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn\" (UID: \"15bc30e5-adb7-45f8-abd5-d5fea229bda7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" Oct 30 18:31:23.591660 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.591640 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15bc30e5-adb7-45f8-abd5-d5fea229bda7-istio-envoy\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn\" (UID: \"15bc30e5-adb7-45f8-abd5-d5fea229bda7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" Oct 30 18:31:23.591923 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.591692 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15bc30e5-adb7-45f8-abd5-d5fea229bda7-istio-podinfo\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn\" (UID: \"15bc30e5-adb7-45f8-abd5-d5fea229bda7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" Oct 30 18:31:23.591923 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.591747 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15bc30e5-adb7-45f8-abd5-d5fea229bda7-workload-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn\" (UID: \"15bc30e5-adb7-45f8-abd5-d5fea229bda7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" Oct 30 18:31:23.591923 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.591803 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15bc30e5-adb7-45f8-abd5-d5fea229bda7-credential-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn\" (UID: \"15bc30e5-adb7-45f8-abd5-d5fea229bda7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" Oct 30 18:31:23.591923 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.591829 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skqh8\" (UniqueName: \"kubernetes.io/projected/15bc30e5-adb7-45f8-abd5-d5fea229bda7-kube-api-access-skqh8\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn\" (UID: \"15bc30e5-adb7-45f8-abd5-d5fea229bda7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" Oct 30 18:31:23.591923 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.591891 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15bc30e5-adb7-45f8-abd5-d5fea229bda7-istio-data\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn\" (UID: \"15bc30e5-adb7-45f8-abd5-d5fea229bda7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" Oct 30 18:31:23.592221 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.591994 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15bc30e5-adb7-45f8-abd5-d5fea229bda7-workload-certs\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn\" (UID: \"15bc30e5-adb7-45f8-abd5-d5fea229bda7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" Oct 30 18:31:23.592221 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.592166 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" Oct 30 18:31:23.592340 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.592220 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15bc30e5-adb7-45f8-abd5-d5fea229bda7-istiod-ca-cert\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn\" (UID: \"15bc30e5-adb7-45f8-abd5-d5fea229bda7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" Oct 30 18:31:23.592340 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.592268 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15bc30e5-adb7-45f8-abd5-d5fea229bda7-credential-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn\" (UID: \"15bc30e5-adb7-45f8-abd5-d5fea229bda7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" Oct 30 18:31:23.592455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.592387 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15bc30e5-adb7-45f8-abd5-d5fea229bda7-workload-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn\" (UID: \"15bc30e5-adb7-45f8-abd5-d5fea229bda7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" Oct 30 18:31:23.594506 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.594479 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15bc30e5-adb7-45f8-abd5-d5fea229bda7-istio-envoy\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn\" (UID: \"15bc30e5-adb7-45f8-abd5-d5fea229bda7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" Oct 30 18:31:23.595023 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.595002 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15bc30e5-adb7-45f8-abd5-d5fea229bda7-istio-podinfo\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn\" (UID: \"15bc30e5-adb7-45f8-abd5-d5fea229bda7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" Oct 30 18:31:23.610584 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.610557 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15bc30e5-adb7-45f8-abd5-d5fea229bda7-istio-token\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn\" (UID: \"15bc30e5-adb7-45f8-abd5-d5fea229bda7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" Oct 30 18:31:23.611552 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.611532 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skqh8\" (UniqueName: \"kubernetes.io/projected/15bc30e5-adb7-45f8-abd5-d5fea229bda7-kube-api-access-skqh8\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn\" (UID: \"15bc30e5-adb7-45f8-abd5-d5fea229bda7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" Oct 30 18:31:23.622369 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.622345 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" Oct 30 18:31:23.726698 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.726275 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" Oct 30 18:31:23.773169 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:23.772847 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" Oct 30 18:31:24.015224 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:31:24.015189 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7df5e06_8248_4312_8abc_72deee20f840.slice/crio-d046c1bba7b85f1a90a6ed5a828dbe9b9cfa71747a77d0240bc9cddf18ba3a0b WatchSource:0}: Error finding container d046c1bba7b85f1a90a6ed5a828dbe9b9cfa71747a77d0240bc9cddf18ba3a0b: Status 404 returned error can't find the container with id d046c1bba7b85f1a90a6ed5a828dbe9b9cfa71747a77d0240bc9cddf18ba3a0b Oct 30 18:31:24.015632 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:24.015564 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2"] Oct 30 18:31:24.018466 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:24.018426 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:31:24.018631 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:24.018597 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:31:24.018784 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:24.018724 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:31:24.163434 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:24.163400 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn"] Oct 30 18:31:24.165645 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:31:24.165616 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15bc30e5_adb7_45f8_abd5_d5fea229bda7.slice/crio-bb6e0c7b97cab0149640daf0eeec4aa4f770215c1b8870e2d5390d2fd522b188 WatchSource:0}: Error finding container bb6e0c7b97cab0149640daf0eeec4aa4f770215c1b8870e2d5390d2fd522b188: Status 404 returned error can't find the container with id bb6e0c7b97cab0149640daf0eeec4aa4f770215c1b8870e2d5390d2fd522b188 Oct 30 18:31:24.168242 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:24.168207 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:31:24.168358 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:24.168288 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:31:24.168358 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:24.168342 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:31:24.482997 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:24.482959 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" event={"ID":"b7df5e06-8248-4312-8abc-72deee20f840","Type":"ContainerStarted","Data":"79273c2320b2eb6d1c43258a027cb6a00632d9c684ddaca131fa84f5032dbaf2"} Oct 30 18:31:24.483533 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:24.483514 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" event={"ID":"b7df5e06-8248-4312-8abc-72deee20f840","Type":"ContainerStarted","Data":"d046c1bba7b85f1a90a6ed5a828dbe9b9cfa71747a77d0240bc9cddf18ba3a0b"} Oct 30 18:31:24.487472 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:24.487180 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" event={"ID":"15bc30e5-adb7-45f8-abd5-d5fea229bda7","Type":"ContainerStarted","Data":"93e0da9fffce6f5476844d2bfe20733255f86ff8b7daafd1ff650e670961f716"} Oct 30 18:31:24.487472 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:24.487241 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" Oct 30 18:31:24.487472 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:24.487272 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" event={"ID":"15bc30e5-adb7-45f8-abd5-d5fea229bda7","Type":"ContainerStarted","Data":"bb6e0c7b97cab0149640daf0eeec4aa4f770215c1b8870e2d5390d2fd522b188"} Oct 30 18:31:24.487472 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:24.487325 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" Oct 30 18:31:24.592656 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:24.592579 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" Oct 30 18:31:24.601540 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:24.601490 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" podStartSLOduration=1.601475816 podStartE2EDuration="1.601475816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:31:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:31:24.544457717 +0000 UTC m=+1981.634520059" watchObservedRunningTime="2026-10-30 18:31:24.601475816 +0000 UTC m=+1981.691538162" Oct 30 18:31:24.622509 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:24.622475 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" Oct 30 18:31:24.689155 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:24.688398 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" podStartSLOduration=1.68837839 podStartE2EDuration="1.68837839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:31:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:31:24.683553836 +0000 UTC m=+1981.773616177" watchObservedRunningTime="2026-10-30 18:31:24.68837839 +0000 UTC m=+1981.778440730" Oct 30 18:31:25.598446 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:25.598407 4442 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 30 18:31:25.599279 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:25.599236 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" podUID="b7df5e06-8248-4312-8abc-72deee20f840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 30 18:31:25.627173 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:25.624192 4442 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Oct 30 18:31:25.627173 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:25.624265 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" podUID="15bc30e5-adb7-45f8-abd5-d5fea229bda7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Oct 30 18:31:26.595336 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:26.594727 4442 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 30 18:31:26.595336 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:26.594797 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" podUID="b7df5e06-8248-4312-8abc-72deee20f840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 30 18:31:26.622864 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:26.622828 4442 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Oct 30 18:31:26.623388 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:26.622876 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" podUID="15bc30e5-adb7-45f8-abd5-d5fea229bda7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Oct 30 18:31:27.592864 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:27.592602 4442 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 30 18:31:27.592864 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:27.592665 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" podUID="b7df5e06-8248-4312-8abc-72deee20f840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 30 18:31:27.644065 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:27.644030 4442 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Oct 30 18:31:27.644691 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:27.644648 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" podUID="15bc30e5-adb7-45f8-abd5-d5fea229bda7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Oct 30 18:31:28.598163 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:28.595582 4442 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 30 18:31:28.598163 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:28.595641 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" podUID="b7df5e06-8248-4312-8abc-72deee20f840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 30 18:31:28.644162 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:28.640958 4442 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Oct 30 18:31:28.644162 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:28.641019 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" podUID="15bc30e5-adb7-45f8-abd5-d5fea229bda7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Oct 30 18:31:29.593078 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:29.593038 4442 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 30 18:31:29.593287 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:29.593104 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" podUID="b7df5e06-8248-4312-8abc-72deee20f840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 30 18:31:29.622978 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:29.622943 4442 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Oct 30 18:31:29.623213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:29.623010 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" podUID="15bc30e5-adb7-45f8-abd5-d5fea229bda7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Oct 30 18:31:30.592528 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:30.592494 4442 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 30 18:31:30.592948 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:30.592544 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" podUID="b7df5e06-8248-4312-8abc-72deee20f840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 30 18:31:30.622955 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:30.622927 4442 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Oct 30 18:31:30.623255 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:30.623223 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" podUID="15bc30e5-adb7-45f8-abd5-d5fea229bda7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Oct 30 18:31:31.592871 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:31.592814 4442 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 30 18:31:31.593513 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:31.592884 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" podUID="b7df5e06-8248-4312-8abc-72deee20f840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 30 18:31:31.623662 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:31.623625 4442 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Oct 30 18:31:31.623874 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:31.623705 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" podUID="15bc30e5-adb7-45f8-abd5-d5fea229bda7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Oct 30 18:31:31.799753 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:31.798266 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst"] Oct 30 18:31:31.858907 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:31.858832 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" Oct 30 18:31:31.861685 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:31.861476 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst"] Oct 30 18:31:31.884718 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:31.884654 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-1e0b31-openshift-default-dockercfg-55qqb\"" Oct 30 18:31:32.192006 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:32.188628 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd132ea6-9517-4402-b079-16d0af22572f-istio-data\") pod \"e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst\" (UID: \"bd132ea6-9517-4402-b079-16d0af22572f\") " pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" Oct 30 18:31:32.192006 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:32.188743 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm9dw\" (UniqueName: \"kubernetes.io/projected/bd132ea6-9517-4402-b079-16d0af22572f-kube-api-access-cm9dw\") pod \"e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst\" (UID: \"bd132ea6-9517-4402-b079-16d0af22572f\") " pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" Oct 30 18:31:32.192006 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:32.188800 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd132ea6-9517-4402-b079-16d0af22572f-istio-envoy\") pod \"e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst\" (UID: \"bd132ea6-9517-4402-b079-16d0af22572f\") " pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" Oct 30 18:31:32.192006 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:32.188848 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd132ea6-9517-4402-b079-16d0af22572f-workload-socket\") pod \"e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst\" (UID: \"bd132ea6-9517-4402-b079-16d0af22572f\") " pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" Oct 30 18:31:32.192006 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:32.188890 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd132ea6-9517-4402-b079-16d0af22572f-credential-socket\") pod \"e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst\" (UID: \"bd132ea6-9517-4402-b079-16d0af22572f\") " pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" Oct 30 18:31:32.192006 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:32.188968 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd132ea6-9517-4402-b079-16d0af22572f-istio-token\") pod \"e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst\" (UID: \"bd132ea6-9517-4402-b079-16d0af22572f\") " pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" Oct 30 18:31:32.192006 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:32.189057 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd132ea6-9517-4402-b079-16d0af22572f-istiod-ca-cert\") pod \"e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst\" (UID: \"bd132ea6-9517-4402-b079-16d0af22572f\") " pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" Oct 30 18:31:32.192006 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:32.189161 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd132ea6-9517-4402-b079-16d0af22572f-istio-podinfo\") pod \"e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst\" (UID: \"bd132ea6-9517-4402-b079-16d0af22572f\") " pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" Oct 30 18:31:32.192006 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:32.189201 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd132ea6-9517-4402-b079-16d0af22572f-workload-certs\") pod \"e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst\" (UID: \"bd132ea6-9517-4402-b079-16d0af22572f\") " pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" Oct 30 18:31:32.297161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:32.294039 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd132ea6-9517-4402-b079-16d0af22572f-istio-envoy\") pod \"e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst\" (UID: \"bd132ea6-9517-4402-b079-16d0af22572f\") " pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" Oct 30 18:31:32.297161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:32.294104 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd132ea6-9517-4402-b079-16d0af22572f-workload-socket\") pod \"e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst\" (UID: \"bd132ea6-9517-4402-b079-16d0af22572f\") " pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" Oct 30 18:31:32.297161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:32.294159 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd132ea6-9517-4402-b079-16d0af22572f-credential-socket\") pod \"e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst\" (UID: \"bd132ea6-9517-4402-b079-16d0af22572f\") " pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" Oct 30 18:31:32.297161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:32.294221 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd132ea6-9517-4402-b079-16d0af22572f-istio-token\") pod \"e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst\" (UID: \"bd132ea6-9517-4402-b079-16d0af22572f\") " pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" Oct 30 18:31:32.297161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:32.294326 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd132ea6-9517-4402-b079-16d0af22572f-istiod-ca-cert\") pod \"e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst\" (UID: \"bd132ea6-9517-4402-b079-16d0af22572f\") " pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" Oct 30 18:31:32.297161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:32.294434 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd132ea6-9517-4402-b079-16d0af22572f-istio-podinfo\") pod \"e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst\" (UID: \"bd132ea6-9517-4402-b079-16d0af22572f\") " pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" Oct 30 18:31:32.297161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:32.294470 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd132ea6-9517-4402-b079-16d0af22572f-workload-certs\") pod \"e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst\" (UID: \"bd132ea6-9517-4402-b079-16d0af22572f\") " pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" Oct 30 18:31:32.297161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:32.294622 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd132ea6-9517-4402-b079-16d0af22572f-istio-data\") pod \"e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst\" (UID: \"bd132ea6-9517-4402-b079-16d0af22572f\") " pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" Oct 30 18:31:32.297161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:32.294713 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm9dw\" (UniqueName: \"kubernetes.io/projected/bd132ea6-9517-4402-b079-16d0af22572f-kube-api-access-cm9dw\") pod \"e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst\" (UID: \"bd132ea6-9517-4402-b079-16d0af22572f\") " pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" Oct 30 18:31:32.297816 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:32.297215 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd132ea6-9517-4402-b079-16d0af22572f-istio-data\") pod \"e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst\" (UID: \"bd132ea6-9517-4402-b079-16d0af22572f\") " pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" Oct 30 18:31:32.297883 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:32.297867 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd132ea6-9517-4402-b079-16d0af22572f-workload-socket\") pod \"e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst\" (UID: \"bd132ea6-9517-4402-b079-16d0af22572f\") " pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" Oct 30 18:31:32.299466 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:32.298094 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd132ea6-9517-4402-b079-16d0af22572f-credential-socket\") pod \"e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst\" (UID: \"bd132ea6-9517-4402-b079-16d0af22572f\") " pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" Oct 30 18:31:32.299466 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:32.299218 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd132ea6-9517-4402-b079-16d0af22572f-istiod-ca-cert\") pod \"e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst\" (UID: \"bd132ea6-9517-4402-b079-16d0af22572f\") " pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" Oct 30 18:31:32.300079 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:32.299849 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd132ea6-9517-4402-b079-16d0af22572f-workload-certs\") pod \"e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst\" (UID: \"bd132ea6-9517-4402-b079-16d0af22572f\") " pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" Oct 30 18:31:32.301633 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:32.301572 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd132ea6-9517-4402-b079-16d0af22572f-istio-envoy\") pod \"e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst\" (UID: \"bd132ea6-9517-4402-b079-16d0af22572f\") " pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" Oct 30 18:31:32.304528 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:32.304476 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd132ea6-9517-4402-b079-16d0af22572f-istio-podinfo\") pod \"e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst\" (UID: \"bd132ea6-9517-4402-b079-16d0af22572f\") " pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" Oct 30 18:31:32.346651 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:32.346575 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd132ea6-9517-4402-b079-16d0af22572f-istio-token\") pod \"e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst\" (UID: \"bd132ea6-9517-4402-b079-16d0af22572f\") " pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" Oct 30 18:31:32.347587 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:32.347519 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm9dw\" (UniqueName: \"kubernetes.io/projected/bd132ea6-9517-4402-b079-16d0af22572f-kube-api-access-cm9dw\") pod \"e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst\" (UID: \"bd132ea6-9517-4402-b079-16d0af22572f\") " pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" Oct 30 18:31:32.488646 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:32.488548 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" Oct 30 18:31:32.773963 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:32.772372 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" Oct 30 18:31:32.774840 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:32.774765 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" Oct 30 18:31:32.778354 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:32.778281 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" Oct 30 18:31:32.847490 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:32.847461 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst"] Oct 30 18:31:32.850831 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:31:32.850801 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd132ea6_9517_4402_b079_16d0af22572f.slice/crio-d3ac499d97246c3bb73b468a5a21ca01377b8e8d9514c15619caf981d3cb92c6 WatchSource:0}: Error finding container d3ac499d97246c3bb73b468a5a21ca01377b8e8d9514c15619caf981d3cb92c6: Status 404 returned error can't find the container with id d3ac499d97246c3bb73b468a5a21ca01377b8e8d9514c15619caf981d3cb92c6 Oct 30 18:31:32.853605 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:32.853556 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:31:32.853695 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:32.853651 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:31:32.853736 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:32.853705 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:31:32.976383 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:32.976297 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" Oct 30 18:31:32.977901 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:32.977871 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" Oct 30 18:31:32.979028 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:32.979005 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" Oct 30 18:31:33.176789 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:33.176749 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr"] Oct 30 18:31:33.185691 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:33.185664 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr"] Oct 30 18:31:33.187361 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:33.187336 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" Oct 30 18:31:33.240161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:33.222831 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a85dd1f3-b655-4c52-b1ef-c5471365634a-workload-certs\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr\" (UID: \"a85dd1f3-b655-4c52-b1ef-c5471365634a\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" Oct 30 18:31:33.240161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:33.222914 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a85dd1f3-b655-4c52-b1ef-c5471365634a-istio-data\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr\" (UID: \"a85dd1f3-b655-4c52-b1ef-c5471365634a\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" Oct 30 18:31:33.240161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:33.222970 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a85dd1f3-b655-4c52-b1ef-c5471365634a-istio-envoy\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr\" (UID: \"a85dd1f3-b655-4c52-b1ef-c5471365634a\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" Oct 30 18:31:33.240161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:33.223081 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfc8v\" (UniqueName: \"kubernetes.io/projected/a85dd1f3-b655-4c52-b1ef-c5471365634a-kube-api-access-rfc8v\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr\" (UID: \"a85dd1f3-b655-4c52-b1ef-c5471365634a\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" Oct 30 18:31:33.240161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:33.223107 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a85dd1f3-b655-4c52-b1ef-c5471365634a-workload-socket\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr\" (UID: \"a85dd1f3-b655-4c52-b1ef-c5471365634a\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" Oct 30 18:31:33.240161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:33.223185 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a85dd1f3-b655-4c52-b1ef-c5471365634a-credential-socket\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr\" (UID: \"a85dd1f3-b655-4c52-b1ef-c5471365634a\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" Oct 30 18:31:33.240161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:33.223224 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a85dd1f3-b655-4c52-b1ef-c5471365634a-istiod-ca-cert\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr\" (UID: \"a85dd1f3-b655-4c52-b1ef-c5471365634a\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" Oct 30 18:31:33.240161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:33.223256 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a85dd1f3-b655-4c52-b1ef-c5471365634a-istio-podinfo\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr\" (UID: \"a85dd1f3-b655-4c52-b1ef-c5471365634a\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" Oct 30 18:31:33.240161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:33.223286 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a85dd1f3-b655-4c52-b1ef-c5471365634a-istio-token\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr\" (UID: \"a85dd1f3-b655-4c52-b1ef-c5471365634a\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" Oct 30 18:31:33.275157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:33.251537 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-4a7508-openshift-default-dockercfg-ncnp4\"" Oct 30 18:31:33.350198 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:33.347225 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a85dd1f3-b655-4c52-b1ef-c5471365634a-credential-socket\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr\" (UID: \"a85dd1f3-b655-4c52-b1ef-c5471365634a\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" Oct 30 18:31:33.350198 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:33.347294 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a85dd1f3-b655-4c52-b1ef-c5471365634a-istiod-ca-cert\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr\" (UID: \"a85dd1f3-b655-4c52-b1ef-c5471365634a\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" Oct 30 18:31:33.350198 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:33.347332 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a85dd1f3-b655-4c52-b1ef-c5471365634a-istio-podinfo\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr\" (UID: \"a85dd1f3-b655-4c52-b1ef-c5471365634a\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" Oct 30 18:31:33.350198 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:33.347367 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a85dd1f3-b655-4c52-b1ef-c5471365634a-istio-token\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr\" (UID: \"a85dd1f3-b655-4c52-b1ef-c5471365634a\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" Oct 30 18:31:33.350198 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:33.347447 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a85dd1f3-b655-4c52-b1ef-c5471365634a-workload-certs\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr\" (UID: \"a85dd1f3-b655-4c52-b1ef-c5471365634a\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" Oct 30 18:31:33.350198 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:33.347552 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a85dd1f3-b655-4c52-b1ef-c5471365634a-istio-data\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr\" (UID: \"a85dd1f3-b655-4c52-b1ef-c5471365634a\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" Oct 30 18:31:33.350198 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:33.347625 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a85dd1f3-b655-4c52-b1ef-c5471365634a-istio-envoy\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr\" (UID: \"a85dd1f3-b655-4c52-b1ef-c5471365634a\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" Oct 30 18:31:33.350198 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:33.347790 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfc8v\" (UniqueName: \"kubernetes.io/projected/a85dd1f3-b655-4c52-b1ef-c5471365634a-kube-api-access-rfc8v\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr\" (UID: \"a85dd1f3-b655-4c52-b1ef-c5471365634a\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" Oct 30 18:31:33.350198 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:33.347825 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a85dd1f3-b655-4c52-b1ef-c5471365634a-workload-socket\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr\" (UID: \"a85dd1f3-b655-4c52-b1ef-c5471365634a\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" Oct 30 18:31:33.350198 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:33.348368 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a85dd1f3-b655-4c52-b1ef-c5471365634a-workload-socket\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr\" (UID: \"a85dd1f3-b655-4c52-b1ef-c5471365634a\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" Oct 30 18:31:33.350198 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:33.348632 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a85dd1f3-b655-4c52-b1ef-c5471365634a-credential-socket\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr\" (UID: \"a85dd1f3-b655-4c52-b1ef-c5471365634a\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" Oct 30 18:31:33.350198 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:33.349278 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a85dd1f3-b655-4c52-b1ef-c5471365634a-istiod-ca-cert\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr\" (UID: \"a85dd1f3-b655-4c52-b1ef-c5471365634a\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" Oct 30 18:31:33.407158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:33.360103 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a85dd1f3-b655-4c52-b1ef-c5471365634a-istio-data\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr\" (UID: \"a85dd1f3-b655-4c52-b1ef-c5471365634a\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" Oct 30 18:31:33.407158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:33.360714 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a85dd1f3-b655-4c52-b1ef-c5471365634a-workload-certs\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr\" (UID: \"a85dd1f3-b655-4c52-b1ef-c5471365634a\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" Oct 30 18:31:33.555929 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:33.538720 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a85dd1f3-b655-4c52-b1ef-c5471365634a-istio-envoy\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr\" (UID: \"a85dd1f3-b655-4c52-b1ef-c5471365634a\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" Oct 30 18:31:33.592976 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:33.591725 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a85dd1f3-b655-4c52-b1ef-c5471365634a-istio-podinfo\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr\" (UID: \"a85dd1f3-b655-4c52-b1ef-c5471365634a\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" Oct 30 18:31:33.594517 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:33.594445 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a85dd1f3-b655-4c52-b1ef-c5471365634a-istio-token\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr\" (UID: \"a85dd1f3-b655-4c52-b1ef-c5471365634a\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" Oct 30 18:31:33.614822 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:33.614788 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfc8v\" (UniqueName: \"kubernetes.io/projected/a85dd1f3-b655-4c52-b1ef-c5471365634a-kube-api-access-rfc8v\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr\" (UID: \"a85dd1f3-b655-4c52-b1ef-c5471365634a\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" Oct 30 18:31:33.734210 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:33.733581 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" event={"ID":"bd132ea6-9517-4402-b079-16d0af22572f","Type":"ContainerStarted","Data":"07a5613dbd40355550531d62d61a6600642eec76b158886aadb432e9d75a5045"} Oct 30 18:31:33.734210 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:33.733637 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" event={"ID":"bd132ea6-9517-4402-b079-16d0af22572f","Type":"ContainerStarted","Data":"d3ac499d97246c3bb73b468a5a21ca01377b8e8d9514c15619caf981d3cb92c6"} Oct 30 18:31:33.780002 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:33.779941 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" podStartSLOduration=2.779924648 podStartE2EDuration="2.779924648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:31:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:31:33.774336031 +0000 UTC m=+1990.864398370" watchObservedRunningTime="2026-10-30 18:31:33.779924648 +0000 UTC m=+1990.869986994" Oct 30 18:31:33.818605 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:33.818513 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" Oct 30 18:31:34.166900 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:31:34.166860 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda85dd1f3_b655_4c52_b1ef_c5471365634a.slice/crio-144ca841c1172762fe73ea0035354966b55523940e9a88322c709e5be43d5885 WatchSource:0}: Error finding container 144ca841c1172762fe73ea0035354966b55523940e9a88322c709e5be43d5885: Status 404 returned error can't find the container with id 144ca841c1172762fe73ea0035354966b55523940e9a88322c709e5be43d5885 Oct 30 18:31:34.170097 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:34.170064 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr"] Oct 30 18:31:34.170260 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:34.170222 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:31:34.170327 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:34.170308 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:31:34.170372 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:34.170359 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:31:34.489144 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:34.488986 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" Oct 30 18:31:34.491420 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:34.491397 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1a2436-7676957859-jxlmq" Oct 30 18:31:34.499270 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:34.499249 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" Oct 30 18:31:34.741283 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:34.741191 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" event={"ID":"a85dd1f3-b655-4c52-b1ef-c5471365634a","Type":"ContainerStarted","Data":"9c571179f15191af5b7055f6a9a40b007a0b1635912253adc928269ba5044e04"} Oct 30 18:31:34.741283 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:34.741249 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" event={"ID":"a85dd1f3-b655-4c52-b1ef-c5471365634a","Type":"ContainerStarted","Data":"144ca841c1172762fe73ea0035354966b55523940e9a88322c709e5be43d5885"} Oct 30 18:31:34.743223 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:34.743193 4442 generic.go:356] "Generic (PLEG): container finished" podID="cd55e4f9-4d9d-4fba-a53e-fd3403134906" containerID="4b6fff8d24713f5365a2e0787679ac6779d2151c989a6708a3b1b6ce309d68bb" exitCode=0 Oct 30 18:31:34.743389 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:34.743285 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" event={"ID":"cd55e4f9-4d9d-4fba-a53e-fd3403134906","Type":"ContainerDied","Data":"4b6fff8d24713f5365a2e0787679ac6779d2151c989a6708a3b1b6ce309d68bb"} Oct 30 18:31:34.743840 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:34.743822 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" Oct 30 18:31:34.744240 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:34.744223 4442 scope.go:122] "RemoveContainer" containerID="4b6fff8d24713f5365a2e0787679ac6779d2151c989a6708a3b1b6ce309d68bb" Oct 30 18:31:34.744975 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:34.744966 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" Oct 30 18:31:34.745958 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:34.745921 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:31:34.746069 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:34.746012 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:31:34.746069 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:34.746062 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:31:34.783261 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:34.783213 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" podStartSLOduration=1.7832011250000002 podStartE2EDuration="1.783201125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:31:34.781850737 +0000 UTC m=+1991.871913089" watchObservedRunningTime="2026-10-30 18:31:34.783201125 +0000 UTC m=+1991.873263463" Oct 30 18:31:34.819755 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:34.819728 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" Oct 30 18:31:35.788254 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:35.786208 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" event={"ID":"cd55e4f9-4d9d-4fba-a53e-fd3403134906","Type":"ContainerStarted","Data":"59c69f00f4e8e3d29ab16a527166616434def2b07649b87dae8f67261cef32ce"} Oct 30 18:31:35.834185 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:35.826514 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" Oct 30 18:31:36.577271 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:36.577236 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" Oct 30 18:31:36.598807 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:36.598778 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" Oct 30 18:31:36.608512 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:36.608478 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 30 18:31:36.609053 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:36.609017 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" podUID="cd55e4f9-4d9d-4fba-a53e-fd3403134906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 30 18:31:36.789619 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:36.789588 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" Oct 30 18:31:36.790811 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:36.790787 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" Oct 30 18:31:37.594356 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:37.594318 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 30 18:31:37.594585 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:37.594372 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" podUID="cd55e4f9-4d9d-4fba-a53e-fd3403134906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 30 18:31:38.357161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:38.353236 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4"] Oct 30 18:31:38.386295 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:38.386253 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" Oct 30 18:31:38.390640 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:38.389638 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4"] Oct 30 18:31:38.427158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:38.423592 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/253b93c5-40cc-444a-8836-06044e3d6dd4-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4\" (UID: \"253b93c5-40cc-444a-8836-06044e3d6dd4\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" Oct 30 18:31:38.427158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:38.423696 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/253b93c5-40cc-444a-8836-06044e3d6dd4-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4\" (UID: \"253b93c5-40cc-444a-8836-06044e3d6dd4\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" Oct 30 18:31:38.427158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:38.423797 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt96c\" (UniqueName: \"kubernetes.io/projected/253b93c5-40cc-444a-8836-06044e3d6dd4-kube-api-access-qt96c\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4\" (UID: \"253b93c5-40cc-444a-8836-06044e3d6dd4\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" Oct 30 18:31:38.427158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:38.423963 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/253b93c5-40cc-444a-8836-06044e3d6dd4-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4\" (UID: \"253b93c5-40cc-444a-8836-06044e3d6dd4\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" Oct 30 18:31:38.427158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:38.423993 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/253b93c5-40cc-444a-8836-06044e3d6dd4-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4\" (UID: \"253b93c5-40cc-444a-8836-06044e3d6dd4\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" Oct 30 18:31:38.427158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:38.424021 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/253b93c5-40cc-444a-8836-06044e3d6dd4-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4\" (UID: \"253b93c5-40cc-444a-8836-06044e3d6dd4\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" Oct 30 18:31:38.427158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:38.424064 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/253b93c5-40cc-444a-8836-06044e3d6dd4-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4\" (UID: \"253b93c5-40cc-444a-8836-06044e3d6dd4\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" Oct 30 18:31:38.427158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:38.424093 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/253b93c5-40cc-444a-8836-06044e3d6dd4-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4\" (UID: \"253b93c5-40cc-444a-8836-06044e3d6dd4\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" Oct 30 18:31:38.427158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:38.424127 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/253b93c5-40cc-444a-8836-06044e3d6dd4-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4\" (UID: \"253b93c5-40cc-444a-8836-06044e3d6dd4\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" Oct 30 18:31:38.547161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:38.535068 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/253b93c5-40cc-444a-8836-06044e3d6dd4-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4\" (UID: \"253b93c5-40cc-444a-8836-06044e3d6dd4\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" Oct 30 18:31:38.547161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:38.543713 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/253b93c5-40cc-444a-8836-06044e3d6dd4-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4\" (UID: \"253b93c5-40cc-444a-8836-06044e3d6dd4\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" Oct 30 18:31:38.547161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:38.543853 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt96c\" (UniqueName: \"kubernetes.io/projected/253b93c5-40cc-444a-8836-06044e3d6dd4-kube-api-access-qt96c\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4\" (UID: \"253b93c5-40cc-444a-8836-06044e3d6dd4\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" Oct 30 18:31:38.547161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:38.544058 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/253b93c5-40cc-444a-8836-06044e3d6dd4-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4\" (UID: \"253b93c5-40cc-444a-8836-06044e3d6dd4\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" Oct 30 18:31:38.547161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:38.544091 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/253b93c5-40cc-444a-8836-06044e3d6dd4-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4\" (UID: \"253b93c5-40cc-444a-8836-06044e3d6dd4\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" Oct 30 18:31:38.549161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:38.544122 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/253b93c5-40cc-444a-8836-06044e3d6dd4-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4\" (UID: \"253b93c5-40cc-444a-8836-06044e3d6dd4\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" Oct 30 18:31:38.549161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:38.548336 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/253b93c5-40cc-444a-8836-06044e3d6dd4-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4\" (UID: \"253b93c5-40cc-444a-8836-06044e3d6dd4\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" Oct 30 18:31:38.549161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:38.548377 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/253b93c5-40cc-444a-8836-06044e3d6dd4-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4\" (UID: \"253b93c5-40cc-444a-8836-06044e3d6dd4\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" Oct 30 18:31:38.549161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:38.548414 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/253b93c5-40cc-444a-8836-06044e3d6dd4-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4\" (UID: \"253b93c5-40cc-444a-8836-06044e3d6dd4\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" Oct 30 18:31:38.549161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:38.548500 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/253b93c5-40cc-444a-8836-06044e3d6dd4-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4\" (UID: \"253b93c5-40cc-444a-8836-06044e3d6dd4\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" Oct 30 18:31:38.549161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:38.548996 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/253b93c5-40cc-444a-8836-06044e3d6dd4-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4\" (UID: \"253b93c5-40cc-444a-8836-06044e3d6dd4\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" Oct 30 18:31:38.549568 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:38.549531 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/253b93c5-40cc-444a-8836-06044e3d6dd4-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4\" (UID: \"253b93c5-40cc-444a-8836-06044e3d6dd4\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" Oct 30 18:31:38.550350 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:38.549820 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/253b93c5-40cc-444a-8836-06044e3d6dd4-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4\" (UID: \"253b93c5-40cc-444a-8836-06044e3d6dd4\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" Oct 30 18:31:38.550350 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:38.550069 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/253b93c5-40cc-444a-8836-06044e3d6dd4-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4\" (UID: \"253b93c5-40cc-444a-8836-06044e3d6dd4\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" Oct 30 18:31:38.552197 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:38.552167 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/253b93c5-40cc-444a-8836-06044e3d6dd4-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4\" (UID: \"253b93c5-40cc-444a-8836-06044e3d6dd4\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" Oct 30 18:31:38.552389 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:38.552371 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/253b93c5-40cc-444a-8836-06044e3d6dd4-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4\" (UID: \"253b93c5-40cc-444a-8836-06044e3d6dd4\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" Oct 30 18:31:38.567825 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:38.567787 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/253b93c5-40cc-444a-8836-06044e3d6dd4-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4\" (UID: \"253b93c5-40cc-444a-8836-06044e3d6dd4\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" Oct 30 18:31:38.567934 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:38.567851 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt96c\" (UniqueName: \"kubernetes.io/projected/253b93c5-40cc-444a-8836-06044e3d6dd4-kube-api-access-qt96c\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4\" (UID: \"253b93c5-40cc-444a-8836-06044e3d6dd4\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" Oct 30 18:31:38.594375 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:38.594338 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 30 18:31:38.594522 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:38.594402 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" podUID="cd55e4f9-4d9d-4fba-a53e-fd3403134906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 30 18:31:38.702410 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:38.702378 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" Oct 30 18:31:39.067206 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:31:39.067122 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod253b93c5_40cc_444a_8836_06044e3d6dd4.slice/crio-548d78ae3300a3938433db70d475ea9adb481e36d1c602d7a27e7e80ba7d8305 WatchSource:0}: Error finding container 548d78ae3300a3938433db70d475ea9adb481e36d1c602d7a27e7e80ba7d8305: Status 404 returned error can't find the container with id 548d78ae3300a3938433db70d475ea9adb481e36d1c602d7a27e7e80ba7d8305 Oct 30 18:31:39.069344 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:39.069304 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:31:39.069474 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:39.069388 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:31:39.069474 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:39.069433 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:31:39.071526 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:39.071506 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4"] Oct 30 18:31:39.594053 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:39.594005 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 30 18:31:39.594569 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:39.594081 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" podUID="cd55e4f9-4d9d-4fba-a53e-fd3403134906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 30 18:31:39.822002 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:39.821968 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" event={"ID":"253b93c5-40cc-444a-8836-06044e3d6dd4","Type":"ContainerStarted","Data":"59139cd792b7b832f56a135c3705848e70368a7884613449f94a12139f104d07"} Oct 30 18:31:39.822210 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:39.822045 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" event={"ID":"253b93c5-40cc-444a-8836-06044e3d6dd4","Type":"ContainerStarted","Data":"548d78ae3300a3938433db70d475ea9adb481e36d1c602d7a27e7e80ba7d8305"} Oct 30 18:31:39.867440 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:39.867073 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" podStartSLOduration=1.867053305 podStartE2EDuration="1.867053305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:31:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:31:39.865607038 +0000 UTC m=+1996.955669378" watchObservedRunningTime="2026-10-30 18:31:39.867053305 +0000 UTC m=+1996.957115645" Oct 30 18:31:40.622501 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:40.622468 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 30 18:31:40.623058 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:40.623028 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" podUID="cd55e4f9-4d9d-4fba-a53e-fd3403134906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 30 18:31:40.705199 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:40.704552 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" Oct 30 18:31:40.714497 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:40.705561 4442 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 18:31:40.714497 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:40.705639 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" podUID="253b93c5-40cc-444a-8836-06044e3d6dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 18:31:41.594594 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:41.594562 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 30 18:31:41.594798 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:41.594622 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" podUID="cd55e4f9-4d9d-4fba-a53e-fd3403134906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 30 18:31:41.703350 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:41.703314 4442 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 18:31:41.703958 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:41.703929 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" podUID="253b93c5-40cc-444a-8836-06044e3d6dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 18:31:42.603313 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:42.603285 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" Oct 30 18:31:42.604594 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:42.604571 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" Oct 30 18:31:42.703398 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:42.703361 4442 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 18:31:42.703922 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:42.703419 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" podUID="253b93c5-40cc-444a-8836-06044e3d6dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 18:31:43.703513 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:43.703479 4442 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 18:31:43.703932 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:43.703531 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" podUID="253b93c5-40cc-444a-8836-06044e3d6dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 18:31:44.702891 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:44.702857 4442 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 18:31:44.703078 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:44.702906 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" podUID="253b93c5-40cc-444a-8836-06044e3d6dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 18:31:45.703125 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:45.703086 4442 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 18:31:45.703671 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:45.703185 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" podUID="253b93c5-40cc-444a-8836-06044e3d6dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 18:31:46.703389 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:46.703346 4442 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 18:31:46.703807 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:46.703419 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" podUID="253b93c5-40cc-444a-8836-06044e3d6dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 18:31:47.813157 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:31:47.809430 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:31:48.555570 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:48.555530 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" Oct 30 18:31:48.563161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:48.557586 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" Oct 30 18:31:48.715100 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:48.714508 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" Oct 30 18:31:52.906162 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:52.905507 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk"] Oct 30 18:31:52.910918 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:52.910806 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" Oct 30 18:31:53.003788 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:53.003755 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-09ef1d-openshift-default-dockercfg-rml7c\"" Oct 30 18:31:53.019898 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:53.019118 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk"] Oct 30 18:31:53.125191 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:53.123093 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-credential-socket\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk\" (UID: \"2c46403e-a9d6-48ca-ad0e-34533d5b22ab\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" Oct 30 18:31:53.125191 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:53.123298 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-workload-certs\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk\" (UID: \"2c46403e-a9d6-48ca-ad0e-34533d5b22ab\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" Oct 30 18:31:53.125191 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:53.123400 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-istio-data\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk\" (UID: \"2c46403e-a9d6-48ca-ad0e-34533d5b22ab\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" Oct 30 18:31:53.125191 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:53.123483 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-istiod-ca-cert\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk\" (UID: \"2c46403e-a9d6-48ca-ad0e-34533d5b22ab\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" Oct 30 18:31:53.125191 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:53.123551 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll4nq\" (UniqueName: \"kubernetes.io/projected/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-kube-api-access-ll4nq\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk\" (UID: \"2c46403e-a9d6-48ca-ad0e-34533d5b22ab\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" Oct 30 18:31:53.125191 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:53.123586 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-workload-socket\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk\" (UID: \"2c46403e-a9d6-48ca-ad0e-34533d5b22ab\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" Oct 30 18:31:53.125191 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:53.123621 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-istio-token\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk\" (UID: \"2c46403e-a9d6-48ca-ad0e-34533d5b22ab\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" Oct 30 18:31:53.125191 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:53.123709 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-istio-podinfo\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk\" (UID: \"2c46403e-a9d6-48ca-ad0e-34533d5b22ab\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" Oct 30 18:31:53.125191 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:53.123780 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-istio-envoy\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk\" (UID: \"2c46403e-a9d6-48ca-ad0e-34533d5b22ab\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" Oct 30 18:31:53.266217 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:53.246438 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-istio-data\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk\" (UID: \"2c46403e-a9d6-48ca-ad0e-34533d5b22ab\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" Oct 30 18:31:53.266769 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:53.266712 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-istiod-ca-cert\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk\" (UID: \"2c46403e-a9d6-48ca-ad0e-34533d5b22ab\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" Oct 30 18:31:53.273611 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:53.273563 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll4nq\" (UniqueName: \"kubernetes.io/projected/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-kube-api-access-ll4nq\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk\" (UID: \"2c46403e-a9d6-48ca-ad0e-34533d5b22ab\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" Oct 30 18:31:53.273871 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:53.273830 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-workload-socket\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk\" (UID: \"2c46403e-a9d6-48ca-ad0e-34533d5b22ab\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" Oct 30 18:31:53.274045 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:53.274028 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-istio-token\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk\" (UID: \"2c46403e-a9d6-48ca-ad0e-34533d5b22ab\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" Oct 30 18:31:53.294402 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:53.294351 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-istio-podinfo\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk\" (UID: \"2c46403e-a9d6-48ca-ad0e-34533d5b22ab\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" Oct 30 18:31:53.294714 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:53.294695 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-istio-envoy\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk\" (UID: \"2c46403e-a9d6-48ca-ad0e-34533d5b22ab\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" Oct 30 18:31:53.294926 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:53.294910 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-credential-socket\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk\" (UID: \"2c46403e-a9d6-48ca-ad0e-34533d5b22ab\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" Oct 30 18:31:53.295242 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:53.295218 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-workload-certs\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk\" (UID: \"2c46403e-a9d6-48ca-ad0e-34533d5b22ab\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" Oct 30 18:31:53.296159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:53.295996 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-workload-certs\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk\" (UID: \"2c46403e-a9d6-48ca-ad0e-34533d5b22ab\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" Oct 30 18:31:53.300537 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:53.274689 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-istiod-ca-cert\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk\" (UID: \"2c46403e-a9d6-48ca-ad0e-34533d5b22ab\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" Oct 30 18:31:53.303167 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:53.302269 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-credential-socket\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk\" (UID: \"2c46403e-a9d6-48ca-ad0e-34533d5b22ab\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" Oct 30 18:31:53.307429 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:53.254725 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-istio-data\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk\" (UID: \"2c46403e-a9d6-48ca-ad0e-34533d5b22ab\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" Oct 30 18:31:53.342375 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:53.342281 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-workload-socket\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk\" (UID: \"2c46403e-a9d6-48ca-ad0e-34533d5b22ab\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" Oct 30 18:31:53.447159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:53.434388 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-istio-token\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk\" (UID: \"2c46403e-a9d6-48ca-ad0e-34533d5b22ab\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" Oct 30 18:31:53.511158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:53.507807 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-istio-podinfo\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk\" (UID: \"2c46403e-a9d6-48ca-ad0e-34533d5b22ab\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" Oct 30 18:31:53.580885 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:53.511628 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-istio-envoy\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk\" (UID: \"2c46403e-a9d6-48ca-ad0e-34533d5b22ab\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" Oct 30 18:31:53.580885 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:53.513356 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll4nq\" (UniqueName: \"kubernetes.io/projected/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-kube-api-access-ll4nq\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk\" (UID: \"2c46403e-a9d6-48ca-ad0e-34533d5b22ab\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" Oct 30 18:31:53.655446 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:53.655408 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" Oct 30 18:31:54.110559 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:31:54.109441 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c46403e_a9d6_48ca_ad0e_34533d5b22ab.slice/crio-627572e798908d4873fbc32c64572c625a1985edd0c7783f3246f6cd676f7262 WatchSource:0}: Error finding container 627572e798908d4873fbc32c64572c625a1985edd0c7783f3246f6cd676f7262: Status 404 returned error can't find the container with id 627572e798908d4873fbc32c64572c625a1985edd0c7783f3246f6cd676f7262 Oct 30 18:31:54.111916 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:54.111871 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:31:54.112046 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:54.111964 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:31:54.112046 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:54.112011 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:31:54.113506 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:54.113480 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk"] Oct 30 18:31:54.479421 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:54.478196 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" event={"ID":"2c46403e-a9d6-48ca-ad0e-34533d5b22ab","Type":"ContainerStarted","Data":"b2d680892430d282d2b2ad6922e0aba3e381d7d7b0de2ccde62ab1ff08d5bf03"} Oct 30 18:31:54.479421 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:54.478262 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" event={"ID":"2c46403e-a9d6-48ca-ad0e-34533d5b22ab","Type":"ContainerStarted","Data":"627572e798908d4873fbc32c64572c625a1985edd0c7783f3246f6cd676f7262"} Oct 30 18:31:54.542764 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:54.542687 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" podStartSLOduration=2.5426645089999997 podStartE2EDuration="2.542664509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:31:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:31:54.522787207 +0000 UTC m=+2011.612849547" watchObservedRunningTime="2026-10-30 18:31:54.542664509 +0000 UTC m=+2011.632726853" Oct 30 18:31:54.656318 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:54.656236 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" Oct 30 18:31:55.663034 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:55.663003 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" Oct 30 18:31:56.020305 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:56.020260 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:31:56.020479 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:56.020323 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" podUID="4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:56.494022 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:56.493993 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" Oct 30 18:31:56.495294 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:56.495221 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" Oct 30 18:31:59.548800 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:59.548715 4442 generic.go:356] "Generic (PLEG): container finished" podID="4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54" containerID="428227e6c5a7d68f2d063fd873665ee979ec1ec52df3f48dd50ed5b36d64cd24" exitCode=0 Oct 30 18:31:59.548800 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:59.548758 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" event={"ID":"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54","Type":"ContainerDied","Data":"428227e6c5a7d68f2d063fd873665ee979ec1ec52df3f48dd50ed5b36d64cd24"} Oct 30 18:31:59.549414 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:59.549264 4442 scope.go:122] "RemoveContainer" containerID="428227e6c5a7d68f2d063fd873665ee979ec1ec52df3f48dd50ed5b36d64cd24" Oct 30 18:31:59.550967 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:59.550933 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:31:59.551091 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:59.551018 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:31:59.551091 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:31:59.551064 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:32:00.565637 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:00.565603 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" event={"ID":"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54","Type":"ContainerStarted","Data":"44aa1e3f40f6b9c728b09f0d8f4db7b7e8e92c838e962d13bd77546b0fab7e06"} Oct 30 18:32:00.567456 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:00.567426 4442 generic.go:356] "Generic (PLEG): container finished" podID="fb9a48e1-6d20-4f44-8dc6-eb0e6125276d" containerID="6d4813a4d8e1241c4cecbf9c43bbf423127bd8b4f6fc0bff157e1e5dddc1868b" exitCode=0 Oct 30 18:32:00.567581 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:00.567496 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" event={"ID":"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d","Type":"ContainerDied","Data":"6d4813a4d8e1241c4cecbf9c43bbf423127bd8b4f6fc0bff157e1e5dddc1868b"} Oct 30 18:32:00.567899 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:00.567886 4442 scope.go:122] "RemoveContainer" containerID="6d4813a4d8e1241c4cecbf9c43bbf423127bd8b4f6fc0bff157e1e5dddc1868b" Oct 30 18:32:00.569596 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:00.569569 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:32:00.569697 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:00.569643 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:32:00.569697 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:00.569679 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:32:01.020320 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:01.020278 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" Oct 30 18:32:01.020973 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:01.020944 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" Oct 30 18:32:01.021697 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:01.021665 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 30 18:32:01.021830 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:01.021721 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" podUID="4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 30 18:32:01.614969 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:01.614933 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" event={"ID":"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d","Type":"ContainerStarted","Data":"9d3eafd4d8bbded92b3012b35447c3b518dd880574e66189e06c655e423764d5"} Oct 30 18:32:01.797795 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:01.797760 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" Oct 30 18:32:01.798868 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:01.798843 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 30 18:32:01.798980 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:01.798886 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" podUID="fb9a48e1-6d20-4f44-8dc6-eb0e6125276d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 30 18:32:02.013211 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:02.012555 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 30 18:32:02.013211 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:02.012616 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" podUID="4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 30 18:32:02.800161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:02.799262 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 30 18:32:02.800161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:02.799318 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" podUID="fb9a48e1-6d20-4f44-8dc6-eb0e6125276d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 30 18:32:03.006535 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:03.006498 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 30 18:32:03.006766 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:03.006569 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" podUID="4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 30 18:32:03.807017 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:03.806624 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 30 18:32:03.807017 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:03.806687 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" podUID="fb9a48e1-6d20-4f44-8dc6-eb0e6125276d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 30 18:32:04.006449 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:04.006417 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 30 18:32:04.006642 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:04.006472 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" podUID="4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 30 18:32:04.180191 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:04.179899 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t"] Oct 30 18:32:04.208351 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:04.208321 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" Oct 30 18:32:04.209692 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:04.209542 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t"] Oct 30 18:32:04.239093 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:04.233862 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-24b124-openshift-default-dockercfg-qbtjb\"" Oct 30 18:32:04.303199 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:04.297240 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-workload-socket\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t\" (UID: \"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" Oct 30 18:32:04.303199 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:04.297433 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-credential-socket\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t\" (UID: \"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" Oct 30 18:32:04.303199 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:04.297481 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-istio-podinfo\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t\" (UID: \"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" Oct 30 18:32:04.303199 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:04.297541 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-istio-token\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t\" (UID: \"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" Oct 30 18:32:04.303199 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:04.297600 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-istiod-ca-cert\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t\" (UID: \"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" Oct 30 18:32:04.303199 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:04.297667 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-istio-data\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t\" (UID: \"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" Oct 30 18:32:04.303199 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:04.297744 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-istio-envoy\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t\" (UID: \"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" Oct 30 18:32:04.303199 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:04.297773 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whlcf\" (UniqueName: \"kubernetes.io/projected/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-kube-api-access-whlcf\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t\" (UID: \"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" Oct 30 18:32:04.303199 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:04.297801 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-workload-certs\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t\" (UID: \"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" Oct 30 18:32:04.410199 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:04.407235 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-istio-envoy\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t\" (UID: \"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" Oct 30 18:32:04.410199 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:04.407283 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whlcf\" (UniqueName: \"kubernetes.io/projected/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-kube-api-access-whlcf\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t\" (UID: \"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" Oct 30 18:32:04.410199 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:04.407321 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-workload-certs\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t\" (UID: \"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" Oct 30 18:32:04.410199 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:04.407401 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-workload-socket\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t\" (UID: \"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" Oct 30 18:32:04.410199 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:04.407539 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-credential-socket\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t\" (UID: \"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" Oct 30 18:32:04.410199 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:04.407577 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-istio-podinfo\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t\" (UID: \"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" Oct 30 18:32:04.410199 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:04.407633 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-istio-token\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t\" (UID: \"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" Oct 30 18:32:04.410199 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:04.407686 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-istiod-ca-cert\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t\" (UID: \"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" Oct 30 18:32:04.410199 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:04.407747 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-istio-data\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t\" (UID: \"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" Oct 30 18:32:04.410199 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:04.408224 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-istio-data\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t\" (UID: \"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" Oct 30 18:32:04.499254 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:04.492514 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-credential-socket\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t\" (UID: \"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" Oct 30 18:32:04.499254 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:04.493202 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-workload-certs\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t\" (UID: \"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" Oct 30 18:32:04.499254 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:04.493461 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-workload-socket\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t\" (UID: \"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" Oct 30 18:32:04.562858 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:04.500287 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-istiod-ca-cert\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t\" (UID: \"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" Oct 30 18:32:04.639118 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:04.639039 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whlcf\" (UniqueName: \"kubernetes.io/projected/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-kube-api-access-whlcf\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t\" (UID: \"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" Oct 30 18:32:04.643843 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:04.643813 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-istio-envoy\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t\" (UID: \"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" Oct 30 18:32:04.654286 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:04.653265 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-istio-podinfo\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t\" (UID: \"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" Oct 30 18:32:04.654286 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:04.653841 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-istio-token\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t\" (UID: \"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" Oct 30 18:32:04.798799 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:04.798690 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 30 18:32:04.799275 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:04.799232 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" podUID="fb9a48e1-6d20-4f44-8dc6-eb0e6125276d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 30 18:32:04.894378 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:04.894339 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" Oct 30 18:32:05.006011 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:05.005981 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 30 18:32:05.006149 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:05.006043 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" podUID="4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 30 18:32:05.290723 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:05.290443 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t"] Oct 30 18:32:05.290723 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:32:05.290665 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cbde8cc_5ae5_4133_9c52_cb30a81b91b1.slice/crio-ffb396db5245ceb20545af738d074ce0e96559fb4ae16efb450fc1035a3c89ae WatchSource:0}: Error finding container ffb396db5245ceb20545af738d074ce0e96559fb4ae16efb450fc1035a3c89ae: Status 404 returned error can't find the container with id ffb396db5245ceb20545af738d074ce0e96559fb4ae16efb450fc1035a3c89ae Oct 30 18:32:05.294202 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:05.294169 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:32:05.294322 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:05.294236 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:32:05.294322 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:05.294269 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:32:05.804166 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:05.803320 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 30 18:32:05.804166 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:05.803383 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" podUID="fb9a48e1-6d20-4f44-8dc6-eb0e6125276d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 30 18:32:05.819156 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:05.817289 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" event={"ID":"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1","Type":"ContainerStarted","Data":"c888cfd579cfb4cbc1e6dfe502cd0317d0e214dfc0a411ed6737b1d2ce023bc1"} Oct 30 18:32:05.819156 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:05.817360 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" event={"ID":"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1","Type":"ContainerStarted","Data":"ffb396db5245ceb20545af738d074ce0e96559fb4ae16efb450fc1035a3c89ae"} Oct 30 18:32:05.895159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:05.880600 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" podStartSLOduration=1.880579701 podStartE2EDuration="1.880579701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:32:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:05.876852187 +0000 UTC m=+2022.966914527" watchObservedRunningTime="2026-10-30 18:32:05.880579701 +0000 UTC m=+2022.970642040" Oct 30 18:32:05.905125 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:05.898409 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" Oct 30 18:32:06.006564 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:06.006523 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 30 18:32:06.006908 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:06.006872 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" podUID="4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 30 18:32:06.798441 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:06.798406 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 30 18:32:06.798668 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:06.798465 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" podUID="fb9a48e1-6d20-4f44-8dc6-eb0e6125276d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 30 18:32:06.901644 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:06.901603 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" Oct 30 18:32:07.011844 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:07.011820 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" Oct 30 18:32:07.012261 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:07.012243 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" Oct 30 18:32:07.013195 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:07.013177 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" Oct 30 18:32:07.851174 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:07.849769 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" Oct 30 18:32:07.851174 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:07.850565 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" Oct 30 18:32:07.906188 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:07.905915 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" Oct 30 18:32:07.907452 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:07.907295 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" Oct 30 18:32:07.908304 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:07.908270 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" Oct 30 18:32:17.608211 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:32:17.608170 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:32:25.528364 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:25.528312 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:32:25.528917 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:25.528392 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" podUID="eb94bb4c-aa2a-4c61-b986-03dd313ab6e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:32:27.251189 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:27.251115 4442 generic.go:356] "Generic (PLEG): container finished" podID="a85dd1f3-b655-4c52-b1ef-c5471365634a" containerID="9c571179f15191af5b7055f6a9a40b007a0b1635912253adc928269ba5044e04" exitCode=0 Oct 30 18:32:27.251749 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:27.251347 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" event={"ID":"a85dd1f3-b655-4c52-b1ef-c5471365634a","Type":"ContainerDied","Data":"9c571179f15191af5b7055f6a9a40b007a0b1635912253adc928269ba5044e04"} Oct 30 18:32:27.251833 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:27.251811 4442 scope.go:122] "RemoveContainer" containerID="9c571179f15191af5b7055f6a9a40b007a0b1635912253adc928269ba5044e04" Oct 30 18:32:27.253912 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:27.253877 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:32:27.254066 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:27.253952 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:32:27.254066 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:27.253993 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:32:27.254431 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:27.254407 4442 generic.go:356] "Generic (PLEG): container finished" podID="eb94bb4c-aa2a-4c61-b986-03dd313ab6e0" containerID="bda4e615fb2dbd2e4c3d093e91f3fcc329e846cc637cecd17211dbb72f73e2de" exitCode=0 Oct 30 18:32:27.254570 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:27.254486 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" event={"ID":"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0","Type":"ContainerDied","Data":"bda4e615fb2dbd2e4c3d093e91f3fcc329e846cc637cecd17211dbb72f73e2de"} Oct 30 18:32:27.255084 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:27.255061 4442 scope.go:122] "RemoveContainer" containerID="bda4e615fb2dbd2e4c3d093e91f3fcc329e846cc637cecd17211dbb72f73e2de" Oct 30 18:32:27.256907 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:27.256878 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:32:27.257024 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:27.257003 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:32:27.257091 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:27.257063 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:32:28.265870 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:28.265837 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" event={"ID":"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0","Type":"ContainerStarted","Data":"3d90cdd69073ad22d74141faf68bbc51694ef320f274e2d726548b4d4d29dd16"} Oct 30 18:32:28.268402 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:28.268376 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" event={"ID":"a85dd1f3-b655-4c52-b1ef-c5471365634a","Type":"ContainerStarted","Data":"2c0cdec945b34a8b050175d5d48fe7c3bd1d39781f53d2d0dd284dab4af85eab"} Oct 30 18:32:28.716957 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:28.716928 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" Oct 30 18:32:28.717331 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:28.717311 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 18:32:28.717391 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:28.717357 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" podUID="eb94bb4c-aa2a-4c61-b986-03dd313ab6e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 18:32:28.843159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:28.829561 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" Oct 30 18:32:28.843159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:28.829704 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 30 18:32:28.843159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:28.829752 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" podUID="a85dd1f3-b655-4c52-b1ef-c5471365634a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 30 18:32:29.718010 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:29.717979 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 18:32:29.718440 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:29.718032 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" podUID="eb94bb4c-aa2a-4c61-b986-03dd313ab6e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 18:32:29.820121 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:29.820083 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 30 18:32:29.820341 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:29.820157 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" podUID="a85dd1f3-b655-4c52-b1ef-c5471365634a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 30 18:32:30.717852 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:30.717290 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 18:32:30.718081 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:30.717894 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" podUID="eb94bb4c-aa2a-4c61-b986-03dd313ab6e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 18:32:30.820277 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:30.819792 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 30 18:32:30.820557 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:30.820530 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" podUID="a85dd1f3-b655-4c52-b1ef-c5471365634a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 30 18:32:31.718086 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:31.718046 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 18:32:31.718632 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:31.718578 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" podUID="eb94bb4c-aa2a-4c61-b986-03dd313ab6e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 18:32:31.827173 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:31.823338 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 30 18:32:31.827780 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:31.827741 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" podUID="a85dd1f3-b655-4c52-b1ef-c5471365634a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 30 18:32:32.718194 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:32.717699 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 18:32:32.718194 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:32.717782 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" podUID="eb94bb4c-aa2a-4c61-b986-03dd313ab6e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 18:32:32.820264 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:32.819814 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 30 18:32:32.820264 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:32.819861 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" podUID="a85dd1f3-b655-4c52-b1ef-c5471365634a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 30 18:32:33.717242 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:33.717203 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 18:32:33.717431 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:33.717263 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" podUID="eb94bb4c-aa2a-4c61-b986-03dd313ab6e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 18:32:33.820157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:33.820100 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 30 18:32:33.820642 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:33.820186 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" podUID="a85dd1f3-b655-4c52-b1ef-c5471365634a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 30 18:32:34.724262 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:34.724228 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" Oct 30 18:32:34.724978 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:34.724951 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" Oct 30 18:32:34.726188 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:34.726162 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" Oct 30 18:32:34.828500 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:34.828462 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" Oct 30 18:32:34.829978 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:34.829953 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" Oct 30 18:32:34.830991 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:34.830972 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" Oct 30 18:32:43.748945 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:43.748901 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d82e82-7d456c8bbc-pmqhf"] Oct 30 18:32:43.752945 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:43.752923 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d82e82-7d456c8bbc-pmqhf" Oct 30 18:32:43.773792 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:43.773753 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d82e82-7d456c8bbc-pmqhf"] Oct 30 18:32:43.778519 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:43.778495 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-d82e82\"" Oct 30 18:32:43.778687 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:43.778603 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-d82e82-dockercfg-5tpnp\"" Oct 30 18:32:43.881160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:43.880107 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f367afff-a23b-47d2-944e-0ab238f13bd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-d82e82-7d456c8bbc-pmqhf\" (UID: \"f367afff-a23b-47d2-944e-0ab238f13bd0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d82e82-7d456c8bbc-pmqhf" Oct 30 18:32:43.881160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:43.880310 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpm6f\" (UniqueName: \"kubernetes.io/projected/f367afff-a23b-47d2-944e-0ab238f13bd0-kube-api-access-mpm6f\") pod \"payload-pre-processing-e2e-shared-b-w1-d82e82-7d456c8bbc-pmqhf\" (UID: \"f367afff-a23b-47d2-944e-0ab238f13bd0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d82e82-7d456c8bbc-pmqhf" Oct 30 18:32:44.160224 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:44.112759 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpm6f\" (UniqueName: \"kubernetes.io/projected/f367afff-a23b-47d2-944e-0ab238f13bd0-kube-api-access-mpm6f\") pod \"payload-pre-processing-e2e-shared-b-w1-d82e82-7d456c8bbc-pmqhf\" (UID: \"f367afff-a23b-47d2-944e-0ab238f13bd0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d82e82-7d456c8bbc-pmqhf" Oct 30 18:32:44.160878 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:44.160842 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f367afff-a23b-47d2-944e-0ab238f13bd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-d82e82-7d456c8bbc-pmqhf\" (UID: \"f367afff-a23b-47d2-944e-0ab238f13bd0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d82e82-7d456c8bbc-pmqhf" Oct 30 18:32:44.240580 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:44.164225 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f367afff-a23b-47d2-944e-0ab238f13bd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-d82e82-7d456c8bbc-pmqhf\" (UID: \"f367afff-a23b-47d2-944e-0ab238f13bd0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d82e82-7d456c8bbc-pmqhf" Oct 30 18:32:44.485617 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:44.465961 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpm6f\" (UniqueName: \"kubernetes.io/projected/f367afff-a23b-47d2-944e-0ab238f13bd0-kube-api-access-mpm6f\") pod \"payload-pre-processing-e2e-shared-b-w1-d82e82-7d456c8bbc-pmqhf\" (UID: \"f367afff-a23b-47d2-944e-0ab238f13bd0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d82e82-7d456c8bbc-pmqhf" Oct 30 18:32:44.583182 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:44.536673 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-d82e82-fdf98fb94-dtq2s"] Oct 30 18:32:44.664874 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:44.664204 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-d82e82-fdf98fb94-dtq2s"] Oct 30 18:32:44.664874 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:44.664320 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d82e82-fdf98fb94-dtq2s" Oct 30 18:32:44.675935 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:44.673803 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d82e82-7d456c8bbc-pmqhf" Oct 30 18:32:44.768627 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:44.768408 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc6d66de-c4d6-4d9f-bfbf-4cfee09dcbda-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-d82e82-fdf98fb94-dtq2s\" (UID: \"bc6d66de-c4d6-4d9f-bfbf-4cfee09dcbda\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d82e82-fdf98fb94-dtq2s" Oct 30 18:32:44.768627 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:44.768493 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8cl4\" (UniqueName: \"kubernetes.io/projected/bc6d66de-c4d6-4d9f-bfbf-4cfee09dcbda-kube-api-access-p8cl4\") pod \"payload-processing-e2e-shared-b-w1-d82e82-fdf98fb94-dtq2s\" (UID: \"bc6d66de-c4d6-4d9f-bfbf-4cfee09dcbda\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d82e82-fdf98fb94-dtq2s" Oct 30 18:32:44.871618 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:44.871305 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8cl4\" (UniqueName: \"kubernetes.io/projected/bc6d66de-c4d6-4d9f-bfbf-4cfee09dcbda-kube-api-access-p8cl4\") pod \"payload-processing-e2e-shared-b-w1-d82e82-fdf98fb94-dtq2s\" (UID: \"bc6d66de-c4d6-4d9f-bfbf-4cfee09dcbda\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d82e82-fdf98fb94-dtq2s" Oct 30 18:32:44.871618 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:44.871585 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc6d66de-c4d6-4d9f-bfbf-4cfee09dcbda-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-d82e82-fdf98fb94-dtq2s\" (UID: \"bc6d66de-c4d6-4d9f-bfbf-4cfee09dcbda\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d82e82-fdf98fb94-dtq2s" Oct 30 18:32:44.872601 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:44.872556 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc6d66de-c4d6-4d9f-bfbf-4cfee09dcbda-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-d82e82-fdf98fb94-dtq2s\" (UID: \"bc6d66de-c4d6-4d9f-bfbf-4cfee09dcbda\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d82e82-fdf98fb94-dtq2s" Oct 30 18:32:44.896230 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:44.896196 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8cl4\" (UniqueName: \"kubernetes.io/projected/bc6d66de-c4d6-4d9f-bfbf-4cfee09dcbda-kube-api-access-p8cl4\") pod \"payload-processing-e2e-shared-b-w1-d82e82-fdf98fb94-dtq2s\" (UID: \"bc6d66de-c4d6-4d9f-bfbf-4cfee09dcbda\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d82e82-fdf98fb94-dtq2s" Oct 30 18:32:45.004477 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:45.004396 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d82e82-fdf98fb94-dtq2s" Oct 30 18:32:45.105343 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:32:45.104385 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf367afff_a23b_47d2_944e_0ab238f13bd0.slice/crio-afa603d097de270129f2e80a5994651c9351d04293c63875e302bb92c6d94db8 WatchSource:0}: Error finding container afa603d097de270129f2e80a5994651c9351d04293c63875e302bb92c6d94db8: Status 404 returned error can't find the container with id afa603d097de270129f2e80a5994651c9351d04293c63875e302bb92c6d94db8 Oct 30 18:32:45.106289 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:45.106260 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d82e82-7d456c8bbc-pmqhf"] Oct 30 18:32:45.362816 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:45.362779 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-d82e82-fdf98fb94-dtq2s"] Oct 30 18:32:45.363773 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:32:45.363751 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc6d66de_c4d6_4d9f_bfbf_4cfee09dcbda.slice/crio-326bcf1cf0282a725c99d4d76b8da4d32fd0e72f471bd618e990848ece078421 WatchSource:0}: Error finding container 326bcf1cf0282a725c99d4d76b8da4d32fd0e72f471bd618e990848ece078421: Status 404 returned error can't find the container with id 326bcf1cf0282a725c99d4d76b8da4d32fd0e72f471bd618e990848ece078421 Oct 30 18:32:45.694206 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:45.693532 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d82e82-fdf98fb94-dtq2s" event={"ID":"bc6d66de-c4d6-4d9f-bfbf-4cfee09dcbda","Type":"ContainerStarted","Data":"dba1d4bd8ac27bc3040bc6cd612dd127443e6b946c04a7fb6e4cc0c56c602d53"} Oct 30 18:32:45.694206 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:45.693599 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d82e82-fdf98fb94-dtq2s" event={"ID":"bc6d66de-c4d6-4d9f-bfbf-4cfee09dcbda","Type":"ContainerStarted","Data":"326bcf1cf0282a725c99d4d76b8da4d32fd0e72f471bd618e990848ece078421"} Oct 30 18:32:45.695207 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:45.695171 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d82e82-fdf98fb94-dtq2s" Oct 30 18:32:45.699375 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:45.699349 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d82e82-7d456c8bbc-pmqhf" event={"ID":"f367afff-a23b-47d2-944e-0ab238f13bd0","Type":"ContainerStarted","Data":"fa3f2cf66434fa744997c2f2786b72c0333f1e2638b2e2485e6d2c1b276b7214"} Oct 30 18:32:45.699523 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:45.699397 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d82e82-7d456c8bbc-pmqhf" event={"ID":"f367afff-a23b-47d2-944e-0ab238f13bd0","Type":"ContainerStarted","Data":"afa603d097de270129f2e80a5994651c9351d04293c63875e302bb92c6d94db8"} Oct 30 18:32:45.699731 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:45.699715 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d82e82-7d456c8bbc-pmqhf" Oct 30 18:32:45.779374 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:45.766242 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d82e82-fdf98fb94-dtq2s" podStartSLOduration=2.766219247 podStartE2EDuration="2.766219247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:32:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:45.733533776 +0000 UTC m=+2062.823596121" watchObservedRunningTime="2026-10-30 18:32:45.766219247 +0000 UTC m=+2062.856281587" Oct 30 18:32:45.846159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:45.837660 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d82e82-7d456c8bbc-pmqhf" podStartSLOduration=2.837637463 podStartE2EDuration="2.837637463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:32:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:45.780348735 +0000 UTC m=+2062.870411075" watchObservedRunningTime="2026-10-30 18:32:45.837637463 +0000 UTC m=+2062.927699806" Oct 30 18:32:47.695163 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:32:47.684322 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:32:55.905251 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:55.905161 4442 generic.go:356] "Generic (PLEG): container finished" podID="2c46403e-a9d6-48ca-ad0e-34533d5b22ab" containerID="b2d680892430d282d2b2ad6922e0aba3e381d7d7b0de2ccde62ab1ff08d5bf03" exitCode=0 Oct 30 18:32:55.905769 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:55.905240 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" event={"ID":"2c46403e-a9d6-48ca-ad0e-34533d5b22ab","Type":"ContainerDied","Data":"b2d680892430d282d2b2ad6922e0aba3e381d7d7b0de2ccde62ab1ff08d5bf03"} Oct 30 18:32:55.905844 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:55.905814 4442 scope.go:122] "RemoveContainer" containerID="b2d680892430d282d2b2ad6922e0aba3e381d7d7b0de2ccde62ab1ff08d5bf03" Oct 30 18:32:55.907423 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:55.907394 4442 generic.go:356] "Generic (PLEG): container finished" podID="bd132ea6-9517-4402-b079-16d0af22572f" containerID="07a5613dbd40355550531d62d61a6600642eec76b158886aadb432e9d75a5045" exitCode=0 Oct 30 18:32:55.907585 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:55.907451 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" event={"ID":"bd132ea6-9517-4402-b079-16d0af22572f","Type":"ContainerDied","Data":"07a5613dbd40355550531d62d61a6600642eec76b158886aadb432e9d75a5045"} Oct 30 18:32:55.907829 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:55.907815 4442 scope.go:122] "RemoveContainer" containerID="07a5613dbd40355550531d62d61a6600642eec76b158886aadb432e9d75a5045" Oct 30 18:32:55.907903 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:55.907839 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:32:55.907943 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:55.907910 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:32:55.907981 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:55.907946 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:32:55.909324 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:55.909298 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:32:55.909436 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:55.909357 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:32:55.909436 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:55.909387 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:32:56.494120 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:56.494090 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" Oct 30 18:32:56.710689 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:56.710660 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d82e82-7d456c8bbc-pmqhf" Oct 30 18:32:56.993230 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:56.992834 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" event={"ID":"2c46403e-a9d6-48ca-ad0e-34533d5b22ab","Type":"ContainerStarted","Data":"74aba9be06e1f2a0300e6db926ed683c3f9f22280513bee037595b8d6e8f8481"} Oct 30 18:32:56.997292 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:56.997259 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" event={"ID":"bd132ea6-9517-4402-b079-16d0af22572f","Type":"ContainerStarted","Data":"0e69866c5a7d3591fbbbf034fca1143075ac005f79bf3ee910011984d72242a1"} Oct 30 18:32:57.491158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:57.490596 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" Oct 30 18:32:57.491158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:57.490765 4442 patch_prober.go:29] interesting pod/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 18:32:57.491158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:57.490818 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" podUID="bd132ea6-9517-4402-b079-16d0af22572f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 18:32:57.668161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:57.666579 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" Oct 30 18:32:57.668161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:57.666706 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 30 18:32:57.668161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:57.666762 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" podUID="2c46403e-a9d6-48ca-ad0e-34533d5b22ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 30 18:32:57.780216 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:57.775326 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d82e82-fdf98fb94-dtq2s" Oct 30 18:32:58.489628 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:58.489599 4442 patch_prober.go:29] interesting pod/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 18:32:58.490173 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:58.490109 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" podUID="bd132ea6-9517-4402-b079-16d0af22572f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 18:32:58.660534 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:58.660495 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 30 18:32:58.660825 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:58.660796 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" podUID="2c46403e-a9d6-48ca-ad0e-34533d5b22ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 30 18:32:59.489716 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:59.489676 4442 patch_prober.go:29] interesting pod/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 18:32:59.490098 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:59.489743 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" podUID="bd132ea6-9517-4402-b079-16d0af22572f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 18:32:59.663155 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:59.659552 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 30 18:32:59.663155 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:32:59.659613 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" podUID="2c46403e-a9d6-48ca-ad0e-34533d5b22ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 30 18:33:00.490012 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:00.489977 4442 patch_prober.go:29] interesting pod/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 18:33:00.490617 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:00.490033 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" podUID="bd132ea6-9517-4402-b079-16d0af22572f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 18:33:00.658157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:00.657244 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 30 18:33:00.658157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:00.657323 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" podUID="2c46403e-a9d6-48ca-ad0e-34533d5b22ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 30 18:33:01.498156 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:01.493520 4442 patch_prober.go:29] interesting pod/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 18:33:01.498156 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:01.493596 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" podUID="bd132ea6-9517-4402-b079-16d0af22572f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 18:33:01.659607 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:01.659573 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 30 18:33:01.659873 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:01.659839 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" podUID="2c46403e-a9d6-48ca-ad0e-34533d5b22ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 30 18:33:02.489357 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:02.489321 4442 patch_prober.go:29] interesting pod/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 18:33:02.489582 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:02.489380 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" podUID="bd132ea6-9517-4402-b079-16d0af22572f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 18:33:02.947911 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:02.947880 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" Oct 30 18:33:02.948636 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:02.948612 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" Oct 30 18:33:02.950669 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:02.949828 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" Oct 30 18:33:03.495036 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:03.495006 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" Oct 30 18:33:03.495482 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:03.495458 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" Oct 30 18:33:03.496303 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:03.496284 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" Oct 30 18:33:12.386840 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:12.386738 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-1a2436/auth-test-model-1a2436-kserve-547bf9b957-hhcgp"] Oct 30 18:33:12.398271 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:12.398242 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-1a2436/auth-test-model-1a2436-kserve-547bf9b957-hhcgp" Oct 30 18:33:12.411371 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:12.411339 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-1a2436/auth-test-model-1a2436-kserve-547bf9b957-hhcgp"] Oct 30 18:33:12.415359 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:12.415337 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-1a2436\"/\"default-dockercfg-kz6gz\"" Oct 30 18:33:12.415539 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:12.415389 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-1a2436\"/\"auth-test-model-1a2436-kserve-self-signed-certs\"" Oct 30 18:33:12.427413 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:12.427384 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-1a2436\"/\"kube-root-ca.crt\"" Oct 30 18:33:12.427589 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:12.427477 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-1a2436\"/\"openshift-service-ca.crt\"" Oct 30 18:33:12.505102 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:12.505071 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2-tls-certs\") pod \"auth-test-model-1a2436-kserve-547bf9b957-hhcgp\" (UID: \"b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2\") " pod="ai-tenant-e2e-shared-a-w4-1a2436/auth-test-model-1a2436-kserve-547bf9b957-hhcgp" Oct 30 18:33:12.505316 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:12.505147 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2-model-cache\") pod \"auth-test-model-1a2436-kserve-547bf9b957-hhcgp\" (UID: \"b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2\") " pod="ai-tenant-e2e-shared-a-w4-1a2436/auth-test-model-1a2436-kserve-547bf9b957-hhcgp" Oct 30 18:33:12.505316 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:12.505270 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2-dshm\") pod \"auth-test-model-1a2436-kserve-547bf9b957-hhcgp\" (UID: \"b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2\") " pod="ai-tenant-e2e-shared-a-w4-1a2436/auth-test-model-1a2436-kserve-547bf9b957-hhcgp" Oct 30 18:33:12.505424 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:12.505326 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2-tmp-dir\") pod \"auth-test-model-1a2436-kserve-547bf9b957-hhcgp\" (UID: \"b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2\") " pod="ai-tenant-e2e-shared-a-w4-1a2436/auth-test-model-1a2436-kserve-547bf9b957-hhcgp" Oct 30 18:33:12.505471 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:12.505424 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2-home\") pod \"auth-test-model-1a2436-kserve-547bf9b957-hhcgp\" (UID: \"b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2\") " pod="ai-tenant-e2e-shared-a-w4-1a2436/auth-test-model-1a2436-kserve-547bf9b957-hhcgp" Oct 30 18:33:12.505529 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:12.505514 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvsmx\" (UniqueName: \"kubernetes.io/projected/b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2-kube-api-access-wvsmx\") pod \"auth-test-model-1a2436-kserve-547bf9b957-hhcgp\" (UID: \"b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2\") " pod="ai-tenant-e2e-shared-a-w4-1a2436/auth-test-model-1a2436-kserve-547bf9b957-hhcgp" Oct 30 18:33:12.606784 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:12.606707 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2-model-cache\") pod \"auth-test-model-1a2436-kserve-547bf9b957-hhcgp\" (UID: \"b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2\") " pod="ai-tenant-e2e-shared-a-w4-1a2436/auth-test-model-1a2436-kserve-547bf9b957-hhcgp" Oct 30 18:33:12.606963 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:12.606825 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2-dshm\") pod \"auth-test-model-1a2436-kserve-547bf9b957-hhcgp\" (UID: \"b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2\") " pod="ai-tenant-e2e-shared-a-w4-1a2436/auth-test-model-1a2436-kserve-547bf9b957-hhcgp" Oct 30 18:33:12.606963 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:12.606889 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2-tmp-dir\") pod \"auth-test-model-1a2436-kserve-547bf9b957-hhcgp\" (UID: \"b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2\") " pod="ai-tenant-e2e-shared-a-w4-1a2436/auth-test-model-1a2436-kserve-547bf9b957-hhcgp" Oct 30 18:33:12.606963 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:12.606932 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2-home\") pod \"auth-test-model-1a2436-kserve-547bf9b957-hhcgp\" (UID: \"b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2\") " pod="ai-tenant-e2e-shared-a-w4-1a2436/auth-test-model-1a2436-kserve-547bf9b957-hhcgp" Oct 30 18:33:12.607174 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:12.607080 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvsmx\" (UniqueName: \"kubernetes.io/projected/b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2-kube-api-access-wvsmx\") pod \"auth-test-model-1a2436-kserve-547bf9b957-hhcgp\" (UID: \"b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2\") " pod="ai-tenant-e2e-shared-a-w4-1a2436/auth-test-model-1a2436-kserve-547bf9b957-hhcgp" Oct 30 18:33:12.607240 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:12.607230 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2-tls-certs\") pod \"auth-test-model-1a2436-kserve-547bf9b957-hhcgp\" (UID: \"b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2\") " pod="ai-tenant-e2e-shared-a-w4-1a2436/auth-test-model-1a2436-kserve-547bf9b957-hhcgp" Oct 30 18:33:12.607331 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:12.607223 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2-model-cache\") pod \"auth-test-model-1a2436-kserve-547bf9b957-hhcgp\" (UID: \"b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2\") " pod="ai-tenant-e2e-shared-a-w4-1a2436/auth-test-model-1a2436-kserve-547bf9b957-hhcgp" Oct 30 18:33:12.607420 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:12.607313 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2-tmp-dir\") pod \"auth-test-model-1a2436-kserve-547bf9b957-hhcgp\" (UID: \"b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2\") " pod="ai-tenant-e2e-shared-a-w4-1a2436/auth-test-model-1a2436-kserve-547bf9b957-hhcgp" Oct 30 18:33:12.607777 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:12.607750 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2-home\") pod \"auth-test-model-1a2436-kserve-547bf9b957-hhcgp\" (UID: \"b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2\") " pod="ai-tenant-e2e-shared-a-w4-1a2436/auth-test-model-1a2436-kserve-547bf9b957-hhcgp" Oct 30 18:33:12.610370 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:12.610340 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2-dshm\") pod \"auth-test-model-1a2436-kserve-547bf9b957-hhcgp\" (UID: \"b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2\") " pod="ai-tenant-e2e-shared-a-w4-1a2436/auth-test-model-1a2436-kserve-547bf9b957-hhcgp" Oct 30 18:33:12.612719 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:12.612695 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2-tls-certs\") pod \"auth-test-model-1a2436-kserve-547bf9b957-hhcgp\" (UID: \"b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2\") " pod="ai-tenant-e2e-shared-a-w4-1a2436/auth-test-model-1a2436-kserve-547bf9b957-hhcgp" Oct 30 18:33:12.628938 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:12.628898 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvsmx\" (UniqueName: \"kubernetes.io/projected/b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2-kube-api-access-wvsmx\") pod \"auth-test-model-1a2436-kserve-547bf9b957-hhcgp\" (UID: \"b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2\") " pod="ai-tenant-e2e-shared-a-w4-1a2436/auth-test-model-1a2436-kserve-547bf9b957-hhcgp" Oct 30 18:33:12.727252 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:12.725038 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-1a2436/auth-test-model-1a2436-kserve-547bf9b957-hhcgp" Oct 30 18:33:13.592543 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:33:13.592503 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9261f2f_05ba_44bf_a5a2_1b4c9a41b6f2.slice/crio-23466f610518422da53b5ec066462e46823d211721a8848ad577be3ca376686d WatchSource:0}: Error finding container 23466f610518422da53b5ec066462e46823d211721a8848ad577be3ca376686d: Status 404 returned error can't find the container with id 23466f610518422da53b5ec066462e46823d211721a8848ad577be3ca376686d Oct 30 18:33:13.592917 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:13.592628 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-1a2436/auth-test-model-1a2436-kserve-547bf9b957-hhcgp"] Oct 30 18:33:14.286720 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:14.286678 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-1a2436/auth-test-model-1a2436-kserve-547bf9b957-hhcgp" event={"ID":"b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2","Type":"ContainerStarted","Data":"c30529165a0d21b495c05cc7652b928c884c1e9a19ddbb244b0686ad8d710b34"} Oct 30 18:33:14.286917 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:14.286728 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-1a2436/auth-test-model-1a2436-kserve-547bf9b957-hhcgp" event={"ID":"b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2","Type":"ContainerStarted","Data":"23466f610518422da53b5ec066462e46823d211721a8848ad577be3ca376686d"} Oct 30 18:33:14.334161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:14.329710 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-1a2436/auth-test-model-1a2436-kserve-547bf9b957-hhcgp" podStartSLOduration=2.329687778 podStartE2EDuration="2.329687778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:14.326570833 +0000 UTC m=+2091.416633209" watchObservedRunningTime="2026-10-30 18:33:14.329687778 +0000 UTC m=+2091.419750119" Oct 30 18:33:14.727276 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:14.727228 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-1a2436/auth-test-model-1a2436-kserve-547bf9b957-hhcgp" Oct 30 18:33:17.581973 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:33:17.581282 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:33:22.725901 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:22.725861 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-1a2436/auth-test-model-1a2436-kserve-547bf9b957-hhcgp" Oct 30 18:33:22.745688 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:22.745656 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-1a2436/auth-test-model-1a2436-kserve-547bf9b957-hhcgp" Oct 30 18:33:23.382773 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:23.382737 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-1a2436/auth-test-model-1a2436-kserve-547bf9b957-hhcgp" Oct 30 18:33:23.406207 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:23.406169 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": read tcp 10.134.0.2:36602->10.134.0.194:15021: read: connection reset by peer" start-of-body= Oct 30 18:33:23.406354 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:23.406253 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" podUID="6cbde8cc-5ae5-4133-9c52-cb30a81b91b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": read tcp 10.134.0.2:36602->10.134.0.194:15021: read: connection reset by peer" Oct 30 18:33:23.524504 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:23.524480 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52rn6_516c6b6c-fc40-49cf-93b5-1c4471b02cf0/ovn-acl-logging/0.log" Oct 30 18:33:23.528332 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:23.528309 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52rn6_516c6b6c-fc40-49cf-93b5-1c4471b02cf0/ovn-acl-logging/0.log" Oct 30 18:33:24.380380 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:24.380348 4442 generic.go:356] "Generic (PLEG): container finished" podID="6cbde8cc-5ae5-4133-9c52-cb30a81b91b1" containerID="c888cfd579cfb4cbc1e6dfe502cd0317d0e214dfc0a411ed6737b1d2ce023bc1" exitCode=0 Oct 30 18:33:24.380780 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:24.380433 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" event={"ID":"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1","Type":"ContainerDied","Data":"c888cfd579cfb4cbc1e6dfe502cd0317d0e214dfc0a411ed6737b1d2ce023bc1"} Oct 30 18:33:24.380991 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:24.380974 4442 scope.go:122] "RemoveContainer" containerID="c888cfd579cfb4cbc1e6dfe502cd0317d0e214dfc0a411ed6737b1d2ce023bc1" Oct 30 18:33:24.382597 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:24.382567 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:33:24.382696 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:24.382638 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:33:24.382696 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:24.382672 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:33:25.390218 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:25.390183 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" event={"ID":"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1","Type":"ContainerStarted","Data":"3d88c64e63c7aeedd0ed117f78b4cc8caa5d73d92f12d22edd5add46abb0e473"} Oct 30 18:33:25.894777 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:25.894733 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" Oct 30 18:33:25.894996 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:25.894932 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 18:33:25.894996 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:25.894974 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" podUID="6cbde8cc-5ae5-4133-9c52-cb30a81b91b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 18:33:26.895917 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:26.895884 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 18:33:26.896476 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:26.895935 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" podUID="6cbde8cc-5ae5-4133-9c52-cb30a81b91b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 18:33:27.895877 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:27.895838 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 18:33:27.896100 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:27.895904 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" podUID="6cbde8cc-5ae5-4133-9c52-cb30a81b91b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 18:33:28.902156 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:28.896878 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 18:33:28.902156 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:28.896946 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" podUID="6cbde8cc-5ae5-4133-9c52-cb30a81b91b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 18:33:29.895072 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:29.895037 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 18:33:29.895299 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:29.895093 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" podUID="6cbde8cc-5ae5-4133-9c52-cb30a81b91b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 18:33:30.897367 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:30.897117 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 18:33:30.900221 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:30.899953 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" podUID="6cbde8cc-5ae5-4133-9c52-cb30a81b91b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 18:33:31.900646 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:31.900612 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" Oct 30 18:33:31.901147 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:31.901093 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" Oct 30 18:33:31.901958 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:31.901940 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" Oct 30 18:33:40.782184 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:40.782145 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1e0b31-mv98l"] Oct 30 18:33:40.786099 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:40.786075 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1e0b31-mv98l" Oct 30 18:33:40.803076 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:40.803050 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1e0b31-mv98l"] Oct 30 18:33:40.808999 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:40.808966 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:33:40.810051 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:40.810035 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:33:40.866079 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:40.866047 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwgx4\" (UniqueName: \"kubernetes.io/projected/6d1c7003-de2b-420e-a921-b37d7a9d6e4f-kube-api-access-jwgx4\") pod \"maas-api-revoke-keys-e2e-autores-1e0b31-mv98l\" (UID: \"6d1c7003-de2b-420e-a921-b37d7a9d6e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1e0b31-mv98l" Oct 30 18:33:40.866079 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:40.866086 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d1c7003-de2b-420e-a921-b37d7a9d6e4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-1e0b31-mv98l\" (UID: \"6d1c7003-de2b-420e-a921-b37d7a9d6e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1e0b31-mv98l" Oct 30 18:33:40.866410 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:40.866220 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d1c7003-de2b-420e-a921-b37d7a9d6e4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-1e0b31-mv98l\" (UID: \"6d1c7003-de2b-420e-a921-b37d7a9d6e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1e0b31-mv98l" Oct 30 18:33:40.967331 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:40.967294 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwgx4\" (UniqueName: \"kubernetes.io/projected/6d1c7003-de2b-420e-a921-b37d7a9d6e4f-kube-api-access-jwgx4\") pod \"maas-api-revoke-keys-e2e-autores-1e0b31-mv98l\" (UID: \"6d1c7003-de2b-420e-a921-b37d7a9d6e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1e0b31-mv98l" Oct 30 18:33:40.967331 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:40.967333 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d1c7003-de2b-420e-a921-b37d7a9d6e4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-1e0b31-mv98l\" (UID: \"6d1c7003-de2b-420e-a921-b37d7a9d6e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1e0b31-mv98l" Oct 30 18:33:40.967617 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:40.967375 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d1c7003-de2b-420e-a921-b37d7a9d6e4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-1e0b31-mv98l\" (UID: \"6d1c7003-de2b-420e-a921-b37d7a9d6e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1e0b31-mv98l" Oct 30 18:33:40.968069 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:40.968047 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d1c7003-de2b-420e-a921-b37d7a9d6e4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-1e0b31-mv98l\" (UID: \"6d1c7003-de2b-420e-a921-b37d7a9d6e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1e0b31-mv98l" Oct 30 18:33:40.970186 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:40.970158 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d1c7003-de2b-420e-a921-b37d7a9d6e4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-1e0b31-mv98l\" (UID: \"6d1c7003-de2b-420e-a921-b37d7a9d6e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1e0b31-mv98l" Oct 30 18:33:40.989953 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:40.989917 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwgx4\" (UniqueName: \"kubernetes.io/projected/6d1c7003-de2b-420e-a921-b37d7a9d6e4f-kube-api-access-jwgx4\") pod \"maas-api-revoke-keys-e2e-autores-1e0b31-mv98l\" (UID: \"6d1c7003-de2b-420e-a921-b37d7a9d6e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1e0b31-mv98l" Oct 30 18:33:41.100124 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:41.100031 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1e0b31-mv98l" Oct 30 18:33:41.437788 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:33:41.437754 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d1c7003_de2b_420e_a921_b37d7a9d6e4f.slice/crio-a6a2cef705248d4ccd6af92e20892c1d2c730da6c0f42ec2d8813061447b2e41 WatchSource:0}: Error finding container a6a2cef705248d4ccd6af92e20892c1d2c730da6c0f42ec2d8813061447b2e41: Status 404 returned error can't find the container with id a6a2cef705248d4ccd6af92e20892c1d2c730da6c0f42ec2d8813061447b2e41 Oct 30 18:33:41.438500 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:41.438473 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1e0b31-mv98l"] Oct 30 18:33:41.640345 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:41.640313 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1e0b31-mv98l" event={"ID":"6d1c7003-de2b-420e-a921-b37d7a9d6e4f","Type":"ContainerStarted","Data":"a6a2cef705248d4ccd6af92e20892c1d2c730da6c0f42ec2d8813061447b2e41"} Oct 30 18:33:42.651074 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:42.651029 4442 generic.go:356] "Generic (PLEG): container finished" podID="6d1c7003-de2b-420e-a921-b37d7a9d6e4f" containerID="d6adaa530df56a7cc17d81cfc919924e8d1513be9ca58a7877737f9ac9b5b075" exitCode=0 Oct 30 18:33:42.651565 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:42.651089 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1e0b31-mv98l" event={"ID":"6d1c7003-de2b-420e-a921-b37d7a9d6e4f","Type":"ContainerDied","Data":"d6adaa530df56a7cc17d81cfc919924e8d1513be9ca58a7877737f9ac9b5b075"} Oct 30 18:33:43.847380 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:43.847350 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1e0b31-mv98l" Oct 30 18:33:43.927446 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:43.927359 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d1c7003-de2b-420e-a921-b37d7a9d6e4f-kube-api-access-jwgx4\" (UniqueName: \"kubernetes.io/projected/6d1c7003-de2b-420e-a921-b37d7a9d6e4f-kube-api-access-jwgx4\") pod \"6d1c7003-de2b-420e-a921-b37d7a9d6e4f\" (UID: \"6d1c7003-de2b-420e-a921-b37d7a9d6e4f\") " Oct 30 18:33:43.927446 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:43.927441 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d1c7003-de2b-420e-a921-b37d7a9d6e4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d1c7003-de2b-420e-a921-b37d7a9d6e4f-maas-api-xks-ca\") pod \"6d1c7003-de2b-420e-a921-b37d7a9d6e4f\" (UID: \"6d1c7003-de2b-420e-a921-b37d7a9d6e4f\") " Oct 30 18:33:43.927684 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:43.927567 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d1c7003-de2b-420e-a921-b37d7a9d6e4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d1c7003-de2b-420e-a921-b37d7a9d6e4f-maas-api-openshift-ca\") pod \"6d1c7003-de2b-420e-a921-b37d7a9d6e4f\" (UID: \"6d1c7003-de2b-420e-a921-b37d7a9d6e4f\") " Oct 30 18:33:43.928217 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:43.928183 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d1c7003-de2b-420e-a921-b37d7a9d6e4f-maas-api-openshift-ca" pod "6d1c7003-de2b-420e-a921-b37d7a9d6e4f" (UID: "6d1c7003-de2b-420e-a921-b37d7a9d6e4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:33:43.929915 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:43.929891 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d1c7003-de2b-420e-a921-b37d7a9d6e4f-maas-api-xks-ca" pod "6d1c7003-de2b-420e-a921-b37d7a9d6e4f" (UID: "6d1c7003-de2b-420e-a921-b37d7a9d6e4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:33:43.931032 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:43.930998 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d1c7003-de2b-420e-a921-b37d7a9d6e4f-kube-api-access-jwgx4" pod "6d1c7003-de2b-420e-a921-b37d7a9d6e4f" (UID: "6d1c7003-de2b-420e-a921-b37d7a9d6e4f"). InnerVolumeSpecName "kube-api-access-jwgx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:44.030080 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:44.030026 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d1c7003-de2b-420e-a921-b37d7a9d6e4f-maas-api-openshift-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:33:44.030080 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:44.030059 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwgx4\" (UniqueName: \"kubernetes.io/projected/6d1c7003-de2b-420e-a921-b37d7a9d6e4f-kube-api-access-jwgx4\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:33:44.030080 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:44.030069 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d1c7003-de2b-420e-a921-b37d7a9d6e4f-maas-api-xks-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:33:44.674329 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:44.674297 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1e0b31-mv98l" event={"ID":"6d1c7003-de2b-420e-a921-b37d7a9d6e4f","Type":"ContainerDied","Data":"a6a2cef705248d4ccd6af92e20892c1d2c730da6c0f42ec2d8813061447b2e41"} Oct 30 18:33:44.674533 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:44.674345 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6a2cef705248d4ccd6af92e20892c1d2c730da6c0f42ec2d8813061447b2e41" Oct 30 18:33:44.674533 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:44.674343 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1e0b31-mv98l" Oct 30 18:33:45.049357 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:45.049273 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-24b124/auth-test-model-24b124-kserve-745594bcc-6lt65"] Oct 30 18:33:45.050285 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:45.050260 4442 cpu_manager.go:414] "containerMap: removing container" podUID="6d1c7003-de2b-420e-a921-b37d7a9d6e4f" containerName="revoke-keys" Oct 30 18:33:45.050285 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:45.050285 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d1c7003-de2b-420e-a921-b37d7a9d6e4f" containerName="revoke-keys" Oct 30 18:33:45.050481 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:45.050424 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d1c7003-de2b-420e-a921-b37d7a9d6e4f" containerName="revoke-keys" Oct 30 18:33:45.053771 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:45.053747 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-24b124/auth-test-model-24b124-kserve-745594bcc-6lt65" Oct 30 18:33:45.081503 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:45.081471 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f10e975-c54c-4949-a213-40fc7d4ffe1f-tmp-dir\") pod \"auth-test-model-24b124-kserve-745594bcc-6lt65\" (UID: \"8f10e975-c54c-4949-a213-40fc7d4ffe1f\") " pod="ai-tenant-e2e-shared-b-w4-24b124/auth-test-model-24b124-kserve-745594bcc-6lt65" Oct 30 18:33:45.081680 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:45.081551 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dpz4\" (UniqueName: \"kubernetes.io/projected/8f10e975-c54c-4949-a213-40fc7d4ffe1f-kube-api-access-8dpz4\") pod \"auth-test-model-24b124-kserve-745594bcc-6lt65\" (UID: \"8f10e975-c54c-4949-a213-40fc7d4ffe1f\") " pod="ai-tenant-e2e-shared-b-w4-24b124/auth-test-model-24b124-kserve-745594bcc-6lt65" Oct 30 18:33:45.081680 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:45.081663 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f10e975-c54c-4949-a213-40fc7d4ffe1f-home\") pod \"auth-test-model-24b124-kserve-745594bcc-6lt65\" (UID: \"8f10e975-c54c-4949-a213-40fc7d4ffe1f\") " pod="ai-tenant-e2e-shared-b-w4-24b124/auth-test-model-24b124-kserve-745594bcc-6lt65" Oct 30 18:33:45.081814 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:45.081692 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f10e975-c54c-4949-a213-40fc7d4ffe1f-tls-certs\") pod \"auth-test-model-24b124-kserve-745594bcc-6lt65\" (UID: \"8f10e975-c54c-4949-a213-40fc7d4ffe1f\") " pod="ai-tenant-e2e-shared-b-w4-24b124/auth-test-model-24b124-kserve-745594bcc-6lt65" Oct 30 18:33:45.081814 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:45.081754 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f10e975-c54c-4949-a213-40fc7d4ffe1f-dshm\") pod \"auth-test-model-24b124-kserve-745594bcc-6lt65\" (UID: \"8f10e975-c54c-4949-a213-40fc7d4ffe1f\") " pod="ai-tenant-e2e-shared-b-w4-24b124/auth-test-model-24b124-kserve-745594bcc-6lt65" Oct 30 18:33:45.081814 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:45.081786 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f10e975-c54c-4949-a213-40fc7d4ffe1f-model-cache\") pod \"auth-test-model-24b124-kserve-745594bcc-6lt65\" (UID: \"8f10e975-c54c-4949-a213-40fc7d4ffe1f\") " pod="ai-tenant-e2e-shared-b-w4-24b124/auth-test-model-24b124-kserve-745594bcc-6lt65" Oct 30 18:33:45.082295 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:45.082272 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-24b124\"/\"auth-test-model-24b124-kserve-self-signed-certs\"" Oct 30 18:33:45.082418 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:45.082349 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-24b124\"/\"default-dockercfg-chvpr\"" Oct 30 18:33:45.082418 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:45.082409 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-24b124\"/\"openshift-service-ca.crt\"" Oct 30 18:33:45.082621 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:45.082600 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-24b124\"/\"kube-root-ca.crt\"" Oct 30 18:33:45.089595 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:45.089573 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-24b124/auth-test-model-24b124-kserve-745594bcc-6lt65"] Oct 30 18:33:45.183469 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:45.183438 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f10e975-c54c-4949-a213-40fc7d4ffe1f-tmp-dir\") pod \"auth-test-model-24b124-kserve-745594bcc-6lt65\" (UID: \"8f10e975-c54c-4949-a213-40fc7d4ffe1f\") " pod="ai-tenant-e2e-shared-b-w4-24b124/auth-test-model-24b124-kserve-745594bcc-6lt65" Oct 30 18:33:45.183652 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:45.183497 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dpz4\" (UniqueName: \"kubernetes.io/projected/8f10e975-c54c-4949-a213-40fc7d4ffe1f-kube-api-access-8dpz4\") pod \"auth-test-model-24b124-kserve-745594bcc-6lt65\" (UID: \"8f10e975-c54c-4949-a213-40fc7d4ffe1f\") " pod="ai-tenant-e2e-shared-b-w4-24b124/auth-test-model-24b124-kserve-745594bcc-6lt65" Oct 30 18:33:45.183652 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:45.183598 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f10e975-c54c-4949-a213-40fc7d4ffe1f-home\") pod \"auth-test-model-24b124-kserve-745594bcc-6lt65\" (UID: \"8f10e975-c54c-4949-a213-40fc7d4ffe1f\") " pod="ai-tenant-e2e-shared-b-w4-24b124/auth-test-model-24b124-kserve-745594bcc-6lt65" Oct 30 18:33:45.183652 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:45.183623 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f10e975-c54c-4949-a213-40fc7d4ffe1f-tls-certs\") pod \"auth-test-model-24b124-kserve-745594bcc-6lt65\" (UID: \"8f10e975-c54c-4949-a213-40fc7d4ffe1f\") " pod="ai-tenant-e2e-shared-b-w4-24b124/auth-test-model-24b124-kserve-745594bcc-6lt65" Oct 30 18:33:45.183834 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:45.183683 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f10e975-c54c-4949-a213-40fc7d4ffe1f-dshm\") pod \"auth-test-model-24b124-kserve-745594bcc-6lt65\" (UID: \"8f10e975-c54c-4949-a213-40fc7d4ffe1f\") " pod="ai-tenant-e2e-shared-b-w4-24b124/auth-test-model-24b124-kserve-745594bcc-6lt65" Oct 30 18:33:45.183834 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:45.183710 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f10e975-c54c-4949-a213-40fc7d4ffe1f-model-cache\") pod \"auth-test-model-24b124-kserve-745594bcc-6lt65\" (UID: \"8f10e975-c54c-4949-a213-40fc7d4ffe1f\") " pod="ai-tenant-e2e-shared-b-w4-24b124/auth-test-model-24b124-kserve-745594bcc-6lt65" Oct 30 18:33:45.183927 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:45.183830 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f10e975-c54c-4949-a213-40fc7d4ffe1f-tmp-dir\") pod \"auth-test-model-24b124-kserve-745594bcc-6lt65\" (UID: \"8f10e975-c54c-4949-a213-40fc7d4ffe1f\") " pod="ai-tenant-e2e-shared-b-w4-24b124/auth-test-model-24b124-kserve-745594bcc-6lt65" Oct 30 18:33:45.183986 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:45.183968 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f10e975-c54c-4949-a213-40fc7d4ffe1f-home\") pod \"auth-test-model-24b124-kserve-745594bcc-6lt65\" (UID: \"8f10e975-c54c-4949-a213-40fc7d4ffe1f\") " pod="ai-tenant-e2e-shared-b-w4-24b124/auth-test-model-24b124-kserve-745594bcc-6lt65" Oct 30 18:33:45.184214 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:45.184191 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f10e975-c54c-4949-a213-40fc7d4ffe1f-model-cache\") pod \"auth-test-model-24b124-kserve-745594bcc-6lt65\" (UID: \"8f10e975-c54c-4949-a213-40fc7d4ffe1f\") " pod="ai-tenant-e2e-shared-b-w4-24b124/auth-test-model-24b124-kserve-745594bcc-6lt65" Oct 30 18:33:45.186122 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:45.186096 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f10e975-c54c-4949-a213-40fc7d4ffe1f-dshm\") pod \"auth-test-model-24b124-kserve-745594bcc-6lt65\" (UID: \"8f10e975-c54c-4949-a213-40fc7d4ffe1f\") " pod="ai-tenant-e2e-shared-b-w4-24b124/auth-test-model-24b124-kserve-745594bcc-6lt65" Oct 30 18:33:45.186384 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:45.186368 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f10e975-c54c-4949-a213-40fc7d4ffe1f-tls-certs\") pod \"auth-test-model-24b124-kserve-745594bcc-6lt65\" (UID: \"8f10e975-c54c-4949-a213-40fc7d4ffe1f\") " pod="ai-tenant-e2e-shared-b-w4-24b124/auth-test-model-24b124-kserve-745594bcc-6lt65" Oct 30 18:33:45.203274 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:45.203248 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dpz4\" (UniqueName: \"kubernetes.io/projected/8f10e975-c54c-4949-a213-40fc7d4ffe1f-kube-api-access-8dpz4\") pod \"auth-test-model-24b124-kserve-745594bcc-6lt65\" (UID: \"8f10e975-c54c-4949-a213-40fc7d4ffe1f\") " pod="ai-tenant-e2e-shared-b-w4-24b124/auth-test-model-24b124-kserve-745594bcc-6lt65" Oct 30 18:33:45.399874 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:45.399531 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-24b124/auth-test-model-24b124-kserve-745594bcc-6lt65" Oct 30 18:33:45.849236 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:45.849201 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-24b124/auth-test-model-24b124-kserve-745594bcc-6lt65"] Oct 30 18:33:45.849677 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:33:45.849654 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f10e975_c54c_4949_a213_40fc7d4ffe1f.slice/crio-4496126b3c8874fcf0222fdbe6a3223fdbb48234001a1233733e7c91884227a5 WatchSource:0}: Error finding container 4496126b3c8874fcf0222fdbe6a3223fdbb48234001a1233733e7c91884227a5: Status 404 returned error can't find the container with id 4496126b3c8874fcf0222fdbe6a3223fdbb48234001a1233733e7c91884227a5 Oct 30 18:33:46.693099 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:46.693066 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-24b124/auth-test-model-24b124-kserve-745594bcc-6lt65" event={"ID":"8f10e975-c54c-4949-a213-40fc7d4ffe1f","Type":"ContainerStarted","Data":"ebdf55d0953228961c3e84c63e5b1ec5984a59228e5b8eb99f3300d90686753d"} Oct 30 18:33:46.693099 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:46.693110 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-24b124/auth-test-model-24b124-kserve-745594bcc-6lt65" event={"ID":"8f10e975-c54c-4949-a213-40fc7d4ffe1f","Type":"ContainerStarted","Data":"4496126b3c8874fcf0222fdbe6a3223fdbb48234001a1233733e7c91884227a5"} Oct 30 18:33:46.736304 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:46.736245 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-24b124/auth-test-model-24b124-kserve-745594bcc-6lt65" podStartSLOduration=1.736225964 podStartE2EDuration="1.736225964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:46.731668335 +0000 UTC m=+2123.821730698" watchObservedRunningTime="2026-10-30 18:33:46.736225964 +0000 UTC m=+2123.826288304" Oct 30 18:33:47.400354 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:47.400323 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-24b124/auth-test-model-24b124-kserve-745594bcc-6lt65" Oct 30 18:33:47.567068 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:33:47.567032 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:33:55.401320 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:55.401270 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-24b124/auth-test-model-24b124-kserve-745594bcc-6lt65" Oct 30 18:33:55.414420 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:55.414398 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-24b124/auth-test-model-24b124-kserve-745594bcc-6lt65" Oct 30 18:33:55.802213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:33:55.802106 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-24b124/auth-test-model-24b124-kserve-745594bcc-6lt65" Oct 30 18:34:08.024879 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:08.024841 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1e0b31-mv98l"] Oct 30 18:34:08.035814 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:08.035786 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1e0b31-mv98l"] Oct 30 18:34:09.068024 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:09.067994 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d1c7003-de2b-420e-a921-b37d7a9d6e4f" path="/var/lib/kubelet/pods/6d1c7003-de2b-420e-a921-b37d7a9d6e4f/volumes" Oct 30 18:34:14.941599 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:14.941546 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst"] Oct 30 18:34:15.003919 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:15.003874 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" podUID="bd132ea6-9517-4402-b079-16d0af22572f" containerName="istio-proxy" containerID="cri-o://0e69866c5a7d3591fbbbf034fca1143075ac005f79bf3ee910011984d72242a1" gracePeriod=30 Oct 30 18:34:17.562523 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:34:17.562474 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:34:18.498616 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:18.498581 4442 patch_prober.go:29] interesting pod/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:34:18.498804 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:18.498651 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" podUID="bd132ea6-9517-4402-b079-16d0af22572f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:34:20.311369 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:20.311341 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" Oct 30 18:34:20.391351 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:20.391321 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd132ea6-9517-4402-b079-16d0af22572f-istio-token\" (UniqueName: \"kubernetes.io/projected/bd132ea6-9517-4402-b079-16d0af22572f-istio-token\") pod \"bd132ea6-9517-4402-b079-16d0af22572f\" (UID: \"bd132ea6-9517-4402-b079-16d0af22572f\") " Oct 30 18:34:20.391351 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:20.391362 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd132ea6-9517-4402-b079-16d0af22572f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd132ea6-9517-4402-b079-16d0af22572f-istiod-ca-cert\") pod \"bd132ea6-9517-4402-b079-16d0af22572f\" (UID: \"bd132ea6-9517-4402-b079-16d0af22572f\") " Oct 30 18:34:20.391627 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:20.391428 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd132ea6-9517-4402-b079-16d0af22572f-kube-api-access-cm9dw\" (UniqueName: \"kubernetes.io/projected/bd132ea6-9517-4402-b079-16d0af22572f-kube-api-access-cm9dw\") pod \"bd132ea6-9517-4402-b079-16d0af22572f\" (UID: \"bd132ea6-9517-4402-b079-16d0af22572f\") " Oct 30 18:34:20.391627 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:20.391479 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd132ea6-9517-4402-b079-16d0af22572f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd132ea6-9517-4402-b079-16d0af22572f-credential-socket\") pod \"bd132ea6-9517-4402-b079-16d0af22572f\" (UID: \"bd132ea6-9517-4402-b079-16d0af22572f\") " Oct 30 18:34:20.391627 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:20.391533 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd132ea6-9517-4402-b079-16d0af22572f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd132ea6-9517-4402-b079-16d0af22572f-istio-envoy\") pod \"bd132ea6-9517-4402-b079-16d0af22572f\" (UID: \"bd132ea6-9517-4402-b079-16d0af22572f\") " Oct 30 18:34:20.391783 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:20.391624 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd132ea6-9517-4402-b079-16d0af22572f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd132ea6-9517-4402-b079-16d0af22572f-workload-socket\") pod \"bd132ea6-9517-4402-b079-16d0af22572f\" (UID: \"bd132ea6-9517-4402-b079-16d0af22572f\") " Oct 30 18:34:20.391783 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:20.391757 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd132ea6-9517-4402-b079-16d0af22572f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd132ea6-9517-4402-b079-16d0af22572f-istio-podinfo\") pod \"bd132ea6-9517-4402-b079-16d0af22572f\" (UID: \"bd132ea6-9517-4402-b079-16d0af22572f\") " Oct 30 18:34:20.391894 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:20.391862 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd132ea6-9517-4402-b079-16d0af22572f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd132ea6-9517-4402-b079-16d0af22572f-istio-data\") pod \"bd132ea6-9517-4402-b079-16d0af22572f\" (UID: \"bd132ea6-9517-4402-b079-16d0af22572f\") " Oct 30 18:34:20.391894 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:20.391858 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd132ea6-9517-4402-b079-16d0af22572f-credential-socket" pod "bd132ea6-9517-4402-b079-16d0af22572f" (UID: "bd132ea6-9517-4402-b079-16d0af22572f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:20.391894 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:20.391871 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd132ea6-9517-4402-b079-16d0af22572f-istiod-ca-cert" pod "bd132ea6-9517-4402-b079-16d0af22572f" (UID: "bd132ea6-9517-4402-b079-16d0af22572f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:34:20.392334 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:20.392110 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd132ea6-9517-4402-b079-16d0af22572f-workload-socket" pod "bd132ea6-9517-4402-b079-16d0af22572f" (UID: "bd132ea6-9517-4402-b079-16d0af22572f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:20.392493 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:20.392370 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd132ea6-9517-4402-b079-16d0af22572f-istio-data" pod "bd132ea6-9517-4402-b079-16d0af22572f" (UID: "bd132ea6-9517-4402-b079-16d0af22572f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:20.392906 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:20.392876 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd132ea6-9517-4402-b079-16d0af22572f-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:34:20.393023 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:20.392908 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd132ea6-9517-4402-b079-16d0af22572f-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:34:20.393023 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:20.392922 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd132ea6-9517-4402-b079-16d0af22572f-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:34:20.393023 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:20.392936 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd132ea6-9517-4402-b079-16d0af22572f-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:34:20.398741 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:20.398615 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd132ea6-9517-4402-b079-16d0af22572f-istio-envoy" pod "bd132ea6-9517-4402-b079-16d0af22572f" (UID: "bd132ea6-9517-4402-b079-16d0af22572f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:20.398741 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:20.398667 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd132ea6-9517-4402-b079-16d0af22572f-istio-token" pod "bd132ea6-9517-4402-b079-16d0af22572f" (UID: "bd132ea6-9517-4402-b079-16d0af22572f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:34:20.398741 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:20.398703 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd132ea6-9517-4402-b079-16d0af22572f-istio-podinfo" pod "bd132ea6-9517-4402-b079-16d0af22572f" (UID: "bd132ea6-9517-4402-b079-16d0af22572f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:34:20.398975 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:20.398911 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd132ea6-9517-4402-b079-16d0af22572f-kube-api-access-cm9dw" pod "bd132ea6-9517-4402-b079-16d0af22572f" (UID: "bd132ea6-9517-4402-b079-16d0af22572f"). InnerVolumeSpecName "kube-api-access-cm9dw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:34:20.493715 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:20.493682 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd132ea6-9517-4402-b079-16d0af22572f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd132ea6-9517-4402-b079-16d0af22572f-workload-certs\") pod \"bd132ea6-9517-4402-b079-16d0af22572f\" (UID: \"bd132ea6-9517-4402-b079-16d0af22572f\") " Oct 30 18:34:20.493941 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:20.493917 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd132ea6-9517-4402-b079-16d0af22572f-workload-certs" pod "bd132ea6-9517-4402-b079-16d0af22572f" (UID: "bd132ea6-9517-4402-b079-16d0af22572f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:20.494451 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:20.494429 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd132ea6-9517-4402-b079-16d0af22572f-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:34:20.494547 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:20.494453 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd132ea6-9517-4402-b079-16d0af22572f-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:34:20.494547 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:20.494468 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd132ea6-9517-4402-b079-16d0af22572f-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:34:20.494547 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:20.494483 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd132ea6-9517-4402-b079-16d0af22572f-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:34:20.494547 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:20.494496 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm9dw\" (UniqueName: \"kubernetes.io/projected/bd132ea6-9517-4402-b079-16d0af22572f-kube-api-access-cm9dw\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:34:21.117812 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:21.117775 4442 generic.go:356] "Generic (PLEG): container finished" podID="bd132ea6-9517-4402-b079-16d0af22572f" containerID="0e69866c5a7d3591fbbbf034fca1143075ac005f79bf3ee910011984d72242a1" exitCode=0 Oct 30 18:34:21.118019 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:21.117854 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" Oct 30 18:34:21.118019 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:21.117871 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" event={"ID":"bd132ea6-9517-4402-b079-16d0af22572f","Type":"ContainerDied","Data":"0e69866c5a7d3591fbbbf034fca1143075ac005f79bf3ee910011984d72242a1"} Oct 30 18:34:21.118019 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:21.117929 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst" event={"ID":"bd132ea6-9517-4402-b079-16d0af22572f","Type":"ContainerDied","Data":"d3ac499d97246c3bb73b468a5a21ca01377b8e8d9514c15619caf981d3cb92c6"} Oct 30 18:34:21.118019 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:21.117953 4442 scope.go:122] "RemoveContainer" containerID="0e69866c5a7d3591fbbbf034fca1143075ac005f79bf3ee910011984d72242a1" Oct 30 18:34:21.130206 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:21.130081 4442 scope.go:122] "RemoveContainer" containerID="07a5613dbd40355550531d62d61a6600642eec76b158886aadb432e9d75a5045" Oct 30 18:34:21.141056 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:21.141037 4442 scope.go:122] "RemoveContainer" containerID="0e69866c5a7d3591fbbbf034fca1143075ac005f79bf3ee910011984d72242a1" Oct 30 18:34:21.141376 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:34:21.141355 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e69866c5a7d3591fbbbf034fca1143075ac005f79bf3ee910011984d72242a1\": container with ID starting with 0e69866c5a7d3591fbbbf034fca1143075ac005f79bf3ee910011984d72242a1 not found: ID does not exist" containerID="0e69866c5a7d3591fbbbf034fca1143075ac005f79bf3ee910011984d72242a1" Oct 30 18:34:21.141438 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:21.141386 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e69866c5a7d3591fbbbf034fca1143075ac005f79bf3ee910011984d72242a1"} err="failed to get container status \"0e69866c5a7d3591fbbbf034fca1143075ac005f79bf3ee910011984d72242a1\": rpc error: code = NotFound desc = could not find container \"0e69866c5a7d3591fbbbf034fca1143075ac005f79bf3ee910011984d72242a1\": container with ID starting with 0e69866c5a7d3591fbbbf034fca1143075ac005f79bf3ee910011984d72242a1 not found: ID does not exist" Oct 30 18:34:21.141438 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:21.141402 4442 scope.go:122] "RemoveContainer" containerID="07a5613dbd40355550531d62d61a6600642eec76b158886aadb432e9d75a5045" Oct 30 18:34:21.141675 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:34:21.141651 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07a5613dbd40355550531d62d61a6600642eec76b158886aadb432e9d75a5045\": container with ID starting with 07a5613dbd40355550531d62d61a6600642eec76b158886aadb432e9d75a5045 not found: ID does not exist" containerID="07a5613dbd40355550531d62d61a6600642eec76b158886aadb432e9d75a5045" Oct 30 18:34:21.141737 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:21.141685 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07a5613dbd40355550531d62d61a6600642eec76b158886aadb432e9d75a5045"} err="failed to get container status \"07a5613dbd40355550531d62d61a6600642eec76b158886aadb432e9d75a5045\": rpc error: code = NotFound desc = could not find container \"07a5613dbd40355550531d62d61a6600642eec76b158886aadb432e9d75a5045\": container with ID starting with 07a5613dbd40355550531d62d61a6600642eec76b158886aadb432e9d75a5045 not found: ID does not exist" Oct 30 18:34:21.184463 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:21.184435 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst"] Oct 30 18:34:21.188731 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:21.188705 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-1e0b31-openshift-default-5b75cf549f-q4wst"] Oct 30 18:34:23.069655 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:23.069623 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd132ea6-9517-4402-b079-16d0af22572f" path="/var/lib/kubelet/pods/bd132ea6-9517-4402-b079-16d0af22572f/volumes" Oct 30 18:34:26.019504 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:26.019462 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77"] Oct 30 18:34:26.020662 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:26.020637 4442 cpu_manager.go:414] "containerMap: removing container" podUID="bd132ea6-9517-4402-b079-16d0af22572f" containerName="istio-proxy" Oct 30 18:34:26.020804 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:26.020663 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd132ea6-9517-4402-b079-16d0af22572f" containerName="istio-proxy" Oct 30 18:34:26.020804 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:26.020687 4442 cpu_manager.go:414] "containerMap: removing container" podUID="bd132ea6-9517-4402-b079-16d0af22572f" containerName="istio-proxy" Oct 30 18:34:26.020804 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:26.020695 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd132ea6-9517-4402-b079-16d0af22572f" containerName="istio-proxy" Oct 30 18:34:26.020924 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:26.020883 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd132ea6-9517-4402-b079-16d0af22572f" containerName="istio-proxy" Oct 30 18:34:26.020924 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:26.020902 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd132ea6-9517-4402-b079-16d0af22572f" containerName="istio-proxy" Oct 30 18:34:26.025343 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:26.025317 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" Oct 30 18:34:26.045501 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:26.045466 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77"] Oct 30 18:34:26.051097 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:26.051073 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-160bf8-openshift-default-dockercfg-9zvw9\"" Oct 30 18:34:26.131165 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:26.131099 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bbe6b88-960d-47db-a270-47fc4cee7f44-istio-token\") pod \"e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77\" (UID: \"6bbe6b88-960d-47db-a270-47fc4cee7f44\") " pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" Oct 30 18:34:26.132281 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:26.131233 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bbe6b88-960d-47db-a270-47fc4cee7f44-istio-data\") pod \"e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77\" (UID: \"6bbe6b88-960d-47db-a270-47fc4cee7f44\") " pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" Oct 30 18:34:26.132281 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:26.131314 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbe6b88-960d-47db-a270-47fc4cee7f44-credential-socket\") pod \"e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77\" (UID: \"6bbe6b88-960d-47db-a270-47fc4cee7f44\") " pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" Oct 30 18:34:26.132281 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:26.131875 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bbe6b88-960d-47db-a270-47fc4cee7f44-istiod-ca-cert\") pod \"e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77\" (UID: \"6bbe6b88-960d-47db-a270-47fc4cee7f44\") " pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" Oct 30 18:34:26.132281 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:26.131914 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b574v\" (UniqueName: \"kubernetes.io/projected/6bbe6b88-960d-47db-a270-47fc4cee7f44-kube-api-access-b574v\") pod \"e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77\" (UID: \"6bbe6b88-960d-47db-a270-47fc4cee7f44\") " pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" Oct 30 18:34:26.132281 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:26.131992 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbe6b88-960d-47db-a270-47fc4cee7f44-workload-socket\") pod \"e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77\" (UID: \"6bbe6b88-960d-47db-a270-47fc4cee7f44\") " pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" Oct 30 18:34:26.132281 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:26.132116 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bbe6b88-960d-47db-a270-47fc4cee7f44-istio-envoy\") pod \"e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77\" (UID: \"6bbe6b88-960d-47db-a270-47fc4cee7f44\") " pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" Oct 30 18:34:26.132281 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:26.132205 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bbe6b88-960d-47db-a270-47fc4cee7f44-istio-podinfo\") pod \"e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77\" (UID: \"6bbe6b88-960d-47db-a270-47fc4cee7f44\") " pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" Oct 30 18:34:26.132585 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:26.132335 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bbe6b88-960d-47db-a270-47fc4cee7f44-workload-certs\") pod \"e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77\" (UID: \"6bbe6b88-960d-47db-a270-47fc4cee7f44\") " pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" Oct 30 18:34:26.233711 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:26.233673 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbe6b88-960d-47db-a270-47fc4cee7f44-credential-socket\") pod \"e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77\" (UID: \"6bbe6b88-960d-47db-a270-47fc4cee7f44\") " pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" Oct 30 18:34:26.233887 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:26.233722 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bbe6b88-960d-47db-a270-47fc4cee7f44-istiod-ca-cert\") pod \"e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77\" (UID: \"6bbe6b88-960d-47db-a270-47fc4cee7f44\") " pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" Oct 30 18:34:26.233887 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:26.233738 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b574v\" (UniqueName: \"kubernetes.io/projected/6bbe6b88-960d-47db-a270-47fc4cee7f44-kube-api-access-b574v\") pod \"e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77\" (UID: \"6bbe6b88-960d-47db-a270-47fc4cee7f44\") " pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" Oct 30 18:34:26.233887 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:26.233785 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbe6b88-960d-47db-a270-47fc4cee7f44-workload-socket\") pod \"e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77\" (UID: \"6bbe6b88-960d-47db-a270-47fc4cee7f44\") " pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" Oct 30 18:34:26.233887 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:26.233881 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bbe6b88-960d-47db-a270-47fc4cee7f44-istio-envoy\") pod \"e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77\" (UID: \"6bbe6b88-960d-47db-a270-47fc4cee7f44\") " pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" Oct 30 18:34:26.234118 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:26.233915 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bbe6b88-960d-47db-a270-47fc4cee7f44-istio-podinfo\") pod \"e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77\" (UID: \"6bbe6b88-960d-47db-a270-47fc4cee7f44\") " pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" Oct 30 18:34:26.234118 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:26.234031 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bbe6b88-960d-47db-a270-47fc4cee7f44-workload-certs\") pod \"e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77\" (UID: \"6bbe6b88-960d-47db-a270-47fc4cee7f44\") " pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" Oct 30 18:34:26.234308 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:26.234179 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bbe6b88-960d-47db-a270-47fc4cee7f44-istio-token\") pod \"e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77\" (UID: \"6bbe6b88-960d-47db-a270-47fc4cee7f44\") " pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" Oct 30 18:34:26.234308 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:26.234204 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbe6b88-960d-47db-a270-47fc4cee7f44-credential-socket\") pod \"e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77\" (UID: \"6bbe6b88-960d-47db-a270-47fc4cee7f44\") " pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" Oct 30 18:34:26.234308 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:26.234237 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bbe6b88-960d-47db-a270-47fc4cee7f44-istio-data\") pod \"e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77\" (UID: \"6bbe6b88-960d-47db-a270-47fc4cee7f44\") " pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" Oct 30 18:34:26.234308 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:26.234298 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbe6b88-960d-47db-a270-47fc4cee7f44-workload-socket\") pod \"e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77\" (UID: \"6bbe6b88-960d-47db-a270-47fc4cee7f44\") " pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" Oct 30 18:34:26.234562 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:26.234537 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bbe6b88-960d-47db-a270-47fc4cee7f44-istiod-ca-cert\") pod \"e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77\" (UID: \"6bbe6b88-960d-47db-a270-47fc4cee7f44\") " pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" Oct 30 18:34:26.234922 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:26.234896 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bbe6b88-960d-47db-a270-47fc4cee7f44-workload-certs\") pod \"e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77\" (UID: \"6bbe6b88-960d-47db-a270-47fc4cee7f44\") " pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" Oct 30 18:34:26.235109 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:26.235067 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bbe6b88-960d-47db-a270-47fc4cee7f44-istio-data\") pod \"e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77\" (UID: \"6bbe6b88-960d-47db-a270-47fc4cee7f44\") " pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" Oct 30 18:34:26.236729 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:26.236681 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bbe6b88-960d-47db-a270-47fc4cee7f44-istio-envoy\") pod \"e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77\" (UID: \"6bbe6b88-960d-47db-a270-47fc4cee7f44\") " pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" Oct 30 18:34:26.237427 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:26.237401 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bbe6b88-960d-47db-a270-47fc4cee7f44-istio-podinfo\") pod \"e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77\" (UID: \"6bbe6b88-960d-47db-a270-47fc4cee7f44\") " pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" Oct 30 18:34:26.377226 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:26.370508 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b574v\" (UniqueName: \"kubernetes.io/projected/6bbe6b88-960d-47db-a270-47fc4cee7f44-kube-api-access-b574v\") pod \"e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77\" (UID: \"6bbe6b88-960d-47db-a270-47fc4cee7f44\") " pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" Oct 30 18:34:26.379236 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:26.379204 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bbe6b88-960d-47db-a270-47fc4cee7f44-istio-token\") pod \"e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77\" (UID: \"6bbe6b88-960d-47db-a270-47fc4cee7f44\") " pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" Oct 30 18:34:26.647530 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:26.647496 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" Oct 30 18:34:26.963561 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:34:26.963532 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bbe6b88_960d_47db_a270_47fc4cee7f44.slice/crio-069cd011d9a91474df255dacbedce04b35fb470d1d66853bdea518e2126634ed WatchSource:0}: Error finding container 069cd011d9a91474df255dacbedce04b35fb470d1d66853bdea518e2126634ed: Status 404 returned error can't find the container with id 069cd011d9a91474df255dacbedce04b35fb470d1d66853bdea518e2126634ed Oct 30 18:34:26.963814 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:26.963788 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77"] Oct 30 18:34:26.966635 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:26.966605 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:34:26.966723 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:26.966668 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:34:26.966723 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:26.966698 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:34:27.178675 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:27.178640 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" event={"ID":"6bbe6b88-960d-47db-a270-47fc4cee7f44","Type":"ContainerStarted","Data":"43183b0e889c19bb57010aa96bdb89000fea7f09d635fce2c57165a4a0ce5a53"} Oct 30 18:34:27.179104 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:27.178703 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" event={"ID":"6bbe6b88-960d-47db-a270-47fc4cee7f44","Type":"ContainerStarted","Data":"069cd011d9a91474df255dacbedce04b35fb470d1d66853bdea518e2126634ed"} Oct 30 18:34:27.220709 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:27.220593 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" podStartSLOduration=2.220577844 podStartE2EDuration="2.220577844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:34:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:34:27.217710969 +0000 UTC m=+2164.307773310" watchObservedRunningTime="2026-10-30 18:34:27.220577844 +0000 UTC m=+2164.310640182" Oct 30 18:34:27.648049 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:27.648012 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" Oct 30 18:34:28.655042 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:28.655011 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" Oct 30 18:34:29.195934 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:29.195897 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" Oct 30 18:34:29.196917 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:29.196895 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" Oct 30 18:34:31.540601 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:31.540558 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-160bf8-857d864c68-p4svr"] Oct 30 18:34:31.545298 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:31.545226 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-160bf8-857d864c68-p4svr" Oct 30 18:34:31.565825 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:31.565796 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-160bf8-857d864c68-p4svr"] Oct 30 18:34:31.571353 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:31.570462 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-160bf8\"" Oct 30 18:34:31.605061 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:31.605027 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-160bf8-74f657fbf9-kfw2h"] Oct 30 18:34:31.646175 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:31.646121 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnsq9\" (UniqueName: \"kubernetes.io/projected/355f5946-71a8-4a78-937c-62d47ebf9e3f-kube-api-access-nnsq9\") pod \"payload-processing-e2e-explicit-160bf8-857d864c68-p4svr\" (UID: \"355f5946-71a8-4a78-937c-62d47ebf9e3f\") " pod="openshift-ingress/payload-processing-e2e-explicit-160bf8-857d864c68-p4svr" Oct 30 18:34:31.646727 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:31.646673 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/355f5946-71a8-4a78-937c-62d47ebf9e3f-plugins-config-volume\") pod \"payload-processing-e2e-explicit-160bf8-857d864c68-p4svr\" (UID: \"355f5946-71a8-4a78-937c-62d47ebf9e3f\") " pod="openshift-ingress/payload-processing-e2e-explicit-160bf8-857d864c68-p4svr" Oct 30 18:34:31.660443 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:31.653378 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-160bf8-74f657fbf9-kfw2h"] Oct 30 18:34:31.660443 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:31.653531 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-160bf8-74f657fbf9-kfw2h" Oct 30 18:34:31.781169 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:31.777989 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnsq9\" (UniqueName: \"kubernetes.io/projected/355f5946-71a8-4a78-937c-62d47ebf9e3f-kube-api-access-nnsq9\") pod \"payload-processing-e2e-explicit-160bf8-857d864c68-p4svr\" (UID: \"355f5946-71a8-4a78-937c-62d47ebf9e3f\") " pod="openshift-ingress/payload-processing-e2e-explicit-160bf8-857d864c68-p4svr" Oct 30 18:34:31.781169 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:31.778155 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64g7r\" (UniqueName: \"kubernetes.io/projected/84e058c7-2d92-46d5-bcf7-4a061945921e-kube-api-access-64g7r\") pod \"payload-pre-processing-e2e-explicit-160bf8-74f657fbf9-kfw2h\" (UID: \"84e058c7-2d92-46d5-bcf7-4a061945921e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-160bf8-74f657fbf9-kfw2h" Oct 30 18:34:31.781169 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:31.778327 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/355f5946-71a8-4a78-937c-62d47ebf9e3f-plugins-config-volume\") pod \"payload-processing-e2e-explicit-160bf8-857d864c68-p4svr\" (UID: \"355f5946-71a8-4a78-937c-62d47ebf9e3f\") " pod="openshift-ingress/payload-processing-e2e-explicit-160bf8-857d864c68-p4svr" Oct 30 18:34:31.781169 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:31.778378 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e058c7-2d92-46d5-bcf7-4a061945921e-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-160bf8-74f657fbf9-kfw2h\" (UID: \"84e058c7-2d92-46d5-bcf7-4a061945921e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-160bf8-74f657fbf9-kfw2h" Oct 30 18:34:31.781169 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:31.779769 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/355f5946-71a8-4a78-937c-62d47ebf9e3f-plugins-config-volume\") pod \"payload-processing-e2e-explicit-160bf8-857d864c68-p4svr\" (UID: \"355f5946-71a8-4a78-937c-62d47ebf9e3f\") " pod="openshift-ingress/payload-processing-e2e-explicit-160bf8-857d864c68-p4svr" Oct 30 18:34:31.926863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:31.885249 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-160bf8-dockercfg-mdpz6\"" Oct 30 18:34:31.961860 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:31.909878 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e058c7-2d92-46d5-bcf7-4a061945921e-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-160bf8-74f657fbf9-kfw2h\" (UID: \"84e058c7-2d92-46d5-bcf7-4a061945921e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-160bf8-74f657fbf9-kfw2h" Oct 30 18:34:31.962729 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:31.962683 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64g7r\" (UniqueName: \"kubernetes.io/projected/84e058c7-2d92-46d5-bcf7-4a061945921e-kube-api-access-64g7r\") pod \"payload-pre-processing-e2e-explicit-160bf8-74f657fbf9-kfw2h\" (UID: \"84e058c7-2d92-46d5-bcf7-4a061945921e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-160bf8-74f657fbf9-kfw2h" Oct 30 18:34:31.963464 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:31.910814 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e058c7-2d92-46d5-bcf7-4a061945921e-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-160bf8-74f657fbf9-kfw2h\" (UID: \"84e058c7-2d92-46d5-bcf7-4a061945921e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-160bf8-74f657fbf9-kfw2h" Oct 30 18:34:31.999572 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:31.996242 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64g7r\" (UniqueName: \"kubernetes.io/projected/84e058c7-2d92-46d5-bcf7-4a061945921e-kube-api-access-64g7r\") pod \"payload-pre-processing-e2e-explicit-160bf8-74f657fbf9-kfw2h\" (UID: \"84e058c7-2d92-46d5-bcf7-4a061945921e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-160bf8-74f657fbf9-kfw2h" Oct 30 18:34:32.001484 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:32.001458 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnsq9\" (UniqueName: \"kubernetes.io/projected/355f5946-71a8-4a78-937c-62d47ebf9e3f-kube-api-access-nnsq9\") pod \"payload-processing-e2e-explicit-160bf8-857d864c68-p4svr\" (UID: \"355f5946-71a8-4a78-937c-62d47ebf9e3f\") " pod="openshift-ingress/payload-processing-e2e-explicit-160bf8-857d864c68-p4svr" Oct 30 18:34:32.168120 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:32.168092 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-160bf8-857d864c68-p4svr" Oct 30 18:34:32.195319 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:32.195289 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-160bf8-74f657fbf9-kfw2h" Oct 30 18:34:32.493436 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:32.493360 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-160bf8-857d864c68-p4svr"] Oct 30 18:34:32.501453 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:34:32.501426 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod355f5946_71a8_4a78_937c_62d47ebf9e3f.slice/crio-6c2fba6c7d8f6c2c9033f66049a9aa0057d5237f99ce7c243491839427be9923 WatchSource:0}: Error finding container 6c2fba6c7d8f6c2c9033f66049a9aa0057d5237f99ce7c243491839427be9923: Status 404 returned error can't find the container with id 6c2fba6c7d8f6c2c9033f66049a9aa0057d5237f99ce7c243491839427be9923 Oct 30 18:34:32.550259 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:34:32.550231 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84e058c7_2d92_46d5_bcf7_4a061945921e.slice/crio-3114191e4a6f46ab0cc474107f878c6d8e37386f196726530427ebb90a564b0a WatchSource:0}: Error finding container 3114191e4a6f46ab0cc474107f878c6d8e37386f196726530427ebb90a564b0a: Status 404 returned error can't find the container with id 3114191e4a6f46ab0cc474107f878c6d8e37386f196726530427ebb90a564b0a Oct 30 18:34:32.550661 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:32.550436 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-160bf8-74f657fbf9-kfw2h"] Oct 30 18:34:33.365998 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:33.365960 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-160bf8-857d864c68-p4svr" event={"ID":"355f5946-71a8-4a78-937c-62d47ebf9e3f","Type":"ContainerStarted","Data":"599298097cdd56fde3ae185ebd69a388a31959323f99ee48c333390aa21d6a23"} Oct 30 18:34:33.366342 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:33.366314 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-160bf8-857d864c68-p4svr" event={"ID":"355f5946-71a8-4a78-937c-62d47ebf9e3f","Type":"ContainerStarted","Data":"6c2fba6c7d8f6c2c9033f66049a9aa0057d5237f99ce7c243491839427be9923"} Oct 30 18:34:33.367485 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:33.367464 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-160bf8-857d864c68-p4svr" Oct 30 18:34:33.369876 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:33.369852 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-160bf8-74f657fbf9-kfw2h" event={"ID":"84e058c7-2d92-46d5-bcf7-4a061945921e","Type":"ContainerStarted","Data":"092f5e5ef486f99032aa2595925838ecafd7fa127dd8c05b4eeb1949652571a2"} Oct 30 18:34:33.370060 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:33.370045 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-160bf8-74f657fbf9-kfw2h" Oct 30 18:34:33.370209 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:33.370194 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-160bf8-74f657fbf9-kfw2h" event={"ID":"84e058c7-2d92-46d5-bcf7-4a061945921e","Type":"ContainerStarted","Data":"3114191e4a6f46ab0cc474107f878c6d8e37386f196726530427ebb90a564b0a"} Oct 30 18:34:33.408574 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:33.408517 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-160bf8-857d864c68-p4svr" podStartSLOduration=2.408502619 podStartE2EDuration="2.408502619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:34:33.406030631 +0000 UTC m=+2170.496092967" watchObservedRunningTime="2026-10-30 18:34:33.408502619 +0000 UTC m=+2170.498564958" Oct 30 18:34:33.447657 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:33.447603 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-160bf8-74f657fbf9-kfw2h" podStartSLOduration=2.447586866 podStartE2EDuration="2.447586866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:34:33.445664949 +0000 UTC m=+2170.535727288" watchObservedRunningTime="2026-10-30 18:34:33.447586866 +0000 UTC m=+2170.537649205" Oct 30 18:34:44.383592 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:44.383557 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-160bf8-74f657fbf9-kfw2h" Oct 30 18:34:45.392455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:45.392425 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-160bf8-857d864c68-p4svr" Oct 30 18:34:47.232734 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:34:47.232684 4442 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-176.ec2.internal" podUID="9e81bd78300430e51b01d30caa1123e7" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:34:47.568320 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:34:47.568225 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:35:12.909799 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:12.909713 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll"] Oct 30 18:35:12.916488 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:12.916466 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" Oct 30 18:35:12.930861 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:12.930835 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-d82e82-openshift-default-dockercfg-sc672\"" Oct 30 18:35:12.948113 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:12.948080 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll"] Oct 30 18:35:13.037839 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:13.037802 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04780470-66fb-45c6-b86e-8befadb35217-istio-token\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll\" (UID: \"04780470-66fb-45c6-b86e-8befadb35217\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" Oct 30 18:35:13.038055 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:13.037853 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04780470-66fb-45c6-b86e-8befadb35217-credential-socket\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll\" (UID: \"04780470-66fb-45c6-b86e-8befadb35217\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" Oct 30 18:35:13.038055 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:13.037982 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04780470-66fb-45c6-b86e-8befadb35217-istiod-ca-cert\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll\" (UID: \"04780470-66fb-45c6-b86e-8befadb35217\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" Oct 30 18:35:13.038055 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:13.038048 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04780470-66fb-45c6-b86e-8befadb35217-workload-socket\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll\" (UID: \"04780470-66fb-45c6-b86e-8befadb35217\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" Oct 30 18:35:13.038304 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:13.038251 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04780470-66fb-45c6-b86e-8befadb35217-istio-envoy\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll\" (UID: \"04780470-66fb-45c6-b86e-8befadb35217\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" Oct 30 18:35:13.038304 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:13.038295 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04780470-66fb-45c6-b86e-8befadb35217-istio-data\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll\" (UID: \"04780470-66fb-45c6-b86e-8befadb35217\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" Oct 30 18:35:13.038442 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:13.038415 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04780470-66fb-45c6-b86e-8befadb35217-workload-certs\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll\" (UID: \"04780470-66fb-45c6-b86e-8befadb35217\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" Oct 30 18:35:13.038602 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:13.038582 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrrz6\" (UniqueName: \"kubernetes.io/projected/04780470-66fb-45c6-b86e-8befadb35217-kube-api-access-zrrz6\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll\" (UID: \"04780470-66fb-45c6-b86e-8befadb35217\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" Oct 30 18:35:13.038653 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:13.038631 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04780470-66fb-45c6-b86e-8befadb35217-istio-podinfo\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll\" (UID: \"04780470-66fb-45c6-b86e-8befadb35217\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" Oct 30 18:35:13.140260 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:13.140220 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04780470-66fb-45c6-b86e-8befadb35217-istio-envoy\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll\" (UID: \"04780470-66fb-45c6-b86e-8befadb35217\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" Oct 30 18:35:13.140260 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:13.140265 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04780470-66fb-45c6-b86e-8befadb35217-istio-data\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll\" (UID: \"04780470-66fb-45c6-b86e-8befadb35217\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" Oct 30 18:35:13.140558 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:13.140300 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04780470-66fb-45c6-b86e-8befadb35217-workload-certs\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll\" (UID: \"04780470-66fb-45c6-b86e-8befadb35217\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" Oct 30 18:35:13.140558 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:13.140433 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrrz6\" (UniqueName: \"kubernetes.io/projected/04780470-66fb-45c6-b86e-8befadb35217-kube-api-access-zrrz6\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll\" (UID: \"04780470-66fb-45c6-b86e-8befadb35217\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" Oct 30 18:35:13.140558 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:13.140477 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04780470-66fb-45c6-b86e-8befadb35217-istio-podinfo\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll\" (UID: \"04780470-66fb-45c6-b86e-8befadb35217\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" Oct 30 18:35:13.140558 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:13.140508 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04780470-66fb-45c6-b86e-8befadb35217-istio-token\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll\" (UID: \"04780470-66fb-45c6-b86e-8befadb35217\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" Oct 30 18:35:13.140558 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:13.140550 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04780470-66fb-45c6-b86e-8befadb35217-credential-socket\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll\" (UID: \"04780470-66fb-45c6-b86e-8befadb35217\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" Oct 30 18:35:13.140810 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:13.140648 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04780470-66fb-45c6-b86e-8befadb35217-istiod-ca-cert\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll\" (UID: \"04780470-66fb-45c6-b86e-8befadb35217\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" Oct 30 18:35:13.140810 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:13.140757 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04780470-66fb-45c6-b86e-8befadb35217-workload-socket\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll\" (UID: \"04780470-66fb-45c6-b86e-8befadb35217\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" Oct 30 18:35:13.141356 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:13.141318 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04780470-66fb-45c6-b86e-8befadb35217-istio-data\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll\" (UID: \"04780470-66fb-45c6-b86e-8befadb35217\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" Oct 30 18:35:13.141481 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:13.141369 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04780470-66fb-45c6-b86e-8befadb35217-workload-certs\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll\" (UID: \"04780470-66fb-45c6-b86e-8befadb35217\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" Oct 30 18:35:13.141481 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:13.141459 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04780470-66fb-45c6-b86e-8befadb35217-credential-socket\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll\" (UID: \"04780470-66fb-45c6-b86e-8befadb35217\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" Oct 30 18:35:13.141640 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:13.141614 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04780470-66fb-45c6-b86e-8befadb35217-workload-socket\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll\" (UID: \"04780470-66fb-45c6-b86e-8befadb35217\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" Oct 30 18:35:13.142112 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:13.142093 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04780470-66fb-45c6-b86e-8befadb35217-istiod-ca-cert\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll\" (UID: \"04780470-66fb-45c6-b86e-8befadb35217\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" Oct 30 18:35:13.144393 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:13.144367 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04780470-66fb-45c6-b86e-8befadb35217-istio-podinfo\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll\" (UID: \"04780470-66fb-45c6-b86e-8befadb35217\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" Oct 30 18:35:13.144585 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:13.144565 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04780470-66fb-45c6-b86e-8befadb35217-istio-envoy\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll\" (UID: \"04780470-66fb-45c6-b86e-8befadb35217\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" Oct 30 18:35:13.161234 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:13.161108 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04780470-66fb-45c6-b86e-8befadb35217-istio-token\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll\" (UID: \"04780470-66fb-45c6-b86e-8befadb35217\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" Oct 30 18:35:13.161234 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:13.161185 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrrz6\" (UniqueName: \"kubernetes.io/projected/04780470-66fb-45c6-b86e-8befadb35217-kube-api-access-zrrz6\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll\" (UID: \"04780470-66fb-45c6-b86e-8befadb35217\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" Oct 30 18:35:13.233787 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:13.233755 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" Oct 30 18:35:13.558958 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:35:13.558915 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04780470_66fb_45c6_b86e_8befadb35217.slice/crio-daa90ee13b402c3fa44c55776cf7eaba845aca9bae3de73569642575a7f897bb WatchSource:0}: Error finding container daa90ee13b402c3fa44c55776cf7eaba845aca9bae3de73569642575a7f897bb: Status 404 returned error can't find the container with id daa90ee13b402c3fa44c55776cf7eaba845aca9bae3de73569642575a7f897bb Oct 30 18:35:13.560279 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:13.560155 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll"] Oct 30 18:35:13.563028 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:13.562991 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:35:13.563277 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:13.563231 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:35:13.563392 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:13.563295 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:35:13.789723 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:13.789685 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" event={"ID":"04780470-66fb-45c6-b86e-8befadb35217","Type":"ContainerStarted","Data":"ad3911fabe4c2c631c1add2c9c8231cb461f13a2b61b3e851896815f13aa0829"} Oct 30 18:35:13.789916 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:13.789750 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" event={"ID":"04780470-66fb-45c6-b86e-8befadb35217","Type":"ContainerStarted","Data":"daa90ee13b402c3fa44c55776cf7eaba845aca9bae3de73569642575a7f897bb"} Oct 30 18:35:13.850610 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:13.850497 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" podStartSLOduration=1.850480282 podStartE2EDuration="1.850480282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:35:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:35:13.834919415 +0000 UTC m=+2210.924981757" watchObservedRunningTime="2026-10-30 18:35:13.850480282 +0000 UTC m=+2210.940542620" Oct 30 18:35:14.234121 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:14.234085 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" Oct 30 18:35:14.236042 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:14.236013 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 30 18:35:14.236202 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:14.236073 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" podUID="04780470-66fb-45c6-b86e-8befadb35217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 30 18:35:15.247155 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:15.244556 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 30 18:35:15.247155 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:15.244617 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" podUID="04780470-66fb-45c6-b86e-8befadb35217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 30 18:35:16.240165 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:16.234510 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 30 18:35:16.240165 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:16.234570 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" podUID="04780470-66fb-45c6-b86e-8befadb35217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 30 18:35:17.245160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:17.241071 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 30 18:35:17.245160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:17.241256 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" podUID="04780470-66fb-45c6-b86e-8befadb35217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 30 18:35:17.582811 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:35:17.582698 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:35:18.239177 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:18.237509 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 30 18:35:18.239177 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:18.237579 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" podUID="04780470-66fb-45c6-b86e-8befadb35217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 30 18:35:19.247160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:19.241561 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 30 18:35:19.247160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:19.241631 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" podUID="04780470-66fb-45c6-b86e-8befadb35217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 30 18:35:20.234372 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:20.234334 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 30 18:35:20.234554 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:20.234399 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" podUID="04780470-66fb-45c6-b86e-8befadb35217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 30 18:35:21.239750 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:21.239710 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" Oct 30 18:35:21.240292 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:21.240089 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" Oct 30 18:35:21.241458 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:21.241432 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" Oct 30 18:35:37.146767 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:37.146723 4442 generic.go:356] "Generic (PLEG): container finished" podID="6bbe6b88-960d-47db-a270-47fc4cee7f44" containerID="43183b0e889c19bb57010aa96bdb89000fea7f09d635fce2c57165a4a0ce5a53" exitCode=0 Oct 30 18:35:37.147342 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:37.146792 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" event={"ID":"6bbe6b88-960d-47db-a270-47fc4cee7f44","Type":"ContainerDied","Data":"43183b0e889c19bb57010aa96bdb89000fea7f09d635fce2c57165a4a0ce5a53"} Oct 30 18:35:37.147342 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:37.147288 4442 scope.go:122] "RemoveContainer" containerID="43183b0e889c19bb57010aa96bdb89000fea7f09d635fce2c57165a4a0ce5a53" Oct 30 18:35:37.149707 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:37.149653 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:35:37.149869 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:37.149745 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:35:37.149869 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:37.149781 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:35:38.169621 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:38.169584 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" event={"ID":"6bbe6b88-960d-47db-a270-47fc4cee7f44","Type":"ContainerStarted","Data":"870d83262330ab1628bebb565ff996a55a75ecba3831a6ce2e6b6d5c310682af"} Oct 30 18:35:38.652158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:38.651594 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" Oct 30 18:35:38.652158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:38.651725 4442 patch_prober.go:29] interesting pod/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Oct 30 18:35:38.652158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:38.651783 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" podUID="6bbe6b88-960d-47db-a270-47fc4cee7f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Oct 30 18:35:39.659569 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:39.659528 4442 patch_prober.go:29] interesting pod/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Oct 30 18:35:39.660228 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:39.660181 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" podUID="6bbe6b88-960d-47db-a270-47fc4cee7f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Oct 30 18:35:40.651765 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:40.651722 4442 patch_prober.go:29] interesting pod/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Oct 30 18:35:40.652091 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:40.652059 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" podUID="6bbe6b88-960d-47db-a270-47fc4cee7f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Oct 30 18:35:41.655210 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:41.648570 4442 patch_prober.go:29] interesting pod/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Oct 30 18:35:41.655210 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:41.648638 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" podUID="6bbe6b88-960d-47db-a270-47fc4cee7f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Oct 30 18:35:42.651590 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:42.651552 4442 patch_prober.go:29] interesting pod/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Oct 30 18:35:42.651910 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:42.651881 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" podUID="6bbe6b88-960d-47db-a270-47fc4cee7f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Oct 30 18:35:43.722996 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:43.722966 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" Oct 30 18:35:43.723954 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:43.723925 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" Oct 30 18:35:43.724989 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:43.724963 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" Oct 30 18:35:47.563180 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:35:47.563116 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:35:53.583179 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:53.581522 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn"] Oct 30 18:35:53.583179 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:53.582285 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" podUID="a122a97d-09d1-485a-902c-2f7832deab2c" containerName="istio-proxy" containerID="cri-o://49ce8ab2a48c1face88a013012ca4aa2d43b32be4f07aaa85b067d1eb2c11664" gracePeriod=30 Oct 30 18:35:53.603631 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:53.602590 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl"] Oct 30 18:35:53.603631 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:53.602790 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" podUID="80927367-fb50-4945-86d8-9d22cf64f0ef" containerName="istio-proxy" containerID="cri-o://9b402b775fb6c429f5ac017e702b505cfa164329c0cef8e0c69685c9ce7cfbb4" gracePeriod=30 Oct 30 18:35:53.611925 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:53.611887 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj"] Oct 30 18:35:53.612961 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:53.612183 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" podUID="9760b73c-522d-4255-b72b-c0786527ded6" containerName="istio-proxy" containerID="cri-o://efb047674294cc3b828900791f861bb68cc61db6bcb6ca5da05d8bc7469bfad5" gracePeriod=30 Oct 30 18:35:53.619747 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:53.618164 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw"] Oct 30 18:35:53.619747 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:53.618353 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" podUID="83443c95-ada0-4803-beec-b66c915f5e50" containerName="istio-proxy" containerID="cri-o://c4fdb8c7bbdcb0df1592046e6b68a79ef6cb462bf943db092005658b54eaa50d" gracePeriod=30 Oct 30 18:35:53.782839 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:53.782449 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-160bf8-dthqr"] Oct 30 18:35:53.789614 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:53.789575 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-160bf8-dthqr" Oct 30 18:35:53.807235 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:53.797489 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-160bf8-dthqr"] Oct 30 18:35:53.809767 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:53.807963 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:35:53.809767 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:53.808091 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:35:53.861416 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:53.861328 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f35562b-b5c5-469e-b445-86a5ece915bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-160bf8-dthqr\" (UID: \"8f35562b-b5c5-469e-b445-86a5ece915bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-160bf8-dthqr" Oct 30 18:35:53.861578 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:53.861413 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f35562b-b5c5-469e-b445-86a5ece915bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-160bf8-dthqr\" (UID: \"8f35562b-b5c5-469e-b445-86a5ece915bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-160bf8-dthqr" Oct 30 18:35:53.861742 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:53.861712 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4jzr\" (UniqueName: \"kubernetes.io/projected/8f35562b-b5c5-469e-b445-86a5ece915bb-kube-api-access-l4jzr\") pod \"maas-api-revoke-keys-e2e-explicit-160bf8-dthqr\" (UID: \"8f35562b-b5c5-469e-b445-86a5ece915bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-160bf8-dthqr" Oct 30 18:35:53.981666 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:53.977586 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f35562b-b5c5-469e-b445-86a5ece915bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-160bf8-dthqr\" (UID: \"8f35562b-b5c5-469e-b445-86a5ece915bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-160bf8-dthqr" Oct 30 18:35:53.981666 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:53.977706 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f35562b-b5c5-469e-b445-86a5ece915bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-160bf8-dthqr\" (UID: \"8f35562b-b5c5-469e-b445-86a5ece915bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-160bf8-dthqr" Oct 30 18:35:53.981666 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:53.977904 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4jzr\" (UniqueName: \"kubernetes.io/projected/8f35562b-b5c5-469e-b445-86a5ece915bb-kube-api-access-l4jzr\") pod \"maas-api-revoke-keys-e2e-explicit-160bf8-dthqr\" (UID: \"8f35562b-b5c5-469e-b445-86a5ece915bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-160bf8-dthqr" Oct 30 18:35:54.009775 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:54.009738 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4jzr\" (UniqueName: \"kubernetes.io/projected/8f35562b-b5c5-469e-b445-86a5ece915bb-kube-api-access-l4jzr\") pod \"maas-api-revoke-keys-e2e-explicit-160bf8-dthqr\" (UID: \"8f35562b-b5c5-469e-b445-86a5ece915bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-160bf8-dthqr" Oct 30 18:35:54.020159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:54.010535 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f35562b-b5c5-469e-b445-86a5ece915bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-160bf8-dthqr\" (UID: \"8f35562b-b5c5-469e-b445-86a5ece915bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-160bf8-dthqr" Oct 30 18:35:54.030570 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:54.030535 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f35562b-b5c5-469e-b445-86a5ece915bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-160bf8-dthqr\" (UID: \"8f35562b-b5c5-469e-b445-86a5ece915bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-160bf8-dthqr" Oct 30 18:35:54.116275 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:54.115409 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-160bf8-dthqr" Oct 30 18:35:54.441762 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:35:54.441728 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f35562b_b5c5_469e_b445_86a5ece915bb.slice/crio-3ca2de65cfe14629cd27e42c8fbf6a8185d9944db6524000dd9003ef0baddaa6 WatchSource:0}: Error finding container 3ca2de65cfe14629cd27e42c8fbf6a8185d9944db6524000dd9003ef0baddaa6: Status 404 returned error can't find the container with id 3ca2de65cfe14629cd27e42c8fbf6a8185d9944db6524000dd9003ef0baddaa6 Oct 30 18:35:54.442508 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:54.442484 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-160bf8-dthqr"] Oct 30 18:35:55.340177 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:55.340109 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-160bf8-dthqr" event={"ID":"8f35562b-b5c5-469e-b445-86a5ece915bb","Type":"ContainerStarted","Data":"3f0f5a9610b7599c6cd39fd20431d3e3be0d3099a8efd90ed8f54b5d556ff206"} Oct 30 18:35:55.340599 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:55.340198 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-160bf8-dthqr" event={"ID":"8f35562b-b5c5-469e-b445-86a5ece915bb","Type":"ContainerStarted","Data":"3ca2de65cfe14629cd27e42c8fbf6a8185d9944db6524000dd9003ef0baddaa6"} Oct 30 18:35:55.380402 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:55.380344 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-160bf8-dthqr" podStartSLOduration=2.380330002 podStartE2EDuration="2.380330002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:35:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:35:55.376672152 +0000 UTC m=+2252.466734491" watchObservedRunningTime="2026-10-30 18:35:55.380330002 +0000 UTC m=+2252.470392340" Oct 30 18:35:56.351272 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:56.351227 4442 generic.go:356] "Generic (PLEG): container finished" podID="8f35562b-b5c5-469e-b445-86a5ece915bb" containerID="3f0f5a9610b7599c6cd39fd20431d3e3be0d3099a8efd90ed8f54b5d556ff206" exitCode=0 Oct 30 18:35:56.351734 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:56.351275 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-160bf8-dthqr" event={"ID":"8f35562b-b5c5-469e-b445-86a5ece915bb","Type":"ContainerDied","Data":"3f0f5a9610b7599c6cd39fd20431d3e3be0d3099a8efd90ed8f54b5d556ff206"} Oct 30 18:35:57.552268 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:57.552238 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-160bf8-dthqr" Oct 30 18:35:57.649073 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:57.649031 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f35562b-b5c5-469e-b445-86a5ece915bb-kube-api-access-l4jzr\" (UniqueName: \"kubernetes.io/projected/8f35562b-b5c5-469e-b445-86a5ece915bb-kube-api-access-l4jzr\") pod \"8f35562b-b5c5-469e-b445-86a5ece915bb\" (UID: \"8f35562b-b5c5-469e-b445-86a5ece915bb\") " Oct 30 18:35:57.649471 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:57.649350 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f35562b-b5c5-469e-b445-86a5ece915bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f35562b-b5c5-469e-b445-86a5ece915bb-maas-api-xks-ca\") pod \"8f35562b-b5c5-469e-b445-86a5ece915bb\" (UID: \"8f35562b-b5c5-469e-b445-86a5ece915bb\") " Oct 30 18:35:57.649671 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:57.649508 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f35562b-b5c5-469e-b445-86a5ece915bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f35562b-b5c5-469e-b445-86a5ece915bb-maas-api-openshift-ca\") pod \"8f35562b-b5c5-469e-b445-86a5ece915bb\" (UID: \"8f35562b-b5c5-469e-b445-86a5ece915bb\") " Oct 30 18:35:57.651212 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:57.651182 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f35562b-b5c5-469e-b445-86a5ece915bb-maas-api-openshift-ca" pod "8f35562b-b5c5-469e-b445-86a5ece915bb" (UID: "8f35562b-b5c5-469e-b445-86a5ece915bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:35:57.652565 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:57.652541 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f35562b-b5c5-469e-b445-86a5ece915bb-kube-api-access-l4jzr" pod "8f35562b-b5c5-469e-b445-86a5ece915bb" (UID: "8f35562b-b5c5-469e-b445-86a5ece915bb"). InnerVolumeSpecName "kube-api-access-l4jzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:35:57.653088 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:57.653073 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f35562b-b5c5-469e-b445-86a5ece915bb-maas-api-xks-ca" pod "8f35562b-b5c5-469e-b445-86a5ece915bb" (UID: "8f35562b-b5c5-469e-b445-86a5ece915bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:35:57.751822 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:57.751791 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f35562b-b5c5-469e-b445-86a5ece915bb-maas-api-xks-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:35:57.751822 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:57.751819 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f35562b-b5c5-469e-b445-86a5ece915bb-maas-api-openshift-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:35:57.751822 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:57.751828 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4jzr\" (UniqueName: \"kubernetes.io/projected/8f35562b-b5c5-469e-b445-86a5ece915bb-kube-api-access-l4jzr\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:35:58.382259 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:58.382227 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-160bf8-dthqr" Oct 30 18:35:58.382469 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:58.382263 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-160bf8-dthqr" event={"ID":"8f35562b-b5c5-469e-b445-86a5ece915bb","Type":"ContainerDied","Data":"3ca2de65cfe14629cd27e42c8fbf6a8185d9944db6524000dd9003ef0baddaa6"} Oct 30 18:35:58.382469 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:58.382341 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ca2de65cfe14629cd27e42c8fbf6a8185d9944db6524000dd9003ef0baddaa6" Oct 30 18:35:58.970289 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:58.966632 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" Oct 30 18:35:59.022189 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.022146 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" Oct 30 18:35:59.054975 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.054932 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" Oct 30 18:35:59.089323 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.088412 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80927367-fb50-4945-86d8-9d22cf64f0ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80927367-fb50-4945-86d8-9d22cf64f0ef-istio-data\") pod \"80927367-fb50-4945-86d8-9d22cf64f0ef\" (UID: \"80927367-fb50-4945-86d8-9d22cf64f0ef\") " Oct 30 18:35:59.089323 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.088498 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80927367-fb50-4945-86d8-9d22cf64f0ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80927367-fb50-4945-86d8-9d22cf64f0ef-workload-certs\") pod \"80927367-fb50-4945-86d8-9d22cf64f0ef\" (UID: \"80927367-fb50-4945-86d8-9d22cf64f0ef\") " Oct 30 18:35:59.089323 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.088560 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80927367-fb50-4945-86d8-9d22cf64f0ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80927367-fb50-4945-86d8-9d22cf64f0ef-istio-envoy\") pod \"80927367-fb50-4945-86d8-9d22cf64f0ef\" (UID: \"80927367-fb50-4945-86d8-9d22cf64f0ef\") " Oct 30 18:35:59.089323 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.088595 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80927367-fb50-4945-86d8-9d22cf64f0ef-kube-api-access-ff68j\" (UniqueName: \"kubernetes.io/projected/80927367-fb50-4945-86d8-9d22cf64f0ef-kube-api-access-ff68j\") pod \"80927367-fb50-4945-86d8-9d22cf64f0ef\" (UID: \"80927367-fb50-4945-86d8-9d22cf64f0ef\") " Oct 30 18:35:59.089323 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.088654 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80927367-fb50-4945-86d8-9d22cf64f0ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80927367-fb50-4945-86d8-9d22cf64f0ef-workload-socket\") pod \"80927367-fb50-4945-86d8-9d22cf64f0ef\" (UID: \"80927367-fb50-4945-86d8-9d22cf64f0ef\") " Oct 30 18:35:59.089323 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.088731 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80927367-fb50-4945-86d8-9d22cf64f0ef-istio-token\" (UniqueName: \"kubernetes.io/projected/80927367-fb50-4945-86d8-9d22cf64f0ef-istio-token\") pod \"80927367-fb50-4945-86d8-9d22cf64f0ef\" (UID: \"80927367-fb50-4945-86d8-9d22cf64f0ef\") " Oct 30 18:35:59.089323 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.088834 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80927367-fb50-4945-86d8-9d22cf64f0ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80927367-fb50-4945-86d8-9d22cf64f0ef-istiod-ca-cert\") pod \"80927367-fb50-4945-86d8-9d22cf64f0ef\" (UID: \"80927367-fb50-4945-86d8-9d22cf64f0ef\") " Oct 30 18:35:59.089323 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.088871 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80927367-fb50-4945-86d8-9d22cf64f0ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80927367-fb50-4945-86d8-9d22cf64f0ef-istio-podinfo\") pod \"80927367-fb50-4945-86d8-9d22cf64f0ef\" (UID: \"80927367-fb50-4945-86d8-9d22cf64f0ef\") " Oct 30 18:35:59.089323 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.088966 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80927367-fb50-4945-86d8-9d22cf64f0ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80927367-fb50-4945-86d8-9d22cf64f0ef-credential-socket\") pod \"80927367-fb50-4945-86d8-9d22cf64f0ef\" (UID: \"80927367-fb50-4945-86d8-9d22cf64f0ef\") " Oct 30 18:35:59.089963 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.089887 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80927367-fb50-4945-86d8-9d22cf64f0ef-istio-data" pod "80927367-fb50-4945-86d8-9d22cf64f0ef" (UID: "80927367-fb50-4945-86d8-9d22cf64f0ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:35:59.096379 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.090082 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80927367-fb50-4945-86d8-9d22cf64f0ef-workload-socket" pod "80927367-fb50-4945-86d8-9d22cf64f0ef" (UID: "80927367-fb50-4945-86d8-9d22cf64f0ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:35:59.096379 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.090380 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80927367-fb50-4945-86d8-9d22cf64f0ef-istiod-ca-cert" pod "80927367-fb50-4945-86d8-9d22cf64f0ef" (UID: "80927367-fb50-4945-86d8-9d22cf64f0ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:35:59.096379 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.090799 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80927367-fb50-4945-86d8-9d22cf64f0ef-workload-certs" pod "80927367-fb50-4945-86d8-9d22cf64f0ef" (UID: "80927367-fb50-4945-86d8-9d22cf64f0ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:35:59.096379 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.091346 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80927367-fb50-4945-86d8-9d22cf64f0ef-credential-socket" pod "80927367-fb50-4945-86d8-9d22cf64f0ef" (UID: "80927367-fb50-4945-86d8-9d22cf64f0ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:35:59.096379 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.091693 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80927367-fb50-4945-86d8-9d22cf64f0ef-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:35:59.096379 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.091717 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80927367-fb50-4945-86d8-9d22cf64f0ef-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:35:59.096379 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.091733 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80927367-fb50-4945-86d8-9d22cf64f0ef-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:35:59.096379 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.091746 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80927367-fb50-4945-86d8-9d22cf64f0ef-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:35:59.096379 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.091759 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80927367-fb50-4945-86d8-9d22cf64f0ef-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:35:59.096379 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.095377 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80927367-fb50-4945-86d8-9d22cf64f0ef-kube-api-access-ff68j" pod "80927367-fb50-4945-86d8-9d22cf64f0ef" (UID: "80927367-fb50-4945-86d8-9d22cf64f0ef"). InnerVolumeSpecName "kube-api-access-ff68j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:35:59.096379 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.096012 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80927367-fb50-4945-86d8-9d22cf64f0ef-istio-token" pod "80927367-fb50-4945-86d8-9d22cf64f0ef" (UID: "80927367-fb50-4945-86d8-9d22cf64f0ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:35:59.097552 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.097521 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80927367-fb50-4945-86d8-9d22cf64f0ef-istio-podinfo" pod "80927367-fb50-4945-86d8-9d22cf64f0ef" (UID: "80927367-fb50-4945-86d8-9d22cf64f0ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:35:59.097970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.097938 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80927367-fb50-4945-86d8-9d22cf64f0ef-istio-envoy" pod "80927367-fb50-4945-86d8-9d22cf64f0ef" (UID: "80927367-fb50-4945-86d8-9d22cf64f0ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:35:59.105365 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.105282 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" Oct 30 18:35:59.192588 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.192547 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a122a97d-09d1-485a-902c-2f7832deab2c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a122a97d-09d1-485a-902c-2f7832deab2c-istio-podinfo\") pod \"a122a97d-09d1-485a-902c-2f7832deab2c\" (UID: \"a122a97d-09d1-485a-902c-2f7832deab2c\") " Oct 30 18:35:59.192838 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.192665 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83443c95-ada0-4803-beec-b66c915f5e50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83443c95-ada0-4803-beec-b66c915f5e50-workload-certs\") pod \"83443c95-ada0-4803-beec-b66c915f5e50\" (UID: \"83443c95-ada0-4803-beec-b66c915f5e50\") " Oct 30 18:35:59.192838 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.192720 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83443c95-ada0-4803-beec-b66c915f5e50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83443c95-ada0-4803-beec-b66c915f5e50-istio-podinfo\") pod \"83443c95-ada0-4803-beec-b66c915f5e50\" (UID: \"83443c95-ada0-4803-beec-b66c915f5e50\") " Oct 30 18:35:59.192838 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.192828 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83443c95-ada0-4803-beec-b66c915f5e50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83443c95-ada0-4803-beec-b66c915f5e50-istiod-ca-cert\") pod \"83443c95-ada0-4803-beec-b66c915f5e50\" (UID: \"83443c95-ada0-4803-beec-b66c915f5e50\") " Oct 30 18:35:59.193091 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.192885 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83443c95-ada0-4803-beec-b66c915f5e50-istio-token\" (UniqueName: \"kubernetes.io/projected/83443c95-ada0-4803-beec-b66c915f5e50-istio-token\") pod \"83443c95-ada0-4803-beec-b66c915f5e50\" (UID: \"83443c95-ada0-4803-beec-b66c915f5e50\") " Oct 30 18:35:59.193091 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.192926 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83443c95-ada0-4803-beec-b66c915f5e50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83443c95-ada0-4803-beec-b66c915f5e50-credential-socket\") pod \"83443c95-ada0-4803-beec-b66c915f5e50\" (UID: \"83443c95-ada0-4803-beec-b66c915f5e50\") " Oct 30 18:35:59.193091 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.192963 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83443c95-ada0-4803-beec-b66c915f5e50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83443c95-ada0-4803-beec-b66c915f5e50-istio-envoy\") pod \"83443c95-ada0-4803-beec-b66c915f5e50\" (UID: \"83443c95-ada0-4803-beec-b66c915f5e50\") " Oct 30 18:35:59.193091 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.193037 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a122a97d-09d1-485a-902c-2f7832deab2c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a122a97d-09d1-485a-902c-2f7832deab2c-workload-socket\") pod \"a122a97d-09d1-485a-902c-2f7832deab2c\" (UID: \"a122a97d-09d1-485a-902c-2f7832deab2c\") " Oct 30 18:35:59.193091 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.193077 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a122a97d-09d1-485a-902c-2f7832deab2c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a122a97d-09d1-485a-902c-2f7832deab2c-istio-data\") pod \"a122a97d-09d1-485a-902c-2f7832deab2c\" (UID: \"a122a97d-09d1-485a-902c-2f7832deab2c\") " Oct 30 18:35:59.193460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.193104 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a122a97d-09d1-485a-902c-2f7832deab2c-istio-token\" (UniqueName: \"kubernetes.io/projected/a122a97d-09d1-485a-902c-2f7832deab2c-istio-token\") pod \"a122a97d-09d1-485a-902c-2f7832deab2c\" (UID: \"a122a97d-09d1-485a-902c-2f7832deab2c\") " Oct 30 18:35:59.193460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.193179 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a122a97d-09d1-485a-902c-2f7832deab2c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a122a97d-09d1-485a-902c-2f7832deab2c-workload-certs\") pod \"a122a97d-09d1-485a-902c-2f7832deab2c\" (UID: \"a122a97d-09d1-485a-902c-2f7832deab2c\") " Oct 30 18:35:59.193460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.193208 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83443c95-ada0-4803-beec-b66c915f5e50-kube-api-access-gdrw5\" (UniqueName: \"kubernetes.io/projected/83443c95-ada0-4803-beec-b66c915f5e50-kube-api-access-gdrw5\") pod \"83443c95-ada0-4803-beec-b66c915f5e50\" (UID: \"83443c95-ada0-4803-beec-b66c915f5e50\") " Oct 30 18:35:59.193460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.193277 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a122a97d-09d1-485a-902c-2f7832deab2c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a122a97d-09d1-485a-902c-2f7832deab2c-istiod-ca-cert\") pod \"a122a97d-09d1-485a-902c-2f7832deab2c\" (UID: \"a122a97d-09d1-485a-902c-2f7832deab2c\") " Oct 30 18:35:59.193460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.193374 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83443c95-ada0-4803-beec-b66c915f5e50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83443c95-ada0-4803-beec-b66c915f5e50-workload-socket\") pod \"83443c95-ada0-4803-beec-b66c915f5e50\" (UID: \"83443c95-ada0-4803-beec-b66c915f5e50\") " Oct 30 18:35:59.193460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.193415 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a122a97d-09d1-485a-902c-2f7832deab2c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a122a97d-09d1-485a-902c-2f7832deab2c-credential-socket\") pod \"a122a97d-09d1-485a-902c-2f7832deab2c\" (UID: \"a122a97d-09d1-485a-902c-2f7832deab2c\") " Oct 30 18:35:59.193785 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.193478 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83443c95-ada0-4803-beec-b66c915f5e50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83443c95-ada0-4803-beec-b66c915f5e50-istio-data\") pod \"83443c95-ada0-4803-beec-b66c915f5e50\" (UID: \"83443c95-ada0-4803-beec-b66c915f5e50\") " Oct 30 18:35:59.193785 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.193538 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a122a97d-09d1-485a-902c-2f7832deab2c-kube-api-access-57w9x\" (UniqueName: \"kubernetes.io/projected/a122a97d-09d1-485a-902c-2f7832deab2c-kube-api-access-57w9x\") pod \"a122a97d-09d1-485a-902c-2f7832deab2c\" (UID: \"a122a97d-09d1-485a-902c-2f7832deab2c\") " Oct 30 18:35:59.193785 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.193600 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a122a97d-09d1-485a-902c-2f7832deab2c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a122a97d-09d1-485a-902c-2f7832deab2c-istio-envoy\") pod \"a122a97d-09d1-485a-902c-2f7832deab2c\" (UID: \"a122a97d-09d1-485a-902c-2f7832deab2c\") " Oct 30 18:35:59.194286 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.194267 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80927367-fb50-4945-86d8-9d22cf64f0ef-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:35:59.194381 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.194292 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff68j\" (UniqueName: \"kubernetes.io/projected/80927367-fb50-4945-86d8-9d22cf64f0ef-kube-api-access-ff68j\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:35:59.194381 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.194305 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80927367-fb50-4945-86d8-9d22cf64f0ef-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:35:59.194381 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.194317 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80927367-fb50-4945-86d8-9d22cf64f0ef-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:35:59.195574 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.195477 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a122a97d-09d1-485a-902c-2f7832deab2c-workload-certs" pod "a122a97d-09d1-485a-902c-2f7832deab2c" (UID: "a122a97d-09d1-485a-902c-2f7832deab2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:35:59.195748 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.195620 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a122a97d-09d1-485a-902c-2f7832deab2c-istiod-ca-cert" pod "a122a97d-09d1-485a-902c-2f7832deab2c" (UID: "a122a97d-09d1-485a-902c-2f7832deab2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:35:59.196064 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.196037 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83443c95-ada0-4803-beec-b66c915f5e50-workload-certs" pod "83443c95-ada0-4803-beec-b66c915f5e50" (UID: "83443c95-ada0-4803-beec-b66c915f5e50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:35:59.196205 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.196156 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83443c95-ada0-4803-beec-b66c915f5e50-istio-data" pod "83443c95-ada0-4803-beec-b66c915f5e50" (UID: "83443c95-ada0-4803-beec-b66c915f5e50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:35:59.196348 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.196311 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83443c95-ada0-4803-beec-b66c915f5e50-workload-socket" pod "83443c95-ada0-4803-beec-b66c915f5e50" (UID: "83443c95-ada0-4803-beec-b66c915f5e50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:35:59.196428 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.196379 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a122a97d-09d1-485a-902c-2f7832deab2c-credential-socket" pod "a122a97d-09d1-485a-902c-2f7832deab2c" (UID: "a122a97d-09d1-485a-902c-2f7832deab2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:35:59.196491 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.196474 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a122a97d-09d1-485a-902c-2f7832deab2c-istio-data" pod "a122a97d-09d1-485a-902c-2f7832deab2c" (UID: "a122a97d-09d1-485a-902c-2f7832deab2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:35:59.196682 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.196662 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a122a97d-09d1-485a-902c-2f7832deab2c-workload-socket" pod "a122a97d-09d1-485a-902c-2f7832deab2c" (UID: "a122a97d-09d1-485a-902c-2f7832deab2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:35:59.197192 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.197159 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83443c95-ada0-4803-beec-b66c915f5e50-istiod-ca-cert" pod "83443c95-ada0-4803-beec-b66c915f5e50" (UID: "83443c95-ada0-4803-beec-b66c915f5e50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:35:59.197295 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.197272 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a122a97d-09d1-485a-902c-2f7832deab2c-istio-podinfo" pod "a122a97d-09d1-485a-902c-2f7832deab2c" (UID: "a122a97d-09d1-485a-902c-2f7832deab2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:35:59.197762 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.197738 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83443c95-ada0-4803-beec-b66c915f5e50-credential-socket" pod "83443c95-ada0-4803-beec-b66c915f5e50" (UID: "83443c95-ada0-4803-beec-b66c915f5e50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:35:59.199259 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.199199 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a122a97d-09d1-485a-902c-2f7832deab2c-istio-token" pod "a122a97d-09d1-485a-902c-2f7832deab2c" (UID: "a122a97d-09d1-485a-902c-2f7832deab2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:35:59.199450 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.199429 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83443c95-ada0-4803-beec-b66c915f5e50-istio-token" pod "83443c95-ada0-4803-beec-b66c915f5e50" (UID: "83443c95-ada0-4803-beec-b66c915f5e50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:35:59.200499 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.200475 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83443c95-ada0-4803-beec-b66c915f5e50-istio-envoy" pod "83443c95-ada0-4803-beec-b66c915f5e50" (UID: "83443c95-ada0-4803-beec-b66c915f5e50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:35:59.200640 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.200513 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a122a97d-09d1-485a-902c-2f7832deab2c-istio-envoy" pod "a122a97d-09d1-485a-902c-2f7832deab2c" (UID: "a122a97d-09d1-485a-902c-2f7832deab2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:35:59.200640 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.200594 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a122a97d-09d1-485a-902c-2f7832deab2c-kube-api-access-57w9x" pod "a122a97d-09d1-485a-902c-2f7832deab2c" (UID: "a122a97d-09d1-485a-902c-2f7832deab2c"). InnerVolumeSpecName "kube-api-access-57w9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:35:59.200762 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.200681 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83443c95-ada0-4803-beec-b66c915f5e50-kube-api-access-gdrw5" pod "83443c95-ada0-4803-beec-b66c915f5e50" (UID: "83443c95-ada0-4803-beec-b66c915f5e50"). InnerVolumeSpecName "kube-api-access-gdrw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:35:59.202052 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.202001 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83443c95-ada0-4803-beec-b66c915f5e50-istio-podinfo" pod "83443c95-ada0-4803-beec-b66c915f5e50" (UID: "83443c95-ada0-4803-beec-b66c915f5e50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:35:59.295394 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.295342 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9760b73c-522d-4255-b72b-c0786527ded6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9760b73c-522d-4255-b72b-c0786527ded6-credential-socket\") pod \"9760b73c-522d-4255-b72b-c0786527ded6\" (UID: \"9760b73c-522d-4255-b72b-c0786527ded6\") " Oct 30 18:35:59.295604 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.295419 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9760b73c-522d-4255-b72b-c0786527ded6-kube-api-access-w9slt\" (UniqueName: \"kubernetes.io/projected/9760b73c-522d-4255-b72b-c0786527ded6-kube-api-access-w9slt\") pod \"9760b73c-522d-4255-b72b-c0786527ded6\" (UID: \"9760b73c-522d-4255-b72b-c0786527ded6\") " Oct 30 18:35:59.295604 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.295469 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9760b73c-522d-4255-b72b-c0786527ded6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9760b73c-522d-4255-b72b-c0786527ded6-workload-socket\") pod \"9760b73c-522d-4255-b72b-c0786527ded6\" (UID: \"9760b73c-522d-4255-b72b-c0786527ded6\") " Oct 30 18:35:59.295604 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.295508 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9760b73c-522d-4255-b72b-c0786527ded6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9760b73c-522d-4255-b72b-c0786527ded6-workload-certs\") pod \"9760b73c-522d-4255-b72b-c0786527ded6\" (UID: \"9760b73c-522d-4255-b72b-c0786527ded6\") " Oct 30 18:35:59.295604 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.295558 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9760b73c-522d-4255-b72b-c0786527ded6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9760b73c-522d-4255-b72b-c0786527ded6-istio-envoy\") pod \"9760b73c-522d-4255-b72b-c0786527ded6\" (UID: \"9760b73c-522d-4255-b72b-c0786527ded6\") " Oct 30 18:35:59.295842 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.295627 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9760b73c-522d-4255-b72b-c0786527ded6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9760b73c-522d-4255-b72b-c0786527ded6-istio-podinfo\") pod \"9760b73c-522d-4255-b72b-c0786527ded6\" (UID: \"9760b73c-522d-4255-b72b-c0786527ded6\") " Oct 30 18:35:59.295842 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.295750 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9760b73c-522d-4255-b72b-c0786527ded6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9760b73c-522d-4255-b72b-c0786527ded6-istiod-ca-cert\") pod \"9760b73c-522d-4255-b72b-c0786527ded6\" (UID: \"9760b73c-522d-4255-b72b-c0786527ded6\") " Oct 30 18:35:59.295956 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.295845 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9760b73c-522d-4255-b72b-c0786527ded6-istio-token\" (UniqueName: \"kubernetes.io/projected/9760b73c-522d-4255-b72b-c0786527ded6-istio-token\") pod \"9760b73c-522d-4255-b72b-c0786527ded6\" (UID: \"9760b73c-522d-4255-b72b-c0786527ded6\") " Oct 30 18:35:59.295956 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.295885 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9760b73c-522d-4255-b72b-c0786527ded6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9760b73c-522d-4255-b72b-c0786527ded6-istio-data\") pod \"9760b73c-522d-4255-b72b-c0786527ded6\" (UID: \"9760b73c-522d-4255-b72b-c0786527ded6\") " Oct 30 18:35:59.295956 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.295895 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9760b73c-522d-4255-b72b-c0786527ded6-workload-socket" pod "9760b73c-522d-4255-b72b-c0786527ded6" (UID: "9760b73c-522d-4255-b72b-c0786527ded6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:35:59.296184 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.295950 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9760b73c-522d-4255-b72b-c0786527ded6-workload-certs" pod "9760b73c-522d-4255-b72b-c0786527ded6" (UID: "9760b73c-522d-4255-b72b-c0786527ded6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:35:59.296356 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.296321 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9760b73c-522d-4255-b72b-c0786527ded6-credential-socket" pod "9760b73c-522d-4255-b72b-c0786527ded6" (UID: "9760b73c-522d-4255-b72b-c0786527ded6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:35:59.296602 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.296565 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9760b73c-522d-4255-b72b-c0786527ded6-istio-data" pod "9760b73c-522d-4255-b72b-c0786527ded6" (UID: "9760b73c-522d-4255-b72b-c0786527ded6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:35:59.296821 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.296799 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83443c95-ada0-4803-beec-b66c915f5e50-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:35:59.296821 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.296823 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a122a97d-09d1-485a-902c-2f7832deab2c-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:35:59.297011 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.296836 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9760b73c-522d-4255-b72b-c0786527ded6-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:35:59.297011 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.296849 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83443c95-ada0-4803-beec-b66c915f5e50-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:35:59.297011 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.296861 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57w9x\" (UniqueName: \"kubernetes.io/projected/a122a97d-09d1-485a-902c-2f7832deab2c-kube-api-access-57w9x\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:35:59.297011 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.296873 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a122a97d-09d1-485a-902c-2f7832deab2c-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:35:59.297011 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.296885 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a122a97d-09d1-485a-902c-2f7832deab2c-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:35:59.297011 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.296896 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9760b73c-522d-4255-b72b-c0786527ded6-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:35:59.297011 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.296909 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83443c95-ada0-4803-beec-b66c915f5e50-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:35:59.297011 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.296930 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9760b73c-522d-4255-b72b-c0786527ded6-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:35:59.297011 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.296943 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83443c95-ada0-4803-beec-b66c915f5e50-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:35:59.297011 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.296954 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9760b73c-522d-4255-b72b-c0786527ded6-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:35:59.297011 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.296969 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83443c95-ada0-4803-beec-b66c915f5e50-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:35:59.297011 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.296982 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83443c95-ada0-4803-beec-b66c915f5e50-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:35:59.297011 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.296995 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83443c95-ada0-4803-beec-b66c915f5e50-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:35:59.297011 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.297007 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83443c95-ada0-4803-beec-b66c915f5e50-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:35:59.297011 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.297004 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9760b73c-522d-4255-b72b-c0786527ded6-istiod-ca-cert" pod "9760b73c-522d-4255-b72b-c0786527ded6" (UID: "9760b73c-522d-4255-b72b-c0786527ded6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:35:59.297011 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.297020 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a122a97d-09d1-485a-902c-2f7832deab2c-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:35:59.298379 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.297032 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a122a97d-09d1-485a-902c-2f7832deab2c-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:35:59.298379 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.297046 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a122a97d-09d1-485a-902c-2f7832deab2c-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:35:59.298379 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.297058 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a122a97d-09d1-485a-902c-2f7832deab2c-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:35:59.298379 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.297071 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdrw5\" (UniqueName: \"kubernetes.io/projected/83443c95-ada0-4803-beec-b66c915f5e50-kube-api-access-gdrw5\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:35:59.298379 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.297084 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a122a97d-09d1-485a-902c-2f7832deab2c-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:35:59.299224 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.299197 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9760b73c-522d-4255-b72b-c0786527ded6-kube-api-access-w9slt" pod "9760b73c-522d-4255-b72b-c0786527ded6" (UID: "9760b73c-522d-4255-b72b-c0786527ded6"). InnerVolumeSpecName "kube-api-access-w9slt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:35:59.299490 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.299470 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9760b73c-522d-4255-b72b-c0786527ded6-istio-podinfo" pod "9760b73c-522d-4255-b72b-c0786527ded6" (UID: "9760b73c-522d-4255-b72b-c0786527ded6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:35:59.299674 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.299649 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9760b73c-522d-4255-b72b-c0786527ded6-istio-token" pod "9760b73c-522d-4255-b72b-c0786527ded6" (UID: "9760b73c-522d-4255-b72b-c0786527ded6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:35:59.300128 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.300097 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9760b73c-522d-4255-b72b-c0786527ded6-istio-envoy" pod "9760b73c-522d-4255-b72b-c0786527ded6" (UID: "9760b73c-522d-4255-b72b-c0786527ded6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:35:59.391561 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.391523 4442 generic.go:356] "Generic (PLEG): container finished" podID="80927367-fb50-4945-86d8-9d22cf64f0ef" containerID="9b402b775fb6c429f5ac017e702b505cfa164329c0cef8e0c69685c9ce7cfbb4" exitCode=0 Oct 30 18:35:59.391767 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.391570 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" Oct 30 18:35:59.391767 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.391606 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" event={"ID":"80927367-fb50-4945-86d8-9d22cf64f0ef","Type":"ContainerDied","Data":"9b402b775fb6c429f5ac017e702b505cfa164329c0cef8e0c69685c9ce7cfbb4"} Oct 30 18:35:59.391767 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.391662 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl" event={"ID":"80927367-fb50-4945-86d8-9d22cf64f0ef","Type":"ContainerDied","Data":"2f051281cefdfbc21768c904bbe0e116626c0fe5141d08c0d533c5d753169a4a"} Oct 30 18:35:59.391767 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.391674 4442 scope.go:122] "RemoveContainer" containerID="9b402b775fb6c429f5ac017e702b505cfa164329c0cef8e0c69685c9ce7cfbb4" Oct 30 18:35:59.393340 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.393311 4442 generic.go:356] "Generic (PLEG): container finished" podID="a122a97d-09d1-485a-902c-2f7832deab2c" containerID="49ce8ab2a48c1face88a013012ca4aa2d43b32be4f07aaa85b067d1eb2c11664" exitCode=0 Oct 30 18:35:59.393637 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.393451 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" event={"ID":"a122a97d-09d1-485a-902c-2f7832deab2c","Type":"ContainerDied","Data":"49ce8ab2a48c1face88a013012ca4aa2d43b32be4f07aaa85b067d1eb2c11664"} Oct 30 18:35:59.393637 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.393492 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" event={"ID":"a122a97d-09d1-485a-902c-2f7832deab2c","Type":"ContainerDied","Data":"fefb3aafe0c28ba23f79f7f2f935a13a8e51a42788b4f0caf8c041fecffea0e4"} Oct 30 18:35:59.393637 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.393567 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn" Oct 30 18:35:59.395894 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.395872 4442 generic.go:356] "Generic (PLEG): container finished" podID="9760b73c-522d-4255-b72b-c0786527ded6" containerID="efb047674294cc3b828900791f861bb68cc61db6bcb6ca5da05d8bc7469bfad5" exitCode=0 Oct 30 18:35:59.396047 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.395927 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" Oct 30 18:35:59.396047 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.395947 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" event={"ID":"9760b73c-522d-4255-b72b-c0786527ded6","Type":"ContainerDied","Data":"efb047674294cc3b828900791f861bb68cc61db6bcb6ca5da05d8bc7469bfad5"} Oct 30 18:35:59.396047 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.395997 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj" event={"ID":"9760b73c-522d-4255-b72b-c0786527ded6","Type":"ContainerDied","Data":"03ae4610beaf1d47531aba2b01a9776530008eadb824db1493fbce562f11f983"} Oct 30 18:35:59.400822 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.400783 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9slt\" (UniqueName: \"kubernetes.io/projected/9760b73c-522d-4255-b72b-c0786527ded6-kube-api-access-w9slt\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:35:59.400822 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.400828 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9760b73c-522d-4255-b72b-c0786527ded6-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:35:59.400822 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.400846 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9760b73c-522d-4255-b72b-c0786527ded6-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:35:59.401206 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.400860 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9760b73c-522d-4255-b72b-c0786527ded6-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:35:59.401206 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.400873 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9760b73c-522d-4255-b72b-c0786527ded6-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:35:59.402669 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.402507 4442 generic.go:356] "Generic (PLEG): container finished" podID="83443c95-ada0-4803-beec-b66c915f5e50" containerID="c4fdb8c7bbdcb0df1592046e6b68a79ef6cb462bf943db092005658b54eaa50d" exitCode=0 Oct 30 18:35:59.402669 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.402575 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" Oct 30 18:35:59.402669 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.402600 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" event={"ID":"83443c95-ada0-4803-beec-b66c915f5e50","Type":"ContainerDied","Data":"c4fdb8c7bbdcb0df1592046e6b68a79ef6cb462bf943db092005658b54eaa50d"} Oct 30 18:35:59.402669 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.402639 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw" event={"ID":"83443c95-ada0-4803-beec-b66c915f5e50","Type":"ContainerDied","Data":"79f77941264e6712acd889f1bb83ae571a9c0a40a24bc23edb36803b2955a6c1"} Oct 30 18:35:59.404606 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.404587 4442 scope.go:122] "RemoveContainer" containerID="9b402b775fb6c429f5ac017e702b505cfa164329c0cef8e0c69685c9ce7cfbb4" Oct 30 18:35:59.404916 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:35:59.404891 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b402b775fb6c429f5ac017e702b505cfa164329c0cef8e0c69685c9ce7cfbb4\": container with ID starting with 9b402b775fb6c429f5ac017e702b505cfa164329c0cef8e0c69685c9ce7cfbb4 not found: ID does not exist" containerID="9b402b775fb6c429f5ac017e702b505cfa164329c0cef8e0c69685c9ce7cfbb4" Oct 30 18:35:59.404987 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.404925 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b402b775fb6c429f5ac017e702b505cfa164329c0cef8e0c69685c9ce7cfbb4"} err="failed to get container status \"9b402b775fb6c429f5ac017e702b505cfa164329c0cef8e0c69685c9ce7cfbb4\": rpc error: code = NotFound desc = could not find container \"9b402b775fb6c429f5ac017e702b505cfa164329c0cef8e0c69685c9ce7cfbb4\": container with ID starting with 9b402b775fb6c429f5ac017e702b505cfa164329c0cef8e0c69685c9ce7cfbb4 not found: ID does not exist" Oct 30 18:35:59.404987 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.404946 4442 scope.go:122] "RemoveContainer" containerID="49ce8ab2a48c1face88a013012ca4aa2d43b32be4f07aaa85b067d1eb2c11664" Oct 30 18:35:59.417125 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.417103 4442 scope.go:122] "RemoveContainer" containerID="49ce8ab2a48c1face88a013012ca4aa2d43b32be4f07aaa85b067d1eb2c11664" Oct 30 18:35:59.417975 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:35:59.417945 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49ce8ab2a48c1face88a013012ca4aa2d43b32be4f07aaa85b067d1eb2c11664\": container with ID starting with 49ce8ab2a48c1face88a013012ca4aa2d43b32be4f07aaa85b067d1eb2c11664 not found: ID does not exist" containerID="49ce8ab2a48c1face88a013012ca4aa2d43b32be4f07aaa85b067d1eb2c11664" Oct 30 18:35:59.418058 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.417983 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49ce8ab2a48c1face88a013012ca4aa2d43b32be4f07aaa85b067d1eb2c11664"} err="failed to get container status \"49ce8ab2a48c1face88a013012ca4aa2d43b32be4f07aaa85b067d1eb2c11664\": rpc error: code = NotFound desc = could not find container \"49ce8ab2a48c1face88a013012ca4aa2d43b32be4f07aaa85b067d1eb2c11664\": container with ID starting with 49ce8ab2a48c1face88a013012ca4aa2d43b32be4f07aaa85b067d1eb2c11664 not found: ID does not exist" Oct 30 18:35:59.418058 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.418009 4442 scope.go:122] "RemoveContainer" containerID="efb047674294cc3b828900791f861bb68cc61db6bcb6ca5da05d8bc7469bfad5" Oct 30 18:35:59.429453 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.429427 4442 scope.go:122] "RemoveContainer" containerID="efb047674294cc3b828900791f861bb68cc61db6bcb6ca5da05d8bc7469bfad5" Oct 30 18:35:59.429814 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:35:59.429783 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efb047674294cc3b828900791f861bb68cc61db6bcb6ca5da05d8bc7469bfad5\": container with ID starting with efb047674294cc3b828900791f861bb68cc61db6bcb6ca5da05d8bc7469bfad5 not found: ID does not exist" containerID="efb047674294cc3b828900791f861bb68cc61db6bcb6ca5da05d8bc7469bfad5" Oct 30 18:35:59.429887 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.429827 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efb047674294cc3b828900791f861bb68cc61db6bcb6ca5da05d8bc7469bfad5"} err="failed to get container status \"efb047674294cc3b828900791f861bb68cc61db6bcb6ca5da05d8bc7469bfad5\": rpc error: code = NotFound desc = could not find container \"efb047674294cc3b828900791f861bb68cc61db6bcb6ca5da05d8bc7469bfad5\": container with ID starting with efb047674294cc3b828900791f861bb68cc61db6bcb6ca5da05d8bc7469bfad5 not found: ID does not exist" Oct 30 18:35:59.429887 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.429852 4442 scope.go:122] "RemoveContainer" containerID="c4fdb8c7bbdcb0df1592046e6b68a79ef6cb462bf943db092005658b54eaa50d" Oct 30 18:35:59.440984 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.440953 4442 scope.go:122] "RemoveContainer" containerID="c4fdb8c7bbdcb0df1592046e6b68a79ef6cb462bf943db092005658b54eaa50d" Oct 30 18:35:59.441336 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:35:59.441313 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4fdb8c7bbdcb0df1592046e6b68a79ef6cb462bf943db092005658b54eaa50d\": container with ID starting with c4fdb8c7bbdcb0df1592046e6b68a79ef6cb462bf943db092005658b54eaa50d not found: ID does not exist" containerID="c4fdb8c7bbdcb0df1592046e6b68a79ef6cb462bf943db092005658b54eaa50d" Oct 30 18:35:59.441424 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.441347 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4fdb8c7bbdcb0df1592046e6b68a79ef6cb462bf943db092005658b54eaa50d"} err="failed to get container status \"c4fdb8c7bbdcb0df1592046e6b68a79ef6cb462bf943db092005658b54eaa50d\": rpc error: code = NotFound desc = could not find container \"c4fdb8c7bbdcb0df1592046e6b68a79ef6cb462bf943db092005658b54eaa50d\": container with ID starting with c4fdb8c7bbdcb0df1592046e6b68a79ef6cb462bf943db092005658b54eaa50d not found: ID does not exist" Oct 30 18:35:59.474161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.474069 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl"] Oct 30 18:35:59.478492 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.478471 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-bv2pl"] Oct 30 18:35:59.510417 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.510391 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn"] Oct 30 18:35:59.515229 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.515205 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rkjpn"] Oct 30 18:35:59.549284 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.549258 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj"] Oct 30 18:35:59.555612 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.555590 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-z9dpj"] Oct 30 18:35:59.582620 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.582591 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw"] Oct 30 18:35:59.587835 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:35:59.587814 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-tk9qw"] Oct 30 18:36:01.071461 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:01.071421 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80927367-fb50-4945-86d8-9d22cf64f0ef" path="/var/lib/kubelet/pods/80927367-fb50-4945-86d8-9d22cf64f0ef/volumes" Oct 30 18:36:01.072206 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:01.072186 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83443c95-ada0-4803-beec-b66c915f5e50" path="/var/lib/kubelet/pods/83443c95-ada0-4803-beec-b66c915f5e50/volumes" Oct 30 18:36:01.072869 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:01.072841 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9760b73c-522d-4255-b72b-c0786527ded6" path="/var/lib/kubelet/pods/9760b73c-522d-4255-b72b-c0786527ded6/volumes" Oct 30 18:36:01.073618 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:01.073599 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a122a97d-09d1-485a-902c-2f7832deab2c" path="/var/lib/kubelet/pods/a122a97d-09d1-485a-902c-2f7832deab2c/volumes" Oct 30 18:36:04.248347 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:04.248309 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-160bf8-857d864c68-p4svr"] Oct 30 18:36:04.279421 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:04.279388 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-160bf8-74f657fbf9-kfw2h"] Oct 30 18:36:09.280640 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:09.280593 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-160bf8-74f657fbf9-kfw2h" podUID="84e058c7-2d92-46d5-bcf7-4a061945921e" containerName="payload-pre-processing" containerID="cri-o://092f5e5ef486f99032aa2595925838ecafd7fa127dd8c05b4eeb1949652571a2" gracePeriod=25 Oct 30 18:36:09.470483 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:09.470365 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-160bf8-857d864c68-p4svr" podUID="355f5946-71a8-4a78-937c-62d47ebf9e3f" containerName="payload-processing" containerID="cri-o://599298097cdd56fde3ae185ebd69a388a31959323f99ee48c333390aa21d6a23" gracePeriod=25 Oct 30 18:36:09.513880 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:09.513841 4442 generic.go:356] "Generic (PLEG): container finished" podID="84e058c7-2d92-46d5-bcf7-4a061945921e" containerID="092f5e5ef486f99032aa2595925838ecafd7fa127dd8c05b4eeb1949652571a2" exitCode=0 Oct 30 18:36:09.514025 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:09.513924 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-160bf8-74f657fbf9-kfw2h" event={"ID":"84e058c7-2d92-46d5-bcf7-4a061945921e","Type":"ContainerDied","Data":"092f5e5ef486f99032aa2595925838ecafd7fa127dd8c05b4eeb1949652571a2"} Oct 30 18:36:09.610316 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:09.610282 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-160bf8-74f657fbf9-kfw2h" Oct 30 18:36:09.777818 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:09.777785 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84e058c7-2d92-46d5-bcf7-4a061945921e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e058c7-2d92-46d5-bcf7-4a061945921e-plugins-config-volume\") pod \"84e058c7-2d92-46d5-bcf7-4a061945921e\" (UID: \"84e058c7-2d92-46d5-bcf7-4a061945921e\") " Oct 30 18:36:09.778271 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:09.778194 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84e058c7-2d92-46d5-bcf7-4a061945921e-kube-api-access-64g7r\" (UniqueName: \"kubernetes.io/projected/84e058c7-2d92-46d5-bcf7-4a061945921e-kube-api-access-64g7r\") pod \"84e058c7-2d92-46d5-bcf7-4a061945921e\" (UID: \"84e058c7-2d92-46d5-bcf7-4a061945921e\") " Oct 30 18:36:09.778271 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:09.778203 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84e058c7-2d92-46d5-bcf7-4a061945921e-plugins-config-volume" pod "84e058c7-2d92-46d5-bcf7-4a061945921e" (UID: "84e058c7-2d92-46d5-bcf7-4a061945921e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:36:09.778842 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:09.778823 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e058c7-2d92-46d5-bcf7-4a061945921e-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:09.780560 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:09.780529 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84e058c7-2d92-46d5-bcf7-4a061945921e-kube-api-access-64g7r" pod "84e058c7-2d92-46d5-bcf7-4a061945921e" (UID: "84e058c7-2d92-46d5-bcf7-4a061945921e"). InnerVolumeSpecName "kube-api-access-64g7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:09.800664 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:09.800636 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-160bf8-857d864c68-p4svr" Oct 30 18:36:09.879677 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:09.879645 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/355f5946-71a8-4a78-937c-62d47ebf9e3f-kube-api-access-nnsq9\" (UniqueName: \"kubernetes.io/projected/355f5946-71a8-4a78-937c-62d47ebf9e3f-kube-api-access-nnsq9\") pod \"355f5946-71a8-4a78-937c-62d47ebf9e3f\" (UID: \"355f5946-71a8-4a78-937c-62d47ebf9e3f\") " Oct 30 18:36:09.880001 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:09.879978 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/355f5946-71a8-4a78-937c-62d47ebf9e3f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/355f5946-71a8-4a78-937c-62d47ebf9e3f-plugins-config-volume\") pod \"355f5946-71a8-4a78-937c-62d47ebf9e3f\" (UID: \"355f5946-71a8-4a78-937c-62d47ebf9e3f\") " Oct 30 18:36:09.880921 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:09.880893 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64g7r\" (UniqueName: \"kubernetes.io/projected/84e058c7-2d92-46d5-bcf7-4a061945921e-kube-api-access-64g7r\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:09.881401 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:09.881375 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/355f5946-71a8-4a78-937c-62d47ebf9e3f-plugins-config-volume" pod "355f5946-71a8-4a78-937c-62d47ebf9e3f" (UID: "355f5946-71a8-4a78-937c-62d47ebf9e3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:36:09.884083 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:09.884056 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/355f5946-71a8-4a78-937c-62d47ebf9e3f-kube-api-access-nnsq9" pod "355f5946-71a8-4a78-937c-62d47ebf9e3f" (UID: "355f5946-71a8-4a78-937c-62d47ebf9e3f"). InnerVolumeSpecName "kube-api-access-nnsq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:09.982596 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:09.982560 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnsq9\" (UniqueName: \"kubernetes.io/projected/355f5946-71a8-4a78-937c-62d47ebf9e3f-kube-api-access-nnsq9\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:09.982596 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:09.982589 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/355f5946-71a8-4a78-937c-62d47ebf9e3f-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:10.552218 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:10.549826 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-160bf8-74f657fbf9-kfw2h" event={"ID":"84e058c7-2d92-46d5-bcf7-4a061945921e","Type":"ContainerDied","Data":"3114191e4a6f46ab0cc474107f878c6d8e37386f196726530427ebb90a564b0a"} Oct 30 18:36:10.552218 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:10.549930 4442 scope.go:122] "RemoveContainer" containerID="092f5e5ef486f99032aa2595925838ecafd7fa127dd8c05b4eeb1949652571a2" Oct 30 18:36:10.552218 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:10.551234 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-160bf8-74f657fbf9-kfw2h" Oct 30 18:36:10.586079 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:10.586041 4442 generic.go:356] "Generic (PLEG): container finished" podID="355f5946-71a8-4a78-937c-62d47ebf9e3f" containerID="599298097cdd56fde3ae185ebd69a388a31959323f99ee48c333390aa21d6a23" exitCode=0 Oct 30 18:36:10.590519 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:10.588897 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-160bf8-857d864c68-p4svr" event={"ID":"355f5946-71a8-4a78-937c-62d47ebf9e3f","Type":"ContainerDied","Data":"599298097cdd56fde3ae185ebd69a388a31959323f99ee48c333390aa21d6a23"} Oct 30 18:36:10.590697 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:10.590561 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-160bf8-857d864c68-p4svr" event={"ID":"355f5946-71a8-4a78-937c-62d47ebf9e3f","Type":"ContainerDied","Data":"6c2fba6c7d8f6c2c9033f66049a9aa0057d5237f99ce7c243491839427be9923"} Oct 30 18:36:10.590697 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:10.589002 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-160bf8-857d864c68-p4svr" Oct 30 18:36:10.598760 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:10.598732 4442 scope.go:122] "RemoveContainer" containerID="599298097cdd56fde3ae185ebd69a388a31959323f99ee48c333390aa21d6a23" Oct 30 18:36:10.616538 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:10.615679 4442 scope.go:122] "RemoveContainer" containerID="599298097cdd56fde3ae185ebd69a388a31959323f99ee48c333390aa21d6a23" Oct 30 18:36:10.616538 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:36:10.616126 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"599298097cdd56fde3ae185ebd69a388a31959323f99ee48c333390aa21d6a23\": container with ID starting with 599298097cdd56fde3ae185ebd69a388a31959323f99ee48c333390aa21d6a23 not found: ID does not exist" containerID="599298097cdd56fde3ae185ebd69a388a31959323f99ee48c333390aa21d6a23" Oct 30 18:36:10.616538 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:10.616237 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"599298097cdd56fde3ae185ebd69a388a31959323f99ee48c333390aa21d6a23"} err="failed to get container status \"599298097cdd56fde3ae185ebd69a388a31959323f99ee48c333390aa21d6a23\": rpc error: code = NotFound desc = could not find container \"599298097cdd56fde3ae185ebd69a388a31959323f99ee48c333390aa21d6a23\": container with ID starting with 599298097cdd56fde3ae185ebd69a388a31959323f99ee48c333390aa21d6a23 not found: ID does not exist" Oct 30 18:36:10.645300 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:10.645273 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-160bf8-74f657fbf9-kfw2h"] Oct 30 18:36:10.656845 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:10.656813 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-160bf8-74f657fbf9-kfw2h"] Oct 30 18:36:10.680119 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:10.680081 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-160bf8-857d864c68-p4svr"] Oct 30 18:36:10.688125 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:10.688099 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-160bf8-857d864c68-p4svr"] Oct 30 18:36:11.067998 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:11.067967 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="355f5946-71a8-4a78-937c-62d47ebf9e3f" path="/var/lib/kubelet/pods/355f5946-71a8-4a78-937c-62d47ebf9e3f/volumes" Oct 30 18:36:11.068591 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:11.068570 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84e058c7-2d92-46d5-bcf7-4a061945921e" path="/var/lib/kubelet/pods/84e058c7-2d92-46d5-bcf7-4a061945921e/volumes" Oct 30 18:36:17.787643 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:36:17.787602 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:36:18.112680 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:18.112596 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-160bf8-dthqr"] Oct 30 18:36:18.124059 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:18.124031 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-160bf8-dthqr"] Oct 30 18:36:19.069062 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:19.069018 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f35562b-b5c5-469e-b445-86a5ece915bb" path="/var/lib/kubelet/pods/8f35562b-b5c5-469e-b445-86a5ece915bb/volumes" Oct 30 18:36:23.460928 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:23.460891 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d"] Oct 30 18:36:23.461395 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:23.461155 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" podUID="f4e00bfc-fc1d-46ce-946d-969438944b7e" containerName="istio-proxy" containerID="cri-o://e613121c7326daf440ea98122f0e0b43d85fb440d0946585ad4e7f3ea4b19e97" gracePeriod=30 Oct 30 18:36:23.472739 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:23.472702 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74"] Oct 30 18:36:23.473029 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:23.472985 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" podUID="a99c9a23-d65d-4950-a153-71c8decd10aa" containerName="istio-proxy" containerID="cri-o://b2dfe6d37d9b6174fec9511a906965354a94f74e5e90d431e1958e6971c118b5" gracePeriod=30 Oct 30 18:36:23.477519 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:23.477494 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n"] Oct 30 18:36:23.477695 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:23.477676 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" podUID="7084397f-85eb-43f9-8314-530af31c81c8" containerName="istio-proxy" containerID="cri-o://cdbdf5c245588d68a0c29b38270a6b0f6c182f8b99256f6a7ed6a8512dedbc57" gracePeriod=30 Oct 30 18:36:23.486023 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:23.485990 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww"] Oct 30 18:36:23.486301 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:23.486255 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" podUID="ffc6ceba-80a2-4722-a064-d409fe90fde4" containerName="istio-proxy" containerID="cri-o://d84f0e5b32a1944e7fd2cb71934548e8c0524a97ef7823dcec34cd9299253a12" gracePeriod=30 Oct 30 18:36:25.107936 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:25.107899 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77"] Oct 30 18:36:25.109371 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:25.109334 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" podUID="6bbe6b88-960d-47db-a270-47fc4cee7f44" containerName="istio-proxy" containerID="cri-o://870d83262330ab1628bebb565ff996a55a75ecba3831a6ce2e6b6d5c310682af" gracePeriod=30 Oct 30 18:36:25.703268 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:25.703234 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7616-ba9af8"] Oct 30 18:36:25.703924 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:25.703908 4442 cpu_manager.go:414] "containerMap: removing container" podUID="a122a97d-09d1-485a-902c-2f7832deab2c" containerName="istio-proxy" Oct 30 18:36:25.703973 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:25.703927 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a122a97d-09d1-485a-902c-2f7832deab2c" containerName="istio-proxy" Oct 30 18:36:25.703973 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:25.703940 4442 cpu_manager.go:414] "containerMap: removing container" podUID="9760b73c-522d-4255-b72b-c0786527ded6" containerName="istio-proxy" Oct 30 18:36:25.703973 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:25.703944 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9760b73c-522d-4255-b72b-c0786527ded6" containerName="istio-proxy" Oct 30 18:36:25.703973 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:25.703959 4442 cpu_manager.go:414] "containerMap: removing container" podUID="80927367-fb50-4945-86d8-9d22cf64f0ef" containerName="istio-proxy" Oct 30 18:36:25.703973 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:25.703964 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80927367-fb50-4945-86d8-9d22cf64f0ef" containerName="istio-proxy" Oct 30 18:36:25.703973 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:25.703970 4442 cpu_manager.go:414] "containerMap: removing container" podUID="8f35562b-b5c5-469e-b445-86a5ece915bb" containerName="revoke-keys" Oct 30 18:36:25.704186 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:25.703976 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f35562b-b5c5-469e-b445-86a5ece915bb" containerName="revoke-keys" Oct 30 18:36:25.704186 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:25.703986 4442 cpu_manager.go:414] "containerMap: removing container" podUID="355f5946-71a8-4a78-937c-62d47ebf9e3f" containerName="payload-processing" Oct 30 18:36:25.704186 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:25.703990 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="355f5946-71a8-4a78-937c-62d47ebf9e3f" containerName="payload-processing" Oct 30 18:36:25.704186 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:25.704000 4442 cpu_manager.go:414] "containerMap: removing container" podUID="83443c95-ada0-4803-beec-b66c915f5e50" containerName="istio-proxy" Oct 30 18:36:25.704186 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:25.704005 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83443c95-ada0-4803-beec-b66c915f5e50" containerName="istio-proxy" Oct 30 18:36:25.704186 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:25.704041 4442 cpu_manager.go:414] "containerMap: removing container" podUID="84e058c7-2d92-46d5-bcf7-4a061945921e" containerName="payload-pre-processing" Oct 30 18:36:25.704186 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:25.704047 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84e058c7-2d92-46d5-bcf7-4a061945921e" containerName="payload-pre-processing" Oct 30 18:36:25.704186 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:25.704162 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="a122a97d-09d1-485a-902c-2f7832deab2c" containerName="istio-proxy" Oct 30 18:36:25.704186 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:25.704172 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="9760b73c-522d-4255-b72b-c0786527ded6" containerName="istio-proxy" Oct 30 18:36:25.704186 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:25.704180 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="80927367-fb50-4945-86d8-9d22cf64f0ef" containerName="istio-proxy" Oct 30 18:36:25.704186 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:25.704185 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f35562b-b5c5-469e-b445-86a5ece915bb" containerName="revoke-keys" Oct 30 18:36:25.704186 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:25.704191 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="355f5946-71a8-4a78-937c-62d47ebf9e3f" containerName="payload-processing" Oct 30 18:36:25.704551 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:25.704197 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="83443c95-ada0-4803-beec-b66c915f5e50" containerName="istio-proxy" Oct 30 18:36:25.704551 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:25.704206 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="84e058c7-2d92-46d5-bcf7-4a061945921e" containerName="payload-pre-processing" Oct 30 18:36:25.708036 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:25.708016 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7616-ba9af8" Oct 30 18:36:25.722810 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:25.722781 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-r2spd\"" Oct 30 18:36:25.724028 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:25.724003 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7616-ba9af8"] Oct 30 18:36:25.851916 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:25.851882 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srzd6\" (UniqueName: \"kubernetes.io/projected/645f6dd5-194f-4b50-a31a-36dad375b5df-kube-api-access-srzd6\") pod \"test-curl-7616-ba9af8\" (UID: \"645f6dd5-194f-4b50-a31a-36dad375b5df\") " pod="opendatahub/test-curl-7616-ba9af8" Oct 30 18:36:25.954293 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:25.954188 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srzd6\" (UniqueName: \"kubernetes.io/projected/645f6dd5-194f-4b50-a31a-36dad375b5df-kube-api-access-srzd6\") pod \"test-curl-7616-ba9af8\" (UID: \"645f6dd5-194f-4b50-a31a-36dad375b5df\") " pod="opendatahub/test-curl-7616-ba9af8" Oct 30 18:36:25.973253 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:25.973227 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srzd6\" (UniqueName: \"kubernetes.io/projected/645f6dd5-194f-4b50-a31a-36dad375b5df-kube-api-access-srzd6\") pod \"test-curl-7616-ba9af8\" (UID: \"645f6dd5-194f-4b50-a31a-36dad375b5df\") " pod="opendatahub/test-curl-7616-ba9af8" Oct 30 18:36:26.022419 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:26.022387 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7616-ba9af8" Oct 30 18:36:26.336934 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:36:26.336900 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod645f6dd5_194f_4b50_a31a_36dad375b5df.slice/crio-4d265a565692ef98d5a8a211293ab375d9a3eda41f69584a3fa38f43ad69db97 WatchSource:0}: Error finding container 4d265a565692ef98d5a8a211293ab375d9a3eda41f69584a3fa38f43ad69db97: Status 404 returned error can't find the container with id 4d265a565692ef98d5a8a211293ab375d9a3eda41f69584a3fa38f43ad69db97 Oct 30 18:36:26.337970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:26.337938 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7616-ba9af8"] Oct 30 18:36:26.338981 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:26.338964 4442 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:36:26.773530 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:26.773352 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7616-ba9af8" event={"ID":"645f6dd5-194f-4b50-a31a-36dad375b5df","Type":"ContainerStarted","Data":"4d265a565692ef98d5a8a211293ab375d9a3eda41f69584a3fa38f43ad69db97"} Oct 30 18:36:28.764316 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:28.749905 4442 patch_prober.go:29] interesting pod/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:36:28.764316 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:28.749966 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" podUID="6bbe6b88-960d-47db-a270-47fc4cee7f44" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:36:28.820447 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:28.820410 4442 generic.go:356] "Generic (PLEG): container finished" podID="ffc6ceba-80a2-4722-a064-d409fe90fde4" containerID="d84f0e5b32a1944e7fd2cb71934548e8c0524a97ef7823dcec34cd9299253a12" exitCode=0 Oct 30 18:36:28.820624 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:28.820575 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" event={"ID":"ffc6ceba-80a2-4722-a064-d409fe90fde4","Type":"ContainerDied","Data":"d84f0e5b32a1944e7fd2cb71934548e8c0524a97ef7823dcec34cd9299253a12"} Oct 30 18:36:28.823110 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:28.823082 4442 generic.go:356] "Generic (PLEG): container finished" podID="a99c9a23-d65d-4950-a153-71c8decd10aa" containerID="b2dfe6d37d9b6174fec9511a906965354a94f74e5e90d431e1958e6971c118b5" exitCode=0 Oct 30 18:36:28.823279 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:28.823217 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" event={"ID":"a99c9a23-d65d-4950-a153-71c8decd10aa","Type":"ContainerDied","Data":"b2dfe6d37d9b6174fec9511a906965354a94f74e5e90d431e1958e6971c118b5"} Oct 30 18:36:28.826961 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:28.826934 4442 generic.go:356] "Generic (PLEG): container finished" podID="f4e00bfc-fc1d-46ce-946d-969438944b7e" containerID="e613121c7326daf440ea98122f0e0b43d85fb440d0946585ad4e7f3ea4b19e97" exitCode=0 Oct 30 18:36:28.827116 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:28.827096 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" event={"ID":"f4e00bfc-fc1d-46ce-946d-969438944b7e","Type":"ContainerDied","Data":"e613121c7326daf440ea98122f0e0b43d85fb440d0946585ad4e7f3ea4b19e97"} Oct 30 18:36:28.829305 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:28.829245 4442 generic.go:356] "Generic (PLEG): container finished" podID="7084397f-85eb-43f9-8314-530af31c81c8" containerID="cdbdf5c245588d68a0c29b38270a6b0f6c182f8b99256f6a7ed6a8512dedbc57" exitCode=0 Oct 30 18:36:28.829403 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:28.829371 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" event={"ID":"7084397f-85eb-43f9-8314-530af31c81c8","Type":"ContainerDied","Data":"cdbdf5c245588d68a0c29b38270a6b0f6c182f8b99256f6a7ed6a8512dedbc57"} Oct 30 18:36:28.895859 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:28.891391 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" Oct 30 18:36:28.984735 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:28.984706 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" Oct 30 18:36:28.993472 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:28.993454 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" Oct 30 18:36:29.013632 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.013607 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" Oct 30 18:36:29.024652 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.024546 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7084397f-85eb-43f9-8314-530af31c81c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7084397f-85eb-43f9-8314-530af31c81c8-workload-socket\") pod \"7084397f-85eb-43f9-8314-530af31c81c8\" (UID: \"7084397f-85eb-43f9-8314-530af31c81c8\") " Oct 30 18:36:29.025657 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.025634 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a99c9a23-d65d-4950-a153-71c8decd10aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a99c9a23-d65d-4950-a153-71c8decd10aa-istiod-ca-cert\") pod \"a99c9a23-d65d-4950-a153-71c8decd10aa\" (UID: \"a99c9a23-d65d-4950-a153-71c8decd10aa\") " Oct 30 18:36:29.026175 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.026115 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4e00bfc-fc1d-46ce-946d-969438944b7e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4e00bfc-fc1d-46ce-946d-969438944b7e-istio-data\") pod \"f4e00bfc-fc1d-46ce-946d-969438944b7e\" (UID: \"f4e00bfc-fc1d-46ce-946d-969438944b7e\") " Oct 30 18:36:29.029182 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.028234 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7084397f-85eb-43f9-8314-530af31c81c8-istio-token\" (UniqueName: \"kubernetes.io/projected/7084397f-85eb-43f9-8314-530af31c81c8-istio-token\") pod \"7084397f-85eb-43f9-8314-530af31c81c8\" (UID: \"7084397f-85eb-43f9-8314-530af31c81c8\") " Oct 30 18:36:29.029182 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.028277 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7084397f-85eb-43f9-8314-530af31c81c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7084397f-85eb-43f9-8314-530af31c81c8-istio-envoy\") pod \"7084397f-85eb-43f9-8314-530af31c81c8\" (UID: \"7084397f-85eb-43f9-8314-530af31c81c8\") " Oct 30 18:36:29.029182 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.028307 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a99c9a23-d65d-4950-a153-71c8decd10aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a99c9a23-d65d-4950-a153-71c8decd10aa-credential-socket\") pod \"a99c9a23-d65d-4950-a153-71c8decd10aa\" (UID: \"a99c9a23-d65d-4950-a153-71c8decd10aa\") " Oct 30 18:36:29.029182 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.028346 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffc6ceba-80a2-4722-a064-d409fe90fde4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffc6ceba-80a2-4722-a064-d409fe90fde4-istiod-ca-cert\") pod \"ffc6ceba-80a2-4722-a064-d409fe90fde4\" (UID: \"ffc6ceba-80a2-4722-a064-d409fe90fde4\") " Oct 30 18:36:29.029182 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.028380 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffc6ceba-80a2-4722-a064-d409fe90fde4-istio-token\" (UniqueName: \"kubernetes.io/projected/ffc6ceba-80a2-4722-a064-d409fe90fde4-istio-token\") pod \"ffc6ceba-80a2-4722-a064-d409fe90fde4\" (UID: \"ffc6ceba-80a2-4722-a064-d409fe90fde4\") " Oct 30 18:36:29.029182 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.028428 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4e00bfc-fc1d-46ce-946d-969438944b7e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4e00bfc-fc1d-46ce-946d-969438944b7e-istio-envoy\") pod \"f4e00bfc-fc1d-46ce-946d-969438944b7e\" (UID: \"f4e00bfc-fc1d-46ce-946d-969438944b7e\") " Oct 30 18:36:29.029182 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.028465 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a99c9a23-d65d-4950-a153-71c8decd10aa-istio-token\" (UniqueName: \"kubernetes.io/projected/a99c9a23-d65d-4950-a153-71c8decd10aa-istio-token\") pod \"a99c9a23-d65d-4950-a153-71c8decd10aa\" (UID: \"a99c9a23-d65d-4950-a153-71c8decd10aa\") " Oct 30 18:36:29.029182 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.028502 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7084397f-85eb-43f9-8314-530af31c81c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7084397f-85eb-43f9-8314-530af31c81c8-istiod-ca-cert\") pod \"7084397f-85eb-43f9-8314-530af31c81c8\" (UID: \"7084397f-85eb-43f9-8314-530af31c81c8\") " Oct 30 18:36:29.029182 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.028555 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffc6ceba-80a2-4722-a064-d409fe90fde4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc6ceba-80a2-4722-a064-d409fe90fde4-workload-socket\") pod \"ffc6ceba-80a2-4722-a064-d409fe90fde4\" (UID: \"ffc6ceba-80a2-4722-a064-d409fe90fde4\") " Oct 30 18:36:29.029182 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.028586 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a99c9a23-d65d-4950-a153-71c8decd10aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a99c9a23-d65d-4950-a153-71c8decd10aa-istio-data\") pod \"a99c9a23-d65d-4950-a153-71c8decd10aa\" (UID: \"a99c9a23-d65d-4950-a153-71c8decd10aa\") " Oct 30 18:36:29.029182 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.028836 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a99c9a23-d65d-4950-a153-71c8decd10aa-kube-api-access-4kx95\" (UniqueName: \"kubernetes.io/projected/a99c9a23-d65d-4950-a153-71c8decd10aa-kube-api-access-4kx95\") pod \"a99c9a23-d65d-4950-a153-71c8decd10aa\" (UID: \"a99c9a23-d65d-4950-a153-71c8decd10aa\") " Oct 30 18:36:29.029890 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.029463 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4e00bfc-fc1d-46ce-946d-969438944b7e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4e00bfc-fc1d-46ce-946d-969438944b7e-credential-socket\") pod \"f4e00bfc-fc1d-46ce-946d-969438944b7e\" (UID: \"f4e00bfc-fc1d-46ce-946d-969438944b7e\") " Oct 30 18:36:29.029890 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.029506 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffc6ceba-80a2-4722-a064-d409fe90fde4-kube-api-access-hdckw\" (UniqueName: \"kubernetes.io/projected/ffc6ceba-80a2-4722-a064-d409fe90fde4-kube-api-access-hdckw\") pod \"ffc6ceba-80a2-4722-a064-d409fe90fde4\" (UID: \"ffc6ceba-80a2-4722-a064-d409fe90fde4\") " Oct 30 18:36:29.029890 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.029545 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffc6ceba-80a2-4722-a064-d409fe90fde4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffc6ceba-80a2-4722-a064-d409fe90fde4-istio-envoy\") pod \"ffc6ceba-80a2-4722-a064-d409fe90fde4\" (UID: \"ffc6ceba-80a2-4722-a064-d409fe90fde4\") " Oct 30 18:36:29.029890 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.029610 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a99c9a23-d65d-4950-a153-71c8decd10aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a99c9a23-d65d-4950-a153-71c8decd10aa-istio-envoy\") pod \"a99c9a23-d65d-4950-a153-71c8decd10aa\" (UID: \"a99c9a23-d65d-4950-a153-71c8decd10aa\") " Oct 30 18:36:29.029890 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.029645 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffc6ceba-80a2-4722-a064-d409fe90fde4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc6ceba-80a2-4722-a064-d409fe90fde4-credential-socket\") pod \"ffc6ceba-80a2-4722-a064-d409fe90fde4\" (UID: \"ffc6ceba-80a2-4722-a064-d409fe90fde4\") " Oct 30 18:36:29.029890 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.029677 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4e00bfc-fc1d-46ce-946d-969438944b7e-kube-api-access-tw4fw\" (UniqueName: \"kubernetes.io/projected/f4e00bfc-fc1d-46ce-946d-969438944b7e-kube-api-access-tw4fw\") pod \"f4e00bfc-fc1d-46ce-946d-969438944b7e\" (UID: \"f4e00bfc-fc1d-46ce-946d-969438944b7e\") " Oct 30 18:36:29.029890 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.029716 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7084397f-85eb-43f9-8314-530af31c81c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7084397f-85eb-43f9-8314-530af31c81c8-istio-podinfo\") pod \"7084397f-85eb-43f9-8314-530af31c81c8\" (UID: \"7084397f-85eb-43f9-8314-530af31c81c8\") " Oct 30 18:36:29.029890 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.029757 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffc6ceba-80a2-4722-a064-d409fe90fde4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffc6ceba-80a2-4722-a064-d409fe90fde4-workload-certs\") pod \"ffc6ceba-80a2-4722-a064-d409fe90fde4\" (UID: \"ffc6ceba-80a2-4722-a064-d409fe90fde4\") " Oct 30 18:36:29.029890 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.029833 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4e00bfc-fc1d-46ce-946d-969438944b7e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4e00bfc-fc1d-46ce-946d-969438944b7e-istiod-ca-cert\") pod \"f4e00bfc-fc1d-46ce-946d-969438944b7e\" (UID: \"f4e00bfc-fc1d-46ce-946d-969438944b7e\") " Oct 30 18:36:29.030395 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.029912 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4e00bfc-fc1d-46ce-946d-969438944b7e-istio-token\" (UniqueName: \"kubernetes.io/projected/f4e00bfc-fc1d-46ce-946d-969438944b7e-istio-token\") pod \"f4e00bfc-fc1d-46ce-946d-969438944b7e\" (UID: \"f4e00bfc-fc1d-46ce-946d-969438944b7e\") " Oct 30 18:36:29.030395 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.029958 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7084397f-85eb-43f9-8314-530af31c81c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7084397f-85eb-43f9-8314-530af31c81c8-workload-certs\") pod \"7084397f-85eb-43f9-8314-530af31c81c8\" (UID: \"7084397f-85eb-43f9-8314-530af31c81c8\") " Oct 30 18:36:29.030395 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.029987 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4e00bfc-fc1d-46ce-946d-969438944b7e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4e00bfc-fc1d-46ce-946d-969438944b7e-workload-certs\") pod \"f4e00bfc-fc1d-46ce-946d-969438944b7e\" (UID: \"f4e00bfc-fc1d-46ce-946d-969438944b7e\") " Oct 30 18:36:29.030395 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.030026 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a99c9a23-d65d-4950-a153-71c8decd10aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a99c9a23-d65d-4950-a153-71c8decd10aa-istio-podinfo\") pod \"a99c9a23-d65d-4950-a153-71c8decd10aa\" (UID: \"a99c9a23-d65d-4950-a153-71c8decd10aa\") " Oct 30 18:36:29.030395 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.030061 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a99c9a23-d65d-4950-a153-71c8decd10aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a99c9a23-d65d-4950-a153-71c8decd10aa-workload-certs\") pod \"a99c9a23-d65d-4950-a153-71c8decd10aa\" (UID: \"a99c9a23-d65d-4950-a153-71c8decd10aa\") " Oct 30 18:36:29.030395 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.030120 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7084397f-85eb-43f9-8314-530af31c81c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7084397f-85eb-43f9-8314-530af31c81c8-credential-socket\") pod \"7084397f-85eb-43f9-8314-530af31c81c8\" (UID: \"7084397f-85eb-43f9-8314-530af31c81c8\") " Oct 30 18:36:29.030395 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.030195 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4e00bfc-fc1d-46ce-946d-969438944b7e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4e00bfc-fc1d-46ce-946d-969438944b7e-workload-socket\") pod \"f4e00bfc-fc1d-46ce-946d-969438944b7e\" (UID: \"f4e00bfc-fc1d-46ce-946d-969438944b7e\") " Oct 30 18:36:29.030395 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.030291 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7084397f-85eb-43f9-8314-530af31c81c8-kube-api-access-9w55w\" (UniqueName: \"kubernetes.io/projected/7084397f-85eb-43f9-8314-530af31c81c8-kube-api-access-9w55w\") pod \"7084397f-85eb-43f9-8314-530af31c81c8\" (UID: \"7084397f-85eb-43f9-8314-530af31c81c8\") " Oct 30 18:36:29.030395 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.030361 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffc6ceba-80a2-4722-a064-d409fe90fde4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffc6ceba-80a2-4722-a064-d409fe90fde4-istio-data\") pod \"ffc6ceba-80a2-4722-a064-d409fe90fde4\" (UID: \"ffc6ceba-80a2-4722-a064-d409fe90fde4\") " Oct 30 18:36:29.030850 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.030399 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a99c9a23-d65d-4950-a153-71c8decd10aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a99c9a23-d65d-4950-a153-71c8decd10aa-workload-socket\") pod \"a99c9a23-d65d-4950-a153-71c8decd10aa\" (UID: \"a99c9a23-d65d-4950-a153-71c8decd10aa\") " Oct 30 18:36:29.030850 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.030428 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4e00bfc-fc1d-46ce-946d-969438944b7e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4e00bfc-fc1d-46ce-946d-969438944b7e-istio-podinfo\") pod \"f4e00bfc-fc1d-46ce-946d-969438944b7e\" (UID: \"f4e00bfc-fc1d-46ce-946d-969438944b7e\") " Oct 30 18:36:29.030850 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.030490 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ffc6ceba-80a2-4722-a064-d409fe90fde4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffc6ceba-80a2-4722-a064-d409fe90fde4-istio-podinfo\") pod \"ffc6ceba-80a2-4722-a064-d409fe90fde4\" (UID: \"ffc6ceba-80a2-4722-a064-d409fe90fde4\") " Oct 30 18:36:29.030850 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.030527 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7084397f-85eb-43f9-8314-530af31c81c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7084397f-85eb-43f9-8314-530af31c81c8-istio-data\") pod \"7084397f-85eb-43f9-8314-530af31c81c8\" (UID: \"7084397f-85eb-43f9-8314-530af31c81c8\") " Oct 30 18:36:29.036251 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.025561 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7084397f-85eb-43f9-8314-530af31c81c8-workload-socket" pod "7084397f-85eb-43f9-8314-530af31c81c8" (UID: "7084397f-85eb-43f9-8314-530af31c81c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:29.036251 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.026062 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a99c9a23-d65d-4950-a153-71c8decd10aa-istiod-ca-cert" pod "a99c9a23-d65d-4950-a153-71c8decd10aa" (UID: "a99c9a23-d65d-4950-a153-71c8decd10aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:36:29.036251 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.027169 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4e00bfc-fc1d-46ce-946d-969438944b7e-istio-data" pod "f4e00bfc-fc1d-46ce-946d-969438944b7e" (UID: "f4e00bfc-fc1d-46ce-946d-969438944b7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:29.036251 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.031626 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7084397f-85eb-43f9-8314-530af31c81c8-istio-data" pod "7084397f-85eb-43f9-8314-530af31c81c8" (UID: "7084397f-85eb-43f9-8314-530af31c81c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:29.036251 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.035996 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a99c9a23-d65d-4950-a153-71c8decd10aa-workload-certs" pod "a99c9a23-d65d-4950-a153-71c8decd10aa" (UID: "a99c9a23-d65d-4950-a153-71c8decd10aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:29.036251 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.036023 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc6ceba-80a2-4722-a064-d409fe90fde4-credential-socket" pod "ffc6ceba-80a2-4722-a064-d409fe90fde4" (UID: "ffc6ceba-80a2-4722-a064-d409fe90fde4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:29.036623 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.036322 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7084397f-85eb-43f9-8314-530af31c81c8-credential-socket" pod "7084397f-85eb-43f9-8314-530af31c81c8" (UID: "7084397f-85eb-43f9-8314-530af31c81c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:29.038863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.038831 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4e00bfc-fc1d-46ce-946d-969438944b7e-workload-socket" pod "f4e00bfc-fc1d-46ce-946d-969438944b7e" (UID: "f4e00bfc-fc1d-46ce-946d-969438944b7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:29.042996 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.042967 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a99c9a23-d65d-4950-a153-71c8decd10aa-workload-socket" pod "a99c9a23-d65d-4950-a153-71c8decd10aa" (UID: "a99c9a23-d65d-4950-a153-71c8decd10aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:29.043312 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.043275 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffc6ceba-80a2-4722-a064-d409fe90fde4-istiod-ca-cert" pod "ffc6ceba-80a2-4722-a064-d409fe90fde4" (UID: "ffc6ceba-80a2-4722-a064-d409fe90fde4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:36:29.043585 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.043553 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc6ceba-80a2-4722-a064-d409fe90fde4-istio-data" pod "ffc6ceba-80a2-4722-a064-d409fe90fde4" (UID: "ffc6ceba-80a2-4722-a064-d409fe90fde4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:29.043865 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.043839 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4e00bfc-fc1d-46ce-946d-969438944b7e-credential-socket" pod "f4e00bfc-fc1d-46ce-946d-969438944b7e" (UID: "f4e00bfc-fc1d-46ce-946d-969438944b7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:29.044033 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.044012 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc6ceba-80a2-4722-a064-d409fe90fde4-workload-socket" pod "ffc6ceba-80a2-4722-a064-d409fe90fde4" (UID: "ffc6ceba-80a2-4722-a064-d409fe90fde4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:29.044220 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.044198 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a99c9a23-d65d-4950-a153-71c8decd10aa-istio-data" pod "a99c9a23-d65d-4950-a153-71c8decd10aa" (UID: "a99c9a23-d65d-4950-a153-71c8decd10aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:29.044631 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.044598 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7084397f-85eb-43f9-8314-530af31c81c8-istiod-ca-cert" pod "7084397f-85eb-43f9-8314-530af31c81c8" (UID: "7084397f-85eb-43f9-8314-530af31c81c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:36:29.047148 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.046932 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7084397f-85eb-43f9-8314-530af31c81c8-istio-envoy" pod "7084397f-85eb-43f9-8314-530af31c81c8" (UID: "7084397f-85eb-43f9-8314-530af31c81c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:29.047561 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.047373 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a99c9a23-d65d-4950-a153-71c8decd10aa-credential-socket" pod "a99c9a23-d65d-4950-a153-71c8decd10aa" (UID: "a99c9a23-d65d-4950-a153-71c8decd10aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:29.050390 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.050114 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7084397f-85eb-43f9-8314-530af31c81c8-kube-api-access-9w55w" pod "7084397f-85eb-43f9-8314-530af31c81c8" (UID: "7084397f-85eb-43f9-8314-530af31c81c8"). InnerVolumeSpecName "kube-api-access-9w55w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:29.050821 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.050561 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4e00bfc-fc1d-46ce-946d-969438944b7e-istio-podinfo" pod "f4e00bfc-fc1d-46ce-946d-969438944b7e" (UID: "f4e00bfc-fc1d-46ce-946d-969438944b7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:36:29.055920 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.055870 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4e00bfc-fc1d-46ce-946d-969438944b7e-istiod-ca-cert" pod "f4e00bfc-fc1d-46ce-946d-969438944b7e" (UID: "f4e00bfc-fc1d-46ce-946d-969438944b7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:36:29.056330 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.056008 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7084397f-85eb-43f9-8314-530af31c81c8-workload-certs" pod "7084397f-85eb-43f9-8314-530af31c81c8" (UID: "7084397f-85eb-43f9-8314-530af31c81c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:29.057061 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.056460 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc6ceba-80a2-4722-a064-d409fe90fde4-workload-certs" pod "ffc6ceba-80a2-4722-a064-d409fe90fde4" (UID: "ffc6ceba-80a2-4722-a064-d409fe90fde4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:29.057061 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.056517 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4e00bfc-fc1d-46ce-946d-969438944b7e-workload-certs" pod "f4e00bfc-fc1d-46ce-946d-969438944b7e" (UID: "f4e00bfc-fc1d-46ce-946d-969438944b7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:29.057061 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.056513 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffc6ceba-80a2-4722-a064-d409fe90fde4-kube-api-access-hdckw" pod "ffc6ceba-80a2-4722-a064-d409fe90fde4" (UID: "ffc6ceba-80a2-4722-a064-d409fe90fde4"). InnerVolumeSpecName "kube-api-access-hdckw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:29.059746 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.059629 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7084397f-85eb-43f9-8314-530af31c81c8-istio-podinfo" pod "7084397f-85eb-43f9-8314-530af31c81c8" (UID: "7084397f-85eb-43f9-8314-530af31c81c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:36:29.059746 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.059719 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a99c9a23-d65d-4950-a153-71c8decd10aa-istio-token" pod "a99c9a23-d65d-4950-a153-71c8decd10aa" (UID: "a99c9a23-d65d-4950-a153-71c8decd10aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:29.060694 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.060679 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a99c9a23-d65d-4950-a153-71c8decd10aa-istio-envoy" pod "a99c9a23-d65d-4950-a153-71c8decd10aa" (UID: "a99c9a23-d65d-4950-a153-71c8decd10aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:29.061109 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.061086 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4e00bfc-fc1d-46ce-946d-969438944b7e-istio-token" pod "f4e00bfc-fc1d-46ce-946d-969438944b7e" (UID: "f4e00bfc-fc1d-46ce-946d-969438944b7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:29.061542 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.061507 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ffc6ceba-80a2-4722-a064-d409fe90fde4-istio-podinfo" pod "ffc6ceba-80a2-4722-a064-d409fe90fde4" (UID: "ffc6ceba-80a2-4722-a064-d409fe90fde4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:36:29.061542 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.061513 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4e00bfc-fc1d-46ce-946d-969438944b7e-kube-api-access-tw4fw" pod "f4e00bfc-fc1d-46ce-946d-969438944b7e" (UID: "f4e00bfc-fc1d-46ce-946d-969438944b7e"). InnerVolumeSpecName "kube-api-access-tw4fw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:29.061651 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.061628 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffc6ceba-80a2-4722-a064-d409fe90fde4-istio-token" pod "ffc6ceba-80a2-4722-a064-d409fe90fde4" (UID: "ffc6ceba-80a2-4722-a064-d409fe90fde4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:29.061821 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.061727 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7084397f-85eb-43f9-8314-530af31c81c8-istio-token" pod "7084397f-85eb-43f9-8314-530af31c81c8" (UID: "7084397f-85eb-43f9-8314-530af31c81c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:29.061821 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.061793 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a99c9a23-d65d-4950-a153-71c8decd10aa-kube-api-access-4kx95" pod "a99c9a23-d65d-4950-a153-71c8decd10aa" (UID: "a99c9a23-d65d-4950-a153-71c8decd10aa"). InnerVolumeSpecName "kube-api-access-4kx95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:29.061821 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.061803 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a99c9a23-d65d-4950-a153-71c8decd10aa-istio-podinfo" pod "a99c9a23-d65d-4950-a153-71c8decd10aa" (UID: "a99c9a23-d65d-4950-a153-71c8decd10aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:36:29.063451 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.063426 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc6ceba-80a2-4722-a064-d409fe90fde4-istio-envoy" pod "ffc6ceba-80a2-4722-a064-d409fe90fde4" (UID: "ffc6ceba-80a2-4722-a064-d409fe90fde4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:29.063763 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.063743 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4e00bfc-fc1d-46ce-946d-969438944b7e-istio-envoy" pod "f4e00bfc-fc1d-46ce-946d-969438944b7e" (UID: "f4e00bfc-fc1d-46ce-946d-969438944b7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:29.136036 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.136014 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a99c9a23-d65d-4950-a153-71c8decd10aa-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:29.136216 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.136193 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc6ceba-80a2-4722-a064-d409fe90fde4-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:29.136216 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.136210 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw4fw\" (UniqueName: \"kubernetes.io/projected/f4e00bfc-fc1d-46ce-946d-969438944b7e-kube-api-access-tw4fw\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:29.136345 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.136224 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7084397f-85eb-43f9-8314-530af31c81c8-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:29.136345 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.136237 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffc6ceba-80a2-4722-a064-d409fe90fde4-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:29.136345 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.136250 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4e00bfc-fc1d-46ce-946d-969438944b7e-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:29.136345 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.136262 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4e00bfc-fc1d-46ce-946d-969438944b7e-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:29.136345 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.136276 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7084397f-85eb-43f9-8314-530af31c81c8-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:29.136345 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.136289 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4e00bfc-fc1d-46ce-946d-969438944b7e-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:29.136345 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.136302 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a99c9a23-d65d-4950-a153-71c8decd10aa-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:29.136345 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.136315 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a99c9a23-d65d-4950-a153-71c8decd10aa-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:29.136345 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.136328 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7084397f-85eb-43f9-8314-530af31c81c8-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:29.136345 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.136341 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4e00bfc-fc1d-46ce-946d-969438944b7e-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:29.136863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.136356 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w55w\" (UniqueName: \"kubernetes.io/projected/7084397f-85eb-43f9-8314-530af31c81c8-kube-api-access-9w55w\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:29.136863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.136373 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffc6ceba-80a2-4722-a064-d409fe90fde4-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:29.136863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.136388 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a99c9a23-d65d-4950-a153-71c8decd10aa-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:29.136863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.136403 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4e00bfc-fc1d-46ce-946d-969438944b7e-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:29.136863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.136415 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffc6ceba-80a2-4722-a064-d409fe90fde4-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:29.136863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.136429 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7084397f-85eb-43f9-8314-530af31c81c8-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:29.136863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.136444 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7084397f-85eb-43f9-8314-530af31c81c8-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:29.136863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.136457 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a99c9a23-d65d-4950-a153-71c8decd10aa-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:29.136863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.136470 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4e00bfc-fc1d-46ce-946d-969438944b7e-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:29.136863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.136482 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7084397f-85eb-43f9-8314-530af31c81c8-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:29.136863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.136494 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7084397f-85eb-43f9-8314-530af31c81c8-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:29.136863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.136507 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a99c9a23-d65d-4950-a153-71c8decd10aa-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:29.136863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.136519 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffc6ceba-80a2-4722-a064-d409fe90fde4-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:29.136863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.136532 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffc6ceba-80a2-4722-a064-d409fe90fde4-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:29.136863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.136544 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4e00bfc-fc1d-46ce-946d-969438944b7e-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:29.136863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.136556 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a99c9a23-d65d-4950-a153-71c8decd10aa-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:29.136863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.136567 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7084397f-85eb-43f9-8314-530af31c81c8-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:29.136863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.136580 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc6ceba-80a2-4722-a064-d409fe90fde4-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:29.136863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.136594 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a99c9a23-d65d-4950-a153-71c8decd10aa-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:29.136863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.136608 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kx95\" (UniqueName: \"kubernetes.io/projected/a99c9a23-d65d-4950-a153-71c8decd10aa-kube-api-access-4kx95\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:29.136863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.136624 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4e00bfc-fc1d-46ce-946d-969438944b7e-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:29.136863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.136638 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdckw\" (UniqueName: \"kubernetes.io/projected/ffc6ceba-80a2-4722-a064-d409fe90fde4-kube-api-access-hdckw\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:29.136863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.136651 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffc6ceba-80a2-4722-a064-d409fe90fde4-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:29.838666 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.838583 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" Oct 30 18:36:29.838666 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.838606 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n" event={"ID":"7084397f-85eb-43f9-8314-530af31c81c8","Type":"ContainerDied","Data":"73ec8eb807c38641ef3be1df3523f8f84f0bca72f8b4fd36e918644f60a72440"} Oct 30 18:36:29.839268 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.838668 4442 scope.go:122] "RemoveContainer" containerID="cdbdf5c245588d68a0c29b38270a6b0f6c182f8b99256f6a7ed6a8512dedbc57" Oct 30 18:36:29.840692 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.840667 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7616-ba9af8" event={"ID":"645f6dd5-194f-4b50-a31a-36dad375b5df","Type":"ContainerStarted","Data":"57b376c1c068799b4906efb58b4c5dd62c8562d54160735208092c1a933830a3"} Oct 30 18:36:29.842365 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.842340 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" Oct 30 18:36:29.842480 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.842367 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww" event={"ID":"ffc6ceba-80a2-4722-a064-d409fe90fde4","Type":"ContainerDied","Data":"1d2b2917757aa278bbac94c713841acfea88bf970aca9a5432848696a3df586e"} Oct 30 18:36:29.843918 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.843901 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" Oct 30 18:36:29.844012 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.843933 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74" event={"ID":"a99c9a23-d65d-4950-a153-71c8decd10aa","Type":"ContainerDied","Data":"50bc0a527445704fc6d26349e5b57e39cb7db470f6dc938111cfe70819cad4a7"} Oct 30 18:36:29.845572 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.845553 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" event={"ID":"f4e00bfc-fc1d-46ce-946d-969438944b7e","Type":"ContainerDied","Data":"788eabc9242cdb66560e32d9ddd75e66c4c71e3c1cfc8264fd5acedba13c0673"} Oct 30 18:36:29.845640 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.845597 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d" Oct 30 18:36:29.851832 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.851814 4442 scope.go:122] "RemoveContainer" containerID="d84f0e5b32a1944e7fd2cb71934548e8c0524a97ef7823dcec34cd9299253a12" Oct 30 18:36:29.862689 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.862663 4442 scope.go:122] "RemoveContainer" containerID="b2dfe6d37d9b6174fec9511a906965354a94f74e5e90d431e1958e6971c118b5" Oct 30 18:36:29.872705 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.872683 4442 scope.go:122] "RemoveContainer" containerID="e613121c7326daf440ea98122f0e0b43d85fb440d0946585ad4e7f3ea4b19e97" Oct 30 18:36:29.903367 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.903335 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d"] Oct 30 18:36:29.914758 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.914730 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-q8b8d"] Oct 30 18:36:29.951170 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.951120 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n"] Oct 30 18:36:29.955157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.955122 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-p8j8n"] Oct 30 18:36:29.989523 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:29.989475 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7616-ba9af8" podStartSLOduration=2.567363655 podStartE2EDuration="4.989461305s" totalImagesPullingTime="2.42209765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:29.987007546 +0000 UTC m=+2287.077069884" watchObservedRunningTime="2026-10-30 18:36:29.989461305 +0000 UTC m=+2287.079523643" Oct 30 18:36:30.025452 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:30.025423 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww"] Oct 30 18:36:30.029620 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:30.029572 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-hnrww"] Oct 30 18:36:30.062036 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:30.062006 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74"] Oct 30 18:36:30.069120 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:30.069094 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-npw74"] Oct 30 18:36:30.813267 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:30.813242 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" Oct 30 18:36:30.851797 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:30.851726 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7616-ba9af8"] Oct 30 18:36:30.863505 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:30.857481 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7616-ba9af8"] Oct 30 18:36:30.868025 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:30.867277 4442 generic.go:356] "Generic (PLEG): container finished" podID="6bbe6b88-960d-47db-a270-47fc4cee7f44" containerID="870d83262330ab1628bebb565ff996a55a75ecba3831a6ce2e6b6d5c310682af" exitCode=0 Oct 30 18:36:30.868025 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:30.867394 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" event={"ID":"6bbe6b88-960d-47db-a270-47fc4cee7f44","Type":"ContainerDied","Data":"870d83262330ab1628bebb565ff996a55a75ecba3831a6ce2e6b6d5c310682af"} Oct 30 18:36:30.868025 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:30.867475 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" event={"ID":"6bbe6b88-960d-47db-a270-47fc4cee7f44","Type":"ContainerDied","Data":"069cd011d9a91474df255dacbedce04b35fb470d1d66853bdea518e2126634ed"} Oct 30 18:36:30.868025 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:30.867504 4442 scope.go:122] "RemoveContainer" containerID="870d83262330ab1628bebb565ff996a55a75ecba3831a6ce2e6b6d5c310682af" Oct 30 18:36:30.868025 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:30.867656 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77" Oct 30 18:36:30.871064 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:30.871035 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bbe6b88-960d-47db-a270-47fc4cee7f44-kube-api-access-b574v\" (UniqueName: \"kubernetes.io/projected/6bbe6b88-960d-47db-a270-47fc4cee7f44-kube-api-access-b574v\") pod \"6bbe6b88-960d-47db-a270-47fc4cee7f44\" (UID: \"6bbe6b88-960d-47db-a270-47fc4cee7f44\") " Oct 30 18:36:30.871370 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:30.871348 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bbe6b88-960d-47db-a270-47fc4cee7f44-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bbe6b88-960d-47db-a270-47fc4cee7f44-workload-certs\") pod \"6bbe6b88-960d-47db-a270-47fc4cee7f44\" (UID: \"6bbe6b88-960d-47db-a270-47fc4cee7f44\") " Oct 30 18:36:30.871626 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:30.871608 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bbe6b88-960d-47db-a270-47fc4cee7f44-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bbe6b88-960d-47db-a270-47fc4cee7f44-istiod-ca-cert\") pod \"6bbe6b88-960d-47db-a270-47fc4cee7f44\" (UID: \"6bbe6b88-960d-47db-a270-47fc4cee7f44\") " Oct 30 18:36:30.871781 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:30.871767 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bbe6b88-960d-47db-a270-47fc4cee7f44-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbe6b88-960d-47db-a270-47fc4cee7f44-credential-socket\") pod \"6bbe6b88-960d-47db-a270-47fc4cee7f44\" (UID: \"6bbe6b88-960d-47db-a270-47fc4cee7f44\") " Oct 30 18:36:30.872031 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:30.872014 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bbe6b88-960d-47db-a270-47fc4cee7f44-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bbe6b88-960d-47db-a270-47fc4cee7f44-istio-envoy\") pod \"6bbe6b88-960d-47db-a270-47fc4cee7f44\" (UID: \"6bbe6b88-960d-47db-a270-47fc4cee7f44\") " Oct 30 18:36:30.872244 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:30.872226 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bbe6b88-960d-47db-a270-47fc4cee7f44-istio-token\" (UniqueName: \"kubernetes.io/projected/6bbe6b88-960d-47db-a270-47fc4cee7f44-istio-token\") pod \"6bbe6b88-960d-47db-a270-47fc4cee7f44\" (UID: \"6bbe6b88-960d-47db-a270-47fc4cee7f44\") " Oct 30 18:36:30.872431 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:30.872414 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bbe6b88-960d-47db-a270-47fc4cee7f44-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bbe6b88-960d-47db-a270-47fc4cee7f44-istio-data\") pod \"6bbe6b88-960d-47db-a270-47fc4cee7f44\" (UID: \"6bbe6b88-960d-47db-a270-47fc4cee7f44\") " Oct 30 18:36:30.872596 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:30.872582 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bbe6b88-960d-47db-a270-47fc4cee7f44-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbe6b88-960d-47db-a270-47fc4cee7f44-workload-socket\") pod \"6bbe6b88-960d-47db-a270-47fc4cee7f44\" (UID: \"6bbe6b88-960d-47db-a270-47fc4cee7f44\") " Oct 30 18:36:30.872724 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:30.872710 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6bbe6b88-960d-47db-a270-47fc4cee7f44-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bbe6b88-960d-47db-a270-47fc4cee7f44-istio-podinfo\") pod \"6bbe6b88-960d-47db-a270-47fc4cee7f44\" (UID: \"6bbe6b88-960d-47db-a270-47fc4cee7f44\") " Oct 30 18:36:30.874224 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:30.873937 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bbe6b88-960d-47db-a270-47fc4cee7f44-credential-socket" pod "6bbe6b88-960d-47db-a270-47fc4cee7f44" (UID: "6bbe6b88-960d-47db-a270-47fc4cee7f44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:30.875782 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:30.875066 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bbe6b88-960d-47db-a270-47fc4cee7f44-istiod-ca-cert" pod "6bbe6b88-960d-47db-a270-47fc4cee7f44" (UID: "6bbe6b88-960d-47db-a270-47fc4cee7f44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:36:30.875782 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:30.875348 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bbe6b88-960d-47db-a270-47fc4cee7f44-workload-socket" pod "6bbe6b88-960d-47db-a270-47fc4cee7f44" (UID: "6bbe6b88-960d-47db-a270-47fc4cee7f44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:30.876191 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:30.876170 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bbe6b88-960d-47db-a270-47fc4cee7f44-istio-data" pod "6bbe6b88-960d-47db-a270-47fc4cee7f44" (UID: "6bbe6b88-960d-47db-a270-47fc4cee7f44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:30.876364 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:30.876345 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bbe6b88-960d-47db-a270-47fc4cee7f44-workload-certs" pod "6bbe6b88-960d-47db-a270-47fc4cee7f44" (UID: "6bbe6b88-960d-47db-a270-47fc4cee7f44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:30.885857 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:30.885836 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bbe6b88-960d-47db-a270-47fc4cee7f44-istio-envoy" pod "6bbe6b88-960d-47db-a270-47fc4cee7f44" (UID: "6bbe6b88-960d-47db-a270-47fc4cee7f44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:30.885999 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:30.885982 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bbe6b88-960d-47db-a270-47fc4cee7f44-istio-token" pod "6bbe6b88-960d-47db-a270-47fc4cee7f44" (UID: "6bbe6b88-960d-47db-a270-47fc4cee7f44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:30.886101 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:30.886083 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bbe6b88-960d-47db-a270-47fc4cee7f44-kube-api-access-b574v" pod "6bbe6b88-960d-47db-a270-47fc4cee7f44" (UID: "6bbe6b88-960d-47db-a270-47fc4cee7f44"). InnerVolumeSpecName "kube-api-access-b574v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:30.888345 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:30.888323 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6bbe6b88-960d-47db-a270-47fc4cee7f44-istio-podinfo" pod "6bbe6b88-960d-47db-a270-47fc4cee7f44" (UID: "6bbe6b88-960d-47db-a270-47fc4cee7f44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:36:30.912921 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:30.912900 4442 scope.go:122] "RemoveContainer" containerID="43183b0e889c19bb57010aa96bdb89000fea7f09d635fce2c57165a4a0ce5a53" Oct 30 18:36:30.924046 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:30.924020 4442 scope.go:122] "RemoveContainer" containerID="870d83262330ab1628bebb565ff996a55a75ecba3831a6ce2e6b6d5c310682af" Oct 30 18:36:30.930462 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:36:30.930290 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"870d83262330ab1628bebb565ff996a55a75ecba3831a6ce2e6b6d5c310682af\": container with ID starting with 870d83262330ab1628bebb565ff996a55a75ecba3831a6ce2e6b6d5c310682af not found: ID does not exist" containerID="870d83262330ab1628bebb565ff996a55a75ecba3831a6ce2e6b6d5c310682af" Oct 30 18:36:30.930462 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:30.930336 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"870d83262330ab1628bebb565ff996a55a75ecba3831a6ce2e6b6d5c310682af"} err="failed to get container status \"870d83262330ab1628bebb565ff996a55a75ecba3831a6ce2e6b6d5c310682af\": rpc error: code = NotFound desc = could not find container \"870d83262330ab1628bebb565ff996a55a75ecba3831a6ce2e6b6d5c310682af\": container with ID starting with 870d83262330ab1628bebb565ff996a55a75ecba3831a6ce2e6b6d5c310682af not found: ID does not exist" Oct 30 18:36:30.930462 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:30.930361 4442 scope.go:122] "RemoveContainer" containerID="43183b0e889c19bb57010aa96bdb89000fea7f09d635fce2c57165a4a0ce5a53" Oct 30 18:36:30.930870 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:36:30.930793 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43183b0e889c19bb57010aa96bdb89000fea7f09d635fce2c57165a4a0ce5a53\": container with ID starting with 43183b0e889c19bb57010aa96bdb89000fea7f09d635fce2c57165a4a0ce5a53 not found: ID does not exist" containerID="43183b0e889c19bb57010aa96bdb89000fea7f09d635fce2c57165a4a0ce5a53" Oct 30 18:36:30.930870 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:30.930835 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43183b0e889c19bb57010aa96bdb89000fea7f09d635fce2c57165a4a0ce5a53"} err="failed to get container status \"43183b0e889c19bb57010aa96bdb89000fea7f09d635fce2c57165a4a0ce5a53\": rpc error: code = NotFound desc = could not find container \"43183b0e889c19bb57010aa96bdb89000fea7f09d635fce2c57165a4a0ce5a53\": container with ID starting with 43183b0e889c19bb57010aa96bdb89000fea7f09d635fce2c57165a4a0ce5a53 not found: ID does not exist" Oct 30 18:36:30.974886 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:30.974856 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bbe6b88-960d-47db-a270-47fc4cee7f44-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:30.974886 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:30.974886 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b574v\" (UniqueName: \"kubernetes.io/projected/6bbe6b88-960d-47db-a270-47fc4cee7f44-kube-api-access-b574v\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:30.975065 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:30.974897 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bbe6b88-960d-47db-a270-47fc4cee7f44-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:30.975065 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:30.974907 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bbe6b88-960d-47db-a270-47fc4cee7f44-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:30.975065 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:30.974915 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbe6b88-960d-47db-a270-47fc4cee7f44-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:30.975065 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:30.974922 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bbe6b88-960d-47db-a270-47fc4cee7f44-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:30.975065 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:30.974932 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bbe6b88-960d-47db-a270-47fc4cee7f44-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:30.975065 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:30.974939 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bbe6b88-960d-47db-a270-47fc4cee7f44-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:30.975065 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:30.974946 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbe6b88-960d-47db-a270-47fc4cee7f44-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:31.011031 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:31.010999 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7616-b9f8ed"] Oct 30 18:36:31.011655 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:31.011640 4442 cpu_manager.go:414] "containerMap: removing container" podUID="a99c9a23-d65d-4950-a153-71c8decd10aa" containerName="istio-proxy" Oct 30 18:36:31.011770 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:31.011657 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a99c9a23-d65d-4950-a153-71c8decd10aa" containerName="istio-proxy" Oct 30 18:36:31.011770 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:31.011677 4442 cpu_manager.go:414] "containerMap: removing container" podUID="7084397f-85eb-43f9-8314-530af31c81c8" containerName="istio-proxy" Oct 30 18:36:31.011770 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:31.011681 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7084397f-85eb-43f9-8314-530af31c81c8" containerName="istio-proxy" Oct 30 18:36:31.011770 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:31.011687 4442 cpu_manager.go:414] "containerMap: removing container" podUID="6bbe6b88-960d-47db-a270-47fc4cee7f44" containerName="istio-proxy" Oct 30 18:36:31.011770 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:31.011692 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bbe6b88-960d-47db-a270-47fc4cee7f44" containerName="istio-proxy" Oct 30 18:36:31.011770 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:31.011700 4442 cpu_manager.go:414] "containerMap: removing container" podUID="645f6dd5-194f-4b50-a31a-36dad375b5df" containerName="test-curl-7616-ba9af8" Oct 30 18:36:31.011770 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:31.011705 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="645f6dd5-194f-4b50-a31a-36dad375b5df" containerName="test-curl-7616-ba9af8" Oct 30 18:36:31.011770 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:31.011716 4442 cpu_manager.go:414] "containerMap: removing container" podUID="ffc6ceba-80a2-4722-a064-d409fe90fde4" containerName="istio-proxy" Oct 30 18:36:31.011770 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:31.011720 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffc6ceba-80a2-4722-a064-d409fe90fde4" containerName="istio-proxy" Oct 30 18:36:31.011770 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:31.011729 4442 cpu_manager.go:414] "containerMap: removing container" podUID="f4e00bfc-fc1d-46ce-946d-969438944b7e" containerName="istio-proxy" Oct 30 18:36:31.011770 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:31.011733 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4e00bfc-fc1d-46ce-946d-969438944b7e" containerName="istio-proxy" Oct 30 18:36:31.011770 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:31.011752 4442 cpu_manager.go:414] "containerMap: removing container" podUID="6bbe6b88-960d-47db-a270-47fc4cee7f44" containerName="istio-proxy" Oct 30 18:36:31.011770 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:31.011755 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bbe6b88-960d-47db-a270-47fc4cee7f44" containerName="istio-proxy" Oct 30 18:36:31.012189 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:31.011840 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="a99c9a23-d65d-4950-a153-71c8decd10aa" containerName="istio-proxy" Oct 30 18:36:31.012189 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:31.011850 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="7084397f-85eb-43f9-8314-530af31c81c8" containerName="istio-proxy" Oct 30 18:36:31.012189 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:31.011857 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bbe6b88-960d-47db-a270-47fc4cee7f44" containerName="istio-proxy" Oct 30 18:36:31.012189 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:31.011864 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="645f6dd5-194f-4b50-a31a-36dad375b5df" containerName="test-curl-7616-ba9af8" Oct 30 18:36:31.012189 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:31.011870 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffc6ceba-80a2-4722-a064-d409fe90fde4" containerName="istio-proxy" Oct 30 18:36:31.012189 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:31.011877 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4e00bfc-fc1d-46ce-946d-969438944b7e" containerName="istio-proxy" Oct 30 18:36:31.012189 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:31.011885 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bbe6b88-960d-47db-a270-47fc4cee7f44" containerName="istio-proxy" Oct 30 18:36:31.016762 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:31.016745 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7616-b9f8ed" Oct 30 18:36:31.033812 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:31.033789 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7616-b9f8ed"] Oct 30 18:36:31.066692 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:31.066656 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7084397f-85eb-43f9-8314-530af31c81c8" path="/var/lib/kubelet/pods/7084397f-85eb-43f9-8314-530af31c81c8/volumes" Oct 30 18:36:31.067503 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:31.067476 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a99c9a23-d65d-4950-a153-71c8decd10aa" path="/var/lib/kubelet/pods/a99c9a23-d65d-4950-a153-71c8decd10aa/volumes" Oct 30 18:36:31.068147 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:31.068108 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4e00bfc-fc1d-46ce-946d-969438944b7e" path="/var/lib/kubelet/pods/f4e00bfc-fc1d-46ce-946d-969438944b7e/volumes" Oct 30 18:36:31.069289 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:31.069266 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffc6ceba-80a2-4722-a064-d409fe90fde4" path="/var/lib/kubelet/pods/ffc6ceba-80a2-4722-a064-d409fe90fde4/volumes" Oct 30 18:36:31.075979 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:31.075944 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4kv6\" (UniqueName: \"kubernetes.io/projected/7c61a56f-c91c-4e23-8abc-8b28d245b733-kube-api-access-x4kv6\") pod \"test-curl-7616-b9f8ed\" (UID: \"7c61a56f-c91c-4e23-8abc-8b28d245b733\") " pod="opendatahub/test-curl-7616-b9f8ed" Oct 30 18:36:31.177183 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:31.177083 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4kv6\" (UniqueName: \"kubernetes.io/projected/7c61a56f-c91c-4e23-8abc-8b28d245b733-kube-api-access-x4kv6\") pod \"test-curl-7616-b9f8ed\" (UID: \"7c61a56f-c91c-4e23-8abc-8b28d245b733\") " pod="opendatahub/test-curl-7616-b9f8ed" Oct 30 18:36:31.196586 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:31.196550 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4kv6\" (UniqueName: \"kubernetes.io/projected/7c61a56f-c91c-4e23-8abc-8b28d245b733-kube-api-access-x4kv6\") pod \"test-curl-7616-b9f8ed\" (UID: \"7c61a56f-c91c-4e23-8abc-8b28d245b733\") " pod="opendatahub/test-curl-7616-b9f8ed" Oct 30 18:36:31.210087 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:31.210059 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77"] Oct 30 18:36:31.213850 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:31.213826 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-160bf8-openshift-default-5c6f5df4b5-p2z77"] Oct 30 18:36:31.340706 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:31.340669 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7616-b9f8ed" Oct 30 18:36:31.641105 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:31.641077 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7616-b9f8ed"] Oct 30 18:36:31.641520 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:36:31.641499 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c61a56f_c91c_4e23_8abc_8b28d245b733.slice/crio-11b8bff9c332a18da4291699304e918777fc613c681a90ea7fc4b434624a0d5f WatchSource:0}: Error finding container 11b8bff9c332a18da4291699304e918777fc613c681a90ea7fc4b434624a0d5f: Status 404 returned error can't find the container with id 11b8bff9c332a18da4291699304e918777fc613c681a90ea7fc4b434624a0d5f Oct 30 18:36:31.893604 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:31.893502 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7616-b9f8ed" event={"ID":"7c61a56f-c91c-4e23-8abc-8b28d245b733","Type":"ContainerStarted","Data":"7903685e0b4c39cd15c8e4583164bf9c90abd96dfd5e0397ae48d69477e8e916"} Oct 30 18:36:31.893604 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:31.893544 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7616-b9f8ed" event={"ID":"7c61a56f-c91c-4e23-8abc-8b28d245b733","Type":"ContainerStarted","Data":"11b8bff9c332a18da4291699304e918777fc613c681a90ea7fc4b434624a0d5f"} Oct 30 18:36:31.894820 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:31.894798 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7616-ba9af8" podUID="645f6dd5-194f-4b50-a31a-36dad375b5df" containerName="test-curl-7616-ba9af8" containerID="cri-o://57b376c1c068799b4906efb58b4c5dd62c8562d54160735208092c1a933830a3" gracePeriod=2 Oct 30 18:36:31.940206 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:31.940113 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7616-b9f8ed" podStartSLOduration=1.940093208 podStartE2EDuration="1.940093208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:31.937348812 +0000 UTC m=+2289.027411151" watchObservedRunningTime="2026-10-30 18:36:31.940093208 +0000 UTC m=+2289.030155549" Oct 30 18:36:33.068686 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:33.068645 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bbe6b88-960d-47db-a270-47fc4cee7f44" path="/var/lib/kubelet/pods/6bbe6b88-960d-47db-a270-47fc4cee7f44/volumes" Oct 30 18:36:34.196245 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:34.196161 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7616-ba9af8" Oct 30 18:36:34.332457 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:34.332415 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/645f6dd5-194f-4b50-a31a-36dad375b5df-kube-api-access-srzd6\" (UniqueName: \"kubernetes.io/projected/645f6dd5-194f-4b50-a31a-36dad375b5df-kube-api-access-srzd6\") pod \"645f6dd5-194f-4b50-a31a-36dad375b5df\" (UID: \"645f6dd5-194f-4b50-a31a-36dad375b5df\") " Oct 30 18:36:34.334877 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:34.334843 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/645f6dd5-194f-4b50-a31a-36dad375b5df-kube-api-access-srzd6" pod "645f6dd5-194f-4b50-a31a-36dad375b5df" (UID: "645f6dd5-194f-4b50-a31a-36dad375b5df"). InnerVolumeSpecName "kube-api-access-srzd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:34.434764 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:34.434719 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srzd6\" (UniqueName: \"kubernetes.io/projected/645f6dd5-194f-4b50-a31a-36dad375b5df-kube-api-access-srzd6\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:34.918965 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:34.918929 4442 generic.go:356] "Generic (PLEG): container finished" podID="645f6dd5-194f-4b50-a31a-36dad375b5df" containerID="57b376c1c068799b4906efb58b4c5dd62c8562d54160735208092c1a933830a3" exitCode=137 Oct 30 18:36:34.919221 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:34.918975 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7616-ba9af8" Oct 30 18:36:34.919221 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:34.919029 4442 scope.go:122] "RemoveContainer" containerID="57b376c1c068799b4906efb58b4c5dd62c8562d54160735208092c1a933830a3" Oct 30 18:36:34.930697 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:34.930666 4442 scope.go:122] "RemoveContainer" containerID="57b376c1c068799b4906efb58b4c5dd62c8562d54160735208092c1a933830a3" Oct 30 18:36:34.931028 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:36:34.931006 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57b376c1c068799b4906efb58b4c5dd62c8562d54160735208092c1a933830a3\": container with ID starting with 57b376c1c068799b4906efb58b4c5dd62c8562d54160735208092c1a933830a3 not found: ID does not exist" containerID="57b376c1c068799b4906efb58b4c5dd62c8562d54160735208092c1a933830a3" Oct 30 18:36:34.931105 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:34.931037 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57b376c1c068799b4906efb58b4c5dd62c8562d54160735208092c1a933830a3"} err="failed to get container status \"57b376c1c068799b4906efb58b4c5dd62c8562d54160735208092c1a933830a3\": rpc error: code = NotFound desc = could not find container \"57b376c1c068799b4906efb58b4c5dd62c8562d54160735208092c1a933830a3\": container with ID starting with 57b376c1c068799b4906efb58b4c5dd62c8562d54160735208092c1a933830a3 not found: ID does not exist" Oct 30 18:36:34.941440 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:36:34.941409 4442 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7616-ba9af8\" is forbidden: User \"system:node:ip-10-0-138-176.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-176.ec2.internal' and this object" podUID="645f6dd5-194f-4b50-a31a-36dad375b5df" pod="opendatahub/test-curl-7616-ba9af8" Oct 30 18:36:35.067071 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:35.067039 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="645f6dd5-194f-4b50-a31a-36dad375b5df" path="/var/lib/kubelet/pods/645f6dd5-194f-4b50-a31a-36dad375b5df/volumes" Oct 30 18:36:38.610247 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:38.610205 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2"] Oct 30 18:36:38.610770 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:38.610538 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" podUID="b7df5e06-8248-4312-8abc-72deee20f840" containerName="istio-proxy" containerID="cri-o://79273c2320b2eb6d1c43258a027cb6a00632d9c684ddaca131fa84f5032dbaf2" gracePeriod=30 Oct 30 18:36:42.798292 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:42.798255 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7616-b9f8ed"] Oct 30 18:36:42.798824 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:42.798424 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7616-b9f8ed" podUID="7c61a56f-c91c-4e23-8abc-8b28d245b733" containerName="test-curl-7616-b9f8ed" containerID="cri-o://7903685e0b4c39cd15c8e4583164bf9c90abd96dfd5e0397ae48d69477e8e916" gracePeriod=2 Oct 30 18:36:42.806255 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:42.806226 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7616-b9f8ed"] Oct 30 18:36:42.811811 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:36:42.811754 4442 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7616-b9f8ed\" is forbidden: User \"system:node:ip-10-0-138-176.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-176.ec2.internal' and this object" podUID="7c61a56f-c91c-4e23-8abc-8b28d245b733" pod="opendatahub/test-curl-7616-b9f8ed" Oct 30 18:36:43.076771 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:36:43.076674 4442 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7616-b9f8ed\" is forbidden: User \"system:node:ip-10-0-138-176.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-176.ec2.internal' and this object" podUID="7c61a56f-c91c-4e23-8abc-8b28d245b733" pod="opendatahub/test-curl-7616-b9f8ed" Oct 30 18:36:43.902793 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:43.902763 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" Oct 30 18:36:44.001111 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.001078 4442 generic.go:356] "Generic (PLEG): container finished" podID="b7df5e06-8248-4312-8abc-72deee20f840" containerID="79273c2320b2eb6d1c43258a027cb6a00632d9c684ddaca131fa84f5032dbaf2" exitCode=0 Oct 30 18:36:44.001394 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.001370 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" event={"ID":"b7df5e06-8248-4312-8abc-72deee20f840","Type":"ContainerDied","Data":"79273c2320b2eb6d1c43258a027cb6a00632d9c684ddaca131fa84f5032dbaf2"} Oct 30 18:36:44.001488 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.001463 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" event={"ID":"b7df5e06-8248-4312-8abc-72deee20f840","Type":"ContainerDied","Data":"d046c1bba7b85f1a90a6ed5a828dbe9b9cfa71747a77d0240bc9cddf18ba3a0b"} Oct 30 18:36:44.001547 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.001497 4442 scope.go:122] "RemoveContainer" containerID="79273c2320b2eb6d1c43258a027cb6a00632d9c684ddaca131fa84f5032dbaf2" Oct 30 18:36:44.001674 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.001659 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2" Oct 30 18:36:44.011241 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.011212 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7df5e06-8248-4312-8abc-72deee20f840-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7df5e06-8248-4312-8abc-72deee20f840-workload-socket\") pod \"b7df5e06-8248-4312-8abc-72deee20f840\" (UID: \"b7df5e06-8248-4312-8abc-72deee20f840\") " Oct 30 18:36:44.011383 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.011296 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7df5e06-8248-4312-8abc-72deee20f840-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7df5e06-8248-4312-8abc-72deee20f840-credential-socket\") pod \"b7df5e06-8248-4312-8abc-72deee20f840\" (UID: \"b7df5e06-8248-4312-8abc-72deee20f840\") " Oct 30 18:36:44.011383 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.011361 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7df5e06-8248-4312-8abc-72deee20f840-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7df5e06-8248-4312-8abc-72deee20f840-workload-certs\") pod \"b7df5e06-8248-4312-8abc-72deee20f840\" (UID: \"b7df5e06-8248-4312-8abc-72deee20f840\") " Oct 30 18:36:44.011514 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.011416 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7df5e06-8248-4312-8abc-72deee20f840-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7df5e06-8248-4312-8abc-72deee20f840-istio-data\") pod \"b7df5e06-8248-4312-8abc-72deee20f840\" (UID: \"b7df5e06-8248-4312-8abc-72deee20f840\") " Oct 30 18:36:44.011514 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.011501 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7df5e06-8248-4312-8abc-72deee20f840-istio-token\" (UniqueName: \"kubernetes.io/projected/b7df5e06-8248-4312-8abc-72deee20f840-istio-token\") pod \"b7df5e06-8248-4312-8abc-72deee20f840\" (UID: \"b7df5e06-8248-4312-8abc-72deee20f840\") " Oct 30 18:36:44.011627 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.011549 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7df5e06-8248-4312-8abc-72deee20f840-kube-api-access-gpg2g\" (UniqueName: \"kubernetes.io/projected/b7df5e06-8248-4312-8abc-72deee20f840-kube-api-access-gpg2g\") pod \"b7df5e06-8248-4312-8abc-72deee20f840\" (UID: \"b7df5e06-8248-4312-8abc-72deee20f840\") " Oct 30 18:36:44.011627 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.011601 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7df5e06-8248-4312-8abc-72deee20f840-credential-socket" pod "b7df5e06-8248-4312-8abc-72deee20f840" (UID: "b7df5e06-8248-4312-8abc-72deee20f840"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:44.011743 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.011703 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7df5e06-8248-4312-8abc-72deee20f840-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7df5e06-8248-4312-8abc-72deee20f840-istiod-ca-cert\") pod \"b7df5e06-8248-4312-8abc-72deee20f840\" (UID: \"b7df5e06-8248-4312-8abc-72deee20f840\") " Oct 30 18:36:44.011826 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.011810 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7df5e06-8248-4312-8abc-72deee20f840-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7df5e06-8248-4312-8abc-72deee20f840-istio-envoy\") pod \"b7df5e06-8248-4312-8abc-72deee20f840\" (UID: \"b7df5e06-8248-4312-8abc-72deee20f840\") " Oct 30 18:36:44.011902 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.011866 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7df5e06-8248-4312-8abc-72deee20f840-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7df5e06-8248-4312-8abc-72deee20f840-istio-podinfo\") pod \"b7df5e06-8248-4312-8abc-72deee20f840\" (UID: \"b7df5e06-8248-4312-8abc-72deee20f840\") " Oct 30 18:36:44.012176 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.012123 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7df5e06-8248-4312-8abc-72deee20f840-istiod-ca-cert" pod "b7df5e06-8248-4312-8abc-72deee20f840" (UID: "b7df5e06-8248-4312-8abc-72deee20f840"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:36:44.012649 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.012624 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7df5e06-8248-4312-8abc-72deee20f840-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:44.012744 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.012654 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7df5e06-8248-4312-8abc-72deee20f840-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:44.013022 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.012994 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7df5e06-8248-4312-8abc-72deee20f840-istio-data" pod "b7df5e06-8248-4312-8abc-72deee20f840" (UID: "b7df5e06-8248-4312-8abc-72deee20f840"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:44.013295 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.013267 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7df5e06-8248-4312-8abc-72deee20f840-workload-certs" pod "b7df5e06-8248-4312-8abc-72deee20f840" (UID: "b7df5e06-8248-4312-8abc-72deee20f840"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:44.013408 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.013331 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7df5e06-8248-4312-8abc-72deee20f840-workload-socket" pod "b7df5e06-8248-4312-8abc-72deee20f840" (UID: "b7df5e06-8248-4312-8abc-72deee20f840"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:44.014946 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.014929 4442 scope.go:122] "RemoveContainer" containerID="79273c2320b2eb6d1c43258a027cb6a00632d9c684ddaca131fa84f5032dbaf2" Oct 30 18:36:44.015278 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:36:44.015259 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79273c2320b2eb6d1c43258a027cb6a00632d9c684ddaca131fa84f5032dbaf2\": container with ID starting with 79273c2320b2eb6d1c43258a027cb6a00632d9c684ddaca131fa84f5032dbaf2 not found: ID does not exist" containerID="79273c2320b2eb6d1c43258a027cb6a00632d9c684ddaca131fa84f5032dbaf2" Oct 30 18:36:44.015365 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.015289 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79273c2320b2eb6d1c43258a027cb6a00632d9c684ddaca131fa84f5032dbaf2"} err="failed to get container status \"79273c2320b2eb6d1c43258a027cb6a00632d9c684ddaca131fa84f5032dbaf2\": rpc error: code = NotFound desc = could not find container \"79273c2320b2eb6d1c43258a027cb6a00632d9c684ddaca131fa84f5032dbaf2\": container with ID starting with 79273c2320b2eb6d1c43258a027cb6a00632d9c684ddaca131fa84f5032dbaf2 not found: ID does not exist" Oct 30 18:36:44.015777 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.015658 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7df5e06-8248-4312-8abc-72deee20f840-kube-api-access-gpg2g" pod "b7df5e06-8248-4312-8abc-72deee20f840" (UID: "b7df5e06-8248-4312-8abc-72deee20f840"). InnerVolumeSpecName "kube-api-access-gpg2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:44.039600 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.039557 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7df5e06-8248-4312-8abc-72deee20f840-istio-podinfo" pod "b7df5e06-8248-4312-8abc-72deee20f840" (UID: "b7df5e06-8248-4312-8abc-72deee20f840"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:36:44.039600 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.039580 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7df5e06-8248-4312-8abc-72deee20f840-istio-token" pod "b7df5e06-8248-4312-8abc-72deee20f840" (UID: "b7df5e06-8248-4312-8abc-72deee20f840"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:44.039822 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.039634 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7df5e06-8248-4312-8abc-72deee20f840-istio-envoy" pod "b7df5e06-8248-4312-8abc-72deee20f840" (UID: "b7df5e06-8248-4312-8abc-72deee20f840"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:44.113954 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.113921 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7df5e06-8248-4312-8abc-72deee20f840-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:44.113954 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.113949 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpg2g\" (UniqueName: \"kubernetes.io/projected/b7df5e06-8248-4312-8abc-72deee20f840-kube-api-access-gpg2g\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:44.113954 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.113959 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7df5e06-8248-4312-8abc-72deee20f840-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:44.114225 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.113968 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7df5e06-8248-4312-8abc-72deee20f840-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:44.114225 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.113976 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7df5e06-8248-4312-8abc-72deee20f840-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:44.114225 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.113984 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7df5e06-8248-4312-8abc-72deee20f840-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:44.114225 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.113991 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7df5e06-8248-4312-8abc-72deee20f840-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:44.351872 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.351833 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2"] Oct 30 18:36:44.361549 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.361515 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-fcxl2"] Oct 30 18:36:44.772784 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.772756 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ec94a8b-b5h8z"] Oct 30 18:36:44.773448 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.773433 4442 cpu_manager.go:414] "containerMap: removing container" podUID="7c61a56f-c91c-4e23-8abc-8b28d245b733" containerName="test-curl-7616-b9f8ed" Oct 30 18:36:44.773510 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.773449 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c61a56f-c91c-4e23-8abc-8b28d245b733" containerName="test-curl-7616-b9f8ed" Oct 30 18:36:44.773510 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.773461 4442 cpu_manager.go:414] "containerMap: removing container" podUID="b7df5e06-8248-4312-8abc-72deee20f840" containerName="istio-proxy" Oct 30 18:36:44.773510 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.773466 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7df5e06-8248-4312-8abc-72deee20f840" containerName="istio-proxy" Oct 30 18:36:44.773634 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.773558 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c61a56f-c91c-4e23-8abc-8b28d245b733" containerName="test-curl-7616-b9f8ed" Oct 30 18:36:44.773634 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.773566 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7df5e06-8248-4312-8abc-72deee20f840" containerName="istio-proxy" Oct 30 18:36:44.778572 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.778550 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec94a8b-b5h8z" Oct 30 18:36:44.793495 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.793473 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:36:44.795311 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.795285 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ec94a8b-b5h8z"] Oct 30 18:36:44.795311 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.795301 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:36:44.826672 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.826644 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9579478-0779-42e0-9073-6175f079d629-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ec94a8b-b5h8z\" (UID: \"a9579478-0779-42e0-9073-6175f079d629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec94a8b-b5h8z" Oct 30 18:36:44.826818 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.826686 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rrjj\" (UniqueName: \"kubernetes.io/projected/a9579478-0779-42e0-9073-6175f079d629-kube-api-access-9rrjj\") pod \"maas-api-revoke-sub-2ec94a8b-b5h8z\" (UID: \"a9579478-0779-42e0-9073-6175f079d629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec94a8b-b5h8z" Oct 30 18:36:44.826904 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.826822 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9579478-0779-42e0-9073-6175f079d629-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ec94a8b-b5h8z\" (UID: \"a9579478-0779-42e0-9073-6175f079d629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec94a8b-b5h8z" Oct 30 18:36:44.927827 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.927792 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9579478-0779-42e0-9073-6175f079d629-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ec94a8b-b5h8z\" (UID: \"a9579478-0779-42e0-9073-6175f079d629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec94a8b-b5h8z" Oct 30 18:36:44.927827 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.927834 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rrjj\" (UniqueName: \"kubernetes.io/projected/a9579478-0779-42e0-9073-6175f079d629-kube-api-access-9rrjj\") pod \"maas-api-revoke-sub-2ec94a8b-b5h8z\" (UID: \"a9579478-0779-42e0-9073-6175f079d629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec94a8b-b5h8z" Oct 30 18:36:44.928422 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.927881 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9579478-0779-42e0-9073-6175f079d629-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ec94a8b-b5h8z\" (UID: \"a9579478-0779-42e0-9073-6175f079d629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec94a8b-b5h8z" Oct 30 18:36:44.928640 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.928613 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9579478-0779-42e0-9073-6175f079d629-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ec94a8b-b5h8z\" (UID: \"a9579478-0779-42e0-9073-6175f079d629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec94a8b-b5h8z" Oct 30 18:36:44.930542 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.930518 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9579478-0779-42e0-9073-6175f079d629-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ec94a8b-b5h8z\" (UID: \"a9579478-0779-42e0-9073-6175f079d629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec94a8b-b5h8z" Oct 30 18:36:44.962271 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:44.962199 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rrjj\" (UniqueName: \"kubernetes.io/projected/a9579478-0779-42e0-9073-6175f079d629-kube-api-access-9rrjj\") pod \"maas-api-revoke-sub-2ec94a8b-b5h8z\" (UID: \"a9579478-0779-42e0-9073-6175f079d629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec94a8b-b5h8z" Oct 30 18:36:45.011228 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:45.011197 4442 generic.go:356] "Generic (PLEG): container finished" podID="7c61a56f-c91c-4e23-8abc-8b28d245b733" containerID="7903685e0b4c39cd15c8e4583164bf9c90abd96dfd5e0397ae48d69477e8e916" exitCode=137 Oct 30 18:36:45.036524 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:45.036458 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7616-b9f8ed" Oct 30 18:36:45.068080 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:45.068046 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7df5e06-8248-4312-8abc-72deee20f840" path="/var/lib/kubelet/pods/b7df5e06-8248-4312-8abc-72deee20f840/volumes" Oct 30 18:36:45.092882 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:45.092854 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec94a8b-b5h8z" Oct 30 18:36:45.130278 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:45.130243 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c61a56f-c91c-4e23-8abc-8b28d245b733-kube-api-access-x4kv6\" (UniqueName: \"kubernetes.io/projected/7c61a56f-c91c-4e23-8abc-8b28d245b733-kube-api-access-x4kv6\") pod \"7c61a56f-c91c-4e23-8abc-8b28d245b733\" (UID: \"7c61a56f-c91c-4e23-8abc-8b28d245b733\") " Oct 30 18:36:45.133389 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:45.133364 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c61a56f-c91c-4e23-8abc-8b28d245b733-kube-api-access-x4kv6" pod "7c61a56f-c91c-4e23-8abc-8b28d245b733" (UID: "7c61a56f-c91c-4e23-8abc-8b28d245b733"). InnerVolumeSpecName "kube-api-access-x4kv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:45.231810 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:45.231772 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4kv6\" (UniqueName: \"kubernetes.io/projected/7c61a56f-c91c-4e23-8abc-8b28d245b733-kube-api-access-x4kv6\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:45.400485 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:45.400453 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ec94a8b-b5h8z"] Oct 30 18:36:45.400910 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:36:45.400878 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9579478_0779_42e0_9073_6175f079d629.slice/crio-b88ad001cba685fa642a53492efae757cd9a87b8c4042908d1dde1c19ab9a167 WatchSource:0}: Error finding container b88ad001cba685fa642a53492efae757cd9a87b8c4042908d1dde1c19ab9a167: Status 404 returned error can't find the container with id b88ad001cba685fa642a53492efae757cd9a87b8c4042908d1dde1c19ab9a167 Oct 30 18:36:45.696527 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:45.696451 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8c97e49-rndq4"] Oct 30 18:36:45.700667 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:45.700650 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8c97e49-rndq4" Oct 30 18:36:45.719528 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:45.719494 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8c97e49-rndq4"] Oct 30 18:36:45.741065 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:45.741026 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8549bb2f-9746-497d-9620-51aa7fa7a218-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8c97e49-rndq4\" (UID: \"8549bb2f-9746-497d-9620-51aa7fa7a218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8c97e49-rndq4" Oct 30 18:36:45.741235 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:45.741094 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5spm\" (UniqueName: \"kubernetes.io/projected/8549bb2f-9746-497d-9620-51aa7fa7a218-kube-api-access-z5spm\") pod \"maas-api-revoke-sub-c8c97e49-rndq4\" (UID: \"8549bb2f-9746-497d-9620-51aa7fa7a218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8c97e49-rndq4" Oct 30 18:36:45.741377 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:45.741354 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8549bb2f-9746-497d-9620-51aa7fa7a218-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8c97e49-rndq4\" (UID: \"8549bb2f-9746-497d-9620-51aa7fa7a218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8c97e49-rndq4" Oct 30 18:36:45.842431 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:45.842389 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8549bb2f-9746-497d-9620-51aa7fa7a218-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8c97e49-rndq4\" (UID: \"8549bb2f-9746-497d-9620-51aa7fa7a218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8c97e49-rndq4" Oct 30 18:36:45.842603 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:45.842499 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8549bb2f-9746-497d-9620-51aa7fa7a218-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8c97e49-rndq4\" (UID: \"8549bb2f-9746-497d-9620-51aa7fa7a218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8c97e49-rndq4" Oct 30 18:36:45.842603 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:45.842541 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5spm\" (UniqueName: \"kubernetes.io/projected/8549bb2f-9746-497d-9620-51aa7fa7a218-kube-api-access-z5spm\") pod \"maas-api-revoke-sub-c8c97e49-rndq4\" (UID: \"8549bb2f-9746-497d-9620-51aa7fa7a218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8c97e49-rndq4" Oct 30 18:36:45.843169 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:45.843127 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8549bb2f-9746-497d-9620-51aa7fa7a218-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8c97e49-rndq4\" (UID: \"8549bb2f-9746-497d-9620-51aa7fa7a218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8c97e49-rndq4" Oct 30 18:36:45.845270 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:45.845243 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8549bb2f-9746-497d-9620-51aa7fa7a218-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8c97e49-rndq4\" (UID: \"8549bb2f-9746-497d-9620-51aa7fa7a218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8c97e49-rndq4" Oct 30 18:36:45.861946 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:45.861912 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5spm\" (UniqueName: \"kubernetes.io/projected/8549bb2f-9746-497d-9620-51aa7fa7a218-kube-api-access-z5spm\") pod \"maas-api-revoke-sub-c8c97e49-rndq4\" (UID: \"8549bb2f-9746-497d-9620-51aa7fa7a218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8c97e49-rndq4" Oct 30 18:36:46.015234 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:46.015123 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8c97e49-rndq4" Oct 30 18:36:46.021513 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:46.021489 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7616-b9f8ed" Oct 30 18:36:46.021513 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:46.021500 4442 scope.go:122] "RemoveContainer" containerID="7903685e0b4c39cd15c8e4583164bf9c90abd96dfd5e0397ae48d69477e8e916" Oct 30 18:36:46.023733 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:46.023704 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec94a8b-b5h8z" event={"ID":"a9579478-0779-42e0-9073-6175f079d629","Type":"ContainerStarted","Data":"cf5b0b0104958619f2e046184ba1917bdb849b385a01d616626beabd041bf064"} Oct 30 18:36:46.023855 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:46.023765 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec94a8b-b5h8z" event={"ID":"a9579478-0779-42e0-9073-6175f079d629","Type":"ContainerStarted","Data":"b88ad001cba685fa642a53492efae757cd9a87b8c4042908d1dde1c19ab9a167"} Oct 30 18:36:46.067306 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:46.064086 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec94a8b-b5h8z" podStartSLOduration=2.064067728 podStartE2EDuration="2.064067728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:46.06171251 +0000 UTC m=+2303.151774850" watchObservedRunningTime="2026-10-30 18:36:46.064067728 +0000 UTC m=+2303.154130068" Oct 30 18:36:46.387376 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:46.380306 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8c97e49-rndq4"] Oct 30 18:36:47.035374 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:47.035267 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8c97e49-rndq4" event={"ID":"8549bb2f-9746-497d-9620-51aa7fa7a218","Type":"ContainerStarted","Data":"d19ad0efef9d5662ef9730586e9353d2773424b163591fd71c7d5ebdacd82b12"} Oct 30 18:36:47.035374 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:47.035330 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8c97e49-rndq4" event={"ID":"8549bb2f-9746-497d-9620-51aa7fa7a218","Type":"ContainerStarted","Data":"c45859da558b9eadc35c4df288fe926e1f700eebfe170096beeaaa9ff9ff4884"} Oct 30 18:36:47.038058 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:47.038027 4442 generic.go:356] "Generic (PLEG): container finished" podID="a9579478-0779-42e0-9073-6175f079d629" containerID="cf5b0b0104958619f2e046184ba1917bdb849b385a01d616626beabd041bf064" exitCode=0 Oct 30 18:36:47.038214 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:47.038074 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec94a8b-b5h8z" event={"ID":"a9579478-0779-42e0-9073-6175f079d629","Type":"ContainerDied","Data":"cf5b0b0104958619f2e046184ba1917bdb849b385a01d616626beabd041bf064"} Oct 30 18:36:47.074822 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:47.074526 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8c97e49-rndq4" podStartSLOduration=2.074511029 podStartE2EDuration="2.074511029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:47.074334368 +0000 UTC m=+2304.164396708" watchObservedRunningTime="2026-10-30 18:36:47.074511029 +0000 UTC m=+2304.164573370" Oct 30 18:36:47.081184 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:47.081121 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c61a56f-c91c-4e23-8abc-8b28d245b733" path="/var/lib/kubelet/pods/7c61a56f-c91c-4e23-8abc-8b28d245b733/volumes" Oct 30 18:36:47.563190 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:36:47.563109 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:36:48.048509 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:48.048475 4442 generic.go:356] "Generic (PLEG): container finished" podID="8549bb2f-9746-497d-9620-51aa7fa7a218" containerID="d19ad0efef9d5662ef9730586e9353d2773424b163591fd71c7d5ebdacd82b12" exitCode=0 Oct 30 18:36:48.049043 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:48.048560 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8c97e49-rndq4" event={"ID":"8549bb2f-9746-497d-9620-51aa7fa7a218","Type":"ContainerDied","Data":"d19ad0efef9d5662ef9730586e9353d2773424b163591fd71c7d5ebdacd82b12"} Oct 30 18:36:48.314490 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:48.314424 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec94a8b-b5h8z" Oct 30 18:36:48.398921 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:48.398889 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9579478-0779-42e0-9073-6175f079d629-kube-api-access-9rrjj\" (UniqueName: \"kubernetes.io/projected/a9579478-0779-42e0-9073-6175f079d629-kube-api-access-9rrjj\") pod \"a9579478-0779-42e0-9073-6175f079d629\" (UID: \"a9579478-0779-42e0-9073-6175f079d629\") " Oct 30 18:36:48.399244 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:48.399035 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9579478-0779-42e0-9073-6175f079d629-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9579478-0779-42e0-9073-6175f079d629-maas-api-xks-ca\") pod \"a9579478-0779-42e0-9073-6175f079d629\" (UID: \"a9579478-0779-42e0-9073-6175f079d629\") " Oct 30 18:36:48.399244 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:48.399119 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9579478-0779-42e0-9073-6175f079d629-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9579478-0779-42e0-9073-6175f079d629-maas-api-openshift-ca\") pod \"a9579478-0779-42e0-9073-6175f079d629\" (UID: \"a9579478-0779-42e0-9073-6175f079d629\") " Oct 30 18:36:48.399599 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:48.399567 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9579478-0779-42e0-9073-6175f079d629-maas-api-openshift-ca" pod "a9579478-0779-42e0-9073-6175f079d629" (UID: "a9579478-0779-42e0-9073-6175f079d629"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:36:48.399761 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:48.399736 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9579478-0779-42e0-9073-6175f079d629-maas-api-openshift-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:48.402429 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:48.402400 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9579478-0779-42e0-9073-6175f079d629-maas-api-xks-ca" pod "a9579478-0779-42e0-9073-6175f079d629" (UID: "a9579478-0779-42e0-9073-6175f079d629"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:36:48.402595 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:48.402574 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9579478-0779-42e0-9073-6175f079d629-kube-api-access-9rrjj" pod "a9579478-0779-42e0-9073-6175f079d629" (UID: "a9579478-0779-42e0-9073-6175f079d629"). InnerVolumeSpecName "kube-api-access-9rrjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:48.500700 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:48.500669 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rrjj\" (UniqueName: \"kubernetes.io/projected/a9579478-0779-42e0-9073-6175f079d629-kube-api-access-9rrjj\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:48.500700 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:48.500699 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9579478-0779-42e0-9073-6175f079d629-maas-api-xks-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:49.061004 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:49.060977 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec94a8b-b5h8z" Oct 30 18:36:49.068896 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:49.068860 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec94a8b-b5h8z" event={"ID":"a9579478-0779-42e0-9073-6175f079d629","Type":"ContainerDied","Data":"b88ad001cba685fa642a53492efae757cd9a87b8c4042908d1dde1c19ab9a167"} Oct 30 18:36:49.069091 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:49.068923 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b88ad001cba685fa642a53492efae757cd9a87b8c4042908d1dde1c19ab9a167" Oct 30 18:36:49.224219 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:49.224191 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8c97e49-rndq4" Oct 30 18:36:49.311596 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:49.311487 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8549bb2f-9746-497d-9620-51aa7fa7a218-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8549bb2f-9746-497d-9620-51aa7fa7a218-maas-api-xks-ca\") pod \"8549bb2f-9746-497d-9620-51aa7fa7a218\" (UID: \"8549bb2f-9746-497d-9620-51aa7fa7a218\") " Oct 30 18:36:49.311779 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:49.311656 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8549bb2f-9746-497d-9620-51aa7fa7a218-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8549bb2f-9746-497d-9620-51aa7fa7a218-maas-api-openshift-ca\") pod \"8549bb2f-9746-497d-9620-51aa7fa7a218\" (UID: \"8549bb2f-9746-497d-9620-51aa7fa7a218\") " Oct 30 18:36:49.311852 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:49.311838 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8549bb2f-9746-497d-9620-51aa7fa7a218-kube-api-access-z5spm\" (UniqueName: \"kubernetes.io/projected/8549bb2f-9746-497d-9620-51aa7fa7a218-kube-api-access-z5spm\") pod \"8549bb2f-9746-497d-9620-51aa7fa7a218\" (UID: \"8549bb2f-9746-497d-9620-51aa7fa7a218\") " Oct 30 18:36:49.312064 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:49.312038 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8549bb2f-9746-497d-9620-51aa7fa7a218-maas-api-openshift-ca" pod "8549bb2f-9746-497d-9620-51aa7fa7a218" (UID: "8549bb2f-9746-497d-9620-51aa7fa7a218"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:36:49.312477 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:49.312455 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8549bb2f-9746-497d-9620-51aa7fa7a218-maas-api-openshift-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:49.314289 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:49.314261 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8549bb2f-9746-497d-9620-51aa7fa7a218-kube-api-access-z5spm" pod "8549bb2f-9746-497d-9620-51aa7fa7a218" (UID: "8549bb2f-9746-497d-9620-51aa7fa7a218"). InnerVolumeSpecName "kube-api-access-z5spm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:49.314434 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:49.314413 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8549bb2f-9746-497d-9620-51aa7fa7a218-maas-api-xks-ca" pod "8549bb2f-9746-497d-9620-51aa7fa7a218" (UID: "8549bb2f-9746-497d-9620-51aa7fa7a218"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:36:49.414035 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:49.414003 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5spm\" (UniqueName: \"kubernetes.io/projected/8549bb2f-9746-497d-9620-51aa7fa7a218-kube-api-access-z5spm\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:49.414035 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:49.414030 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8549bb2f-9746-497d-9620-51aa7fa7a218-maas-api-xks-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:50.073014 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:50.072987 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8c97e49-rndq4" Oct 30 18:36:50.073439 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:50.073028 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8c97e49-rndq4" event={"ID":"8549bb2f-9746-497d-9620-51aa7fa7a218","Type":"ContainerDied","Data":"c45859da558b9eadc35c4df288fe926e1f700eebfe170096beeaaa9ff9ff4884"} Oct 30 18:36:50.073439 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:50.073075 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c45859da558b9eadc35c4df288fe926e1f700eebfe170096beeaaa9ff9ff4884" Oct 30 18:36:54.618091 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:54.618056 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs"] Oct 30 18:36:54.618807 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:54.618787 4442 cpu_manager.go:414] "containerMap: removing container" podUID="8549bb2f-9746-497d-9620-51aa7fa7a218" containerName="revoke-keys" Oct 30 18:36:54.618807 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:54.618806 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8549bb2f-9746-497d-9620-51aa7fa7a218" containerName="revoke-keys" Oct 30 18:36:54.618970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:54.618834 4442 cpu_manager.go:414] "containerMap: removing container" podUID="a9579478-0779-42e0-9073-6175f079d629" containerName="revoke-keys" Oct 30 18:36:54.618970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:54.618839 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9579478-0779-42e0-9073-6175f079d629" containerName="revoke-keys" Oct 30 18:36:54.618970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:54.618944 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9579478-0779-42e0-9073-6175f079d629" containerName="revoke-keys" Oct 30 18:36:54.618970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:54.618960 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="8549bb2f-9746-497d-9620-51aa7fa7a218" containerName="revoke-keys" Oct 30 18:36:54.622828 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:54.622807 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" Oct 30 18:36:54.644440 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:54.644410 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs"] Oct 30 18:36:54.649622 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:54.649596 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-236fc4-openshift-default-dockercfg-ggnqj\"" Oct 30 18:36:54.726157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:54.710527 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfbb6b47-1274-4d3d-8845-627898246065-istio-token\") pod \"e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs\" (UID: \"bfbb6b47-1274-4d3d-8845-627898246065\") " pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" Oct 30 18:36:54.726157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:54.710605 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfbb6b47-1274-4d3d-8845-627898246065-workload-socket\") pod \"e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs\" (UID: \"bfbb6b47-1274-4d3d-8845-627898246065\") " pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" Oct 30 18:36:54.726157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:54.710651 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfbb6b47-1274-4d3d-8845-627898246065-workload-certs\") pod \"e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs\" (UID: \"bfbb6b47-1274-4d3d-8845-627898246065\") " pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" Oct 30 18:36:54.726157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:54.710771 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5jhm\" (UniqueName: \"kubernetes.io/projected/bfbb6b47-1274-4d3d-8845-627898246065-kube-api-access-g5jhm\") pod \"e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs\" (UID: \"bfbb6b47-1274-4d3d-8845-627898246065\") " pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" Oct 30 18:36:54.726157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:54.710836 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfbb6b47-1274-4d3d-8845-627898246065-istiod-ca-cert\") pod \"e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs\" (UID: \"bfbb6b47-1274-4d3d-8845-627898246065\") " pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" Oct 30 18:36:54.726157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:54.710882 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfbb6b47-1274-4d3d-8845-627898246065-istio-envoy\") pod \"e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs\" (UID: \"bfbb6b47-1274-4d3d-8845-627898246065\") " pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" Oct 30 18:36:54.726157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:54.710934 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfbb6b47-1274-4d3d-8845-627898246065-credential-socket\") pod \"e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs\" (UID: \"bfbb6b47-1274-4d3d-8845-627898246065\") " pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" Oct 30 18:36:54.783159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:54.710986 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfbb6b47-1274-4d3d-8845-627898246065-istio-podinfo\") pod \"e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs\" (UID: \"bfbb6b47-1274-4d3d-8845-627898246065\") " pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" Oct 30 18:36:54.783159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:54.778308 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfbb6b47-1274-4d3d-8845-627898246065-istio-data\") pod \"e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs\" (UID: \"bfbb6b47-1274-4d3d-8845-627898246065\") " pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" Oct 30 18:36:54.879965 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:54.879866 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfbb6b47-1274-4d3d-8845-627898246065-istio-podinfo\") pod \"e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs\" (UID: \"bfbb6b47-1274-4d3d-8845-627898246065\") " pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" Oct 30 18:36:54.879965 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:54.879914 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfbb6b47-1274-4d3d-8845-627898246065-istio-data\") pod \"e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs\" (UID: \"bfbb6b47-1274-4d3d-8845-627898246065\") " pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" Oct 30 18:36:54.880276 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:54.879988 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfbb6b47-1274-4d3d-8845-627898246065-istio-token\") pod \"e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs\" (UID: \"bfbb6b47-1274-4d3d-8845-627898246065\") " pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" Oct 30 18:36:54.880276 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:54.880027 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfbb6b47-1274-4d3d-8845-627898246065-workload-socket\") pod \"e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs\" (UID: \"bfbb6b47-1274-4d3d-8845-627898246065\") " pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" Oct 30 18:36:54.880276 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:54.880058 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfbb6b47-1274-4d3d-8845-627898246065-workload-certs\") pod \"e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs\" (UID: \"bfbb6b47-1274-4d3d-8845-627898246065\") " pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" Oct 30 18:36:54.880276 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:54.880182 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5jhm\" (UniqueName: \"kubernetes.io/projected/bfbb6b47-1274-4d3d-8845-627898246065-kube-api-access-g5jhm\") pod \"e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs\" (UID: \"bfbb6b47-1274-4d3d-8845-627898246065\") " pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" Oct 30 18:36:54.880276 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:54.880245 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfbb6b47-1274-4d3d-8845-627898246065-istiod-ca-cert\") pod \"e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs\" (UID: \"bfbb6b47-1274-4d3d-8845-627898246065\") " pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" Oct 30 18:36:54.880547 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:54.880292 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfbb6b47-1274-4d3d-8845-627898246065-istio-envoy\") pod \"e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs\" (UID: \"bfbb6b47-1274-4d3d-8845-627898246065\") " pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" Oct 30 18:36:54.880547 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:54.880341 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfbb6b47-1274-4d3d-8845-627898246065-credential-socket\") pod \"e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs\" (UID: \"bfbb6b47-1274-4d3d-8845-627898246065\") " pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" Oct 30 18:36:54.880547 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:54.880431 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfbb6b47-1274-4d3d-8845-627898246065-istio-data\") pod \"e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs\" (UID: \"bfbb6b47-1274-4d3d-8845-627898246065\") " pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" Oct 30 18:36:54.880547 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:54.880443 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfbb6b47-1274-4d3d-8845-627898246065-workload-certs\") pod \"e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs\" (UID: \"bfbb6b47-1274-4d3d-8845-627898246065\") " pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" Oct 30 18:36:54.880713 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:54.880688 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfbb6b47-1274-4d3d-8845-627898246065-credential-socket\") pod \"e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs\" (UID: \"bfbb6b47-1274-4d3d-8845-627898246065\") " pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" Oct 30 18:36:54.880971 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:54.880945 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfbb6b47-1274-4d3d-8845-627898246065-workload-socket\") pod \"e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs\" (UID: \"bfbb6b47-1274-4d3d-8845-627898246065\") " pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" Oct 30 18:36:54.881053 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:54.881027 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfbb6b47-1274-4d3d-8845-627898246065-istiod-ca-cert\") pod \"e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs\" (UID: \"bfbb6b47-1274-4d3d-8845-627898246065\") " pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" Oct 30 18:36:54.882889 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:54.882864 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfbb6b47-1274-4d3d-8845-627898246065-istio-envoy\") pod \"e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs\" (UID: \"bfbb6b47-1274-4d3d-8845-627898246065\") " pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" Oct 30 18:36:54.883328 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:54.883307 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfbb6b47-1274-4d3d-8845-627898246065-istio-podinfo\") pod \"e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs\" (UID: \"bfbb6b47-1274-4d3d-8845-627898246065\") " pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" Oct 30 18:36:54.900517 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:54.900487 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfbb6b47-1274-4d3d-8845-627898246065-istio-token\") pod \"e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs\" (UID: \"bfbb6b47-1274-4d3d-8845-627898246065\") " pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" Oct 30 18:36:54.902498 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:54.902475 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5jhm\" (UniqueName: \"kubernetes.io/projected/bfbb6b47-1274-4d3d-8845-627898246065-kube-api-access-g5jhm\") pod \"e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs\" (UID: \"bfbb6b47-1274-4d3d-8845-627898246065\") " pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" Oct 30 18:36:54.959570 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:54.955478 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" Oct 30 18:36:55.288704 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:55.288625 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db7d40cf-dd9pg"] Oct 30 18:36:55.293303 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:55.293283 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7d40cf-dd9pg" Oct 30 18:36:55.319227 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:55.319184 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db7d40cf-dd9pg"] Oct 30 18:36:55.322560 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:55.322528 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:36:55.326590 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:55.326557 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:36:55.332249 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:55.332221 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs"] Oct 30 18:36:55.345318 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:36:55.344930 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfbb6b47_1274_4d3d_8845_627898246065.slice/crio-6984700f5380d2c175c5f8f7bbc26284ab4671adcb3988d48fa2b302d7589364 WatchSource:0}: Error finding container 6984700f5380d2c175c5f8f7bbc26284ab4671adcb3988d48fa2b302d7589364: Status 404 returned error can't find the container with id 6984700f5380d2c175c5f8f7bbc26284ab4671adcb3988d48fa2b302d7589364 Oct 30 18:36:55.350330 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:55.350285 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:36:55.350438 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:55.350378 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:36:55.350438 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:55.350421 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:36:55.390551 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:55.390524 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc54b885-2ec0-4053-b535-54db5f1ca5a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db7d40cf-dd9pg\" (UID: \"fc54b885-2ec0-4053-b535-54db5f1ca5a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7d40cf-dd9pg" Oct 30 18:36:55.390695 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:55.390653 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc54b885-2ec0-4053-b535-54db5f1ca5a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db7d40cf-dd9pg\" (UID: \"fc54b885-2ec0-4053-b535-54db5f1ca5a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7d40cf-dd9pg" Oct 30 18:36:55.390758 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:55.390698 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bsts\" (UniqueName: \"kubernetes.io/projected/fc54b885-2ec0-4053-b535-54db5f1ca5a7-kube-api-access-6bsts\") pod \"maas-api-revoke-sub-db7d40cf-dd9pg\" (UID: \"fc54b885-2ec0-4053-b535-54db5f1ca5a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7d40cf-dd9pg" Oct 30 18:36:55.492826 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:55.492787 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bsts\" (UniqueName: \"kubernetes.io/projected/fc54b885-2ec0-4053-b535-54db5f1ca5a7-kube-api-access-6bsts\") pod \"maas-api-revoke-sub-db7d40cf-dd9pg\" (UID: \"fc54b885-2ec0-4053-b535-54db5f1ca5a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7d40cf-dd9pg" Oct 30 18:36:55.493052 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:55.493035 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc54b885-2ec0-4053-b535-54db5f1ca5a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db7d40cf-dd9pg\" (UID: \"fc54b885-2ec0-4053-b535-54db5f1ca5a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7d40cf-dd9pg" Oct 30 18:36:55.493184 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:55.493124 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc54b885-2ec0-4053-b535-54db5f1ca5a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db7d40cf-dd9pg\" (UID: \"fc54b885-2ec0-4053-b535-54db5f1ca5a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7d40cf-dd9pg" Oct 30 18:36:55.493801 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:55.493781 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc54b885-2ec0-4053-b535-54db5f1ca5a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db7d40cf-dd9pg\" (UID: \"fc54b885-2ec0-4053-b535-54db5f1ca5a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7d40cf-dd9pg" Oct 30 18:36:55.495800 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:55.495776 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc54b885-2ec0-4053-b535-54db5f1ca5a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db7d40cf-dd9pg\" (UID: \"fc54b885-2ec0-4053-b535-54db5f1ca5a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7d40cf-dd9pg" Oct 30 18:36:55.513469 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:55.513436 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bsts\" (UniqueName: \"kubernetes.io/projected/fc54b885-2ec0-4053-b535-54db5f1ca5a7-kube-api-access-6bsts\") pod \"maas-api-revoke-sub-db7d40cf-dd9pg\" (UID: \"fc54b885-2ec0-4053-b535-54db5f1ca5a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7d40cf-dd9pg" Oct 30 18:36:55.608056 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:55.607977 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7d40cf-dd9pg" Oct 30 18:36:55.971635 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:55.971599 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db7d40cf-dd9pg"] Oct 30 18:36:56.133212 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:56.133169 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7d40cf-dd9pg" event={"ID":"fc54b885-2ec0-4053-b535-54db5f1ca5a7","Type":"ContainerStarted","Data":"e3ce08cbcc97128b05750044f93e13949f449725a7f0e6bae352fe7846f32e6f"} Oct 30 18:36:56.135802 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:56.135768 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" event={"ID":"bfbb6b47-1274-4d3d-8845-627898246065","Type":"ContainerStarted","Data":"09643b155ae61d3db212a12fe01e8b94e6d8211a3180916351b902bcce2dad97"} Oct 30 18:36:56.135982 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:56.135821 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" event={"ID":"bfbb6b47-1274-4d3d-8845-627898246065","Type":"ContainerStarted","Data":"6984700f5380d2c175c5f8f7bbc26284ab4671adcb3988d48fa2b302d7589364"} Oct 30 18:36:56.184543 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:56.184488 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" podStartSLOduration=2.184472327 podStartE2EDuration="2.184472327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:56.179258522 +0000 UTC m=+2313.269320876" watchObservedRunningTime="2026-10-30 18:36:56.184472327 +0000 UTC m=+2313.274534665" Oct 30 18:36:56.956514 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:56.956453 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" Oct 30 18:36:56.964200 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:56.964171 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" Oct 30 18:36:57.147062 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:57.147025 4442 generic.go:356] "Generic (PLEG): container finished" podID="fc54b885-2ec0-4053-b535-54db5f1ca5a7" containerID="9eae321ca072e1d2ac322f9901fa58096bda69ebef9ed3ece8e41e6c61604fd9" exitCode=0 Oct 30 18:36:57.147612 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:57.147112 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7d40cf-dd9pg" event={"ID":"fc54b885-2ec0-4053-b535-54db5f1ca5a7","Type":"ContainerDied","Data":"9eae321ca072e1d2ac322f9901fa58096bda69ebef9ed3ece8e41e6c61604fd9"} Oct 30 18:36:57.147702 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:57.147683 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" Oct 30 18:36:57.148850 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:57.148828 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" Oct 30 18:36:58.337287 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:58.337263 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7d40cf-dd9pg" Oct 30 18:36:58.362875 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:58.362844 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc54b885-2ec0-4053-b535-54db5f1ca5a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc54b885-2ec0-4053-b535-54db5f1ca5a7-maas-api-xks-ca\") pod \"fc54b885-2ec0-4053-b535-54db5f1ca5a7\" (UID: \"fc54b885-2ec0-4053-b535-54db5f1ca5a7\") " Oct 30 18:36:58.363034 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:58.362892 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc54b885-2ec0-4053-b535-54db5f1ca5a7-kube-api-access-6bsts\" (UniqueName: \"kubernetes.io/projected/fc54b885-2ec0-4053-b535-54db5f1ca5a7-kube-api-access-6bsts\") pod \"fc54b885-2ec0-4053-b535-54db5f1ca5a7\" (UID: \"fc54b885-2ec0-4053-b535-54db5f1ca5a7\") " Oct 30 18:36:58.363034 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:58.362971 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc54b885-2ec0-4053-b535-54db5f1ca5a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc54b885-2ec0-4053-b535-54db5f1ca5a7-maas-api-openshift-ca\") pod \"fc54b885-2ec0-4053-b535-54db5f1ca5a7\" (UID: \"fc54b885-2ec0-4053-b535-54db5f1ca5a7\") " Oct 30 18:36:58.363339 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:58.363316 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc54b885-2ec0-4053-b535-54db5f1ca5a7-maas-api-openshift-ca" pod "fc54b885-2ec0-4053-b535-54db5f1ca5a7" (UID: "fc54b885-2ec0-4053-b535-54db5f1ca5a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:36:58.363475 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:58.363461 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc54b885-2ec0-4053-b535-54db5f1ca5a7-maas-api-openshift-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:58.365121 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:58.365098 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc54b885-2ec0-4053-b535-54db5f1ca5a7-maas-api-xks-ca" pod "fc54b885-2ec0-4053-b535-54db5f1ca5a7" (UID: "fc54b885-2ec0-4053-b535-54db5f1ca5a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:36:58.365280 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:58.365259 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc54b885-2ec0-4053-b535-54db5f1ca5a7-kube-api-access-6bsts" pod "fc54b885-2ec0-4053-b535-54db5f1ca5a7" (UID: "fc54b885-2ec0-4053-b535-54db5f1ca5a7"). InnerVolumeSpecName "kube-api-access-6bsts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:58.464972 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:58.464856 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc54b885-2ec0-4053-b535-54db5f1ca5a7-maas-api-xks-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:58.464972 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:58.464885 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bsts\" (UniqueName: \"kubernetes.io/projected/fc54b885-2ec0-4053-b535-54db5f1ca5a7-kube-api-access-6bsts\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:36:59.166876 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:59.166844 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7d40cf-dd9pg" Oct 30 18:36:59.167025 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:59.166875 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7d40cf-dd9pg" event={"ID":"fc54b885-2ec0-4053-b535-54db5f1ca5a7","Type":"ContainerDied","Data":"e3ce08cbcc97128b05750044f93e13949f449725a7f0e6bae352fe7846f32e6f"} Oct 30 18:36:59.167025 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:36:59.166922 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3ce08cbcc97128b05750044f93e13949f449725a7f0e6bae352fe7846f32e6f" Oct 30 18:37:00.531030 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:00.530996 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-236fc4-6c5fd847b7-xxv8l"] Oct 30 18:37:00.531810 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:00.531788 4442 cpu_manager.go:414] "containerMap: removing container" podUID="fc54b885-2ec0-4053-b535-54db5f1ca5a7" containerName="revoke-keys" Oct 30 18:37:00.531867 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:00.531815 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc54b885-2ec0-4053-b535-54db5f1ca5a7" containerName="revoke-keys" Oct 30 18:37:00.532029 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:00.532016 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc54b885-2ec0-4053-b535-54db5f1ca5a7" containerName="revoke-keys" Oct 30 18:37:00.535738 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:00.535723 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-236fc4-6c5fd847b7-xxv8l" Oct 30 18:37:00.572186 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:00.564033 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-236fc4-dockercfg-n8n5h\"" Oct 30 18:37:00.572186 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:00.564078 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-236fc4\"" Oct 30 18:37:00.572186 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:00.571074 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-236fc4-6c5fd847b7-xxv8l"] Oct 30 18:37:00.594284 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:00.594251 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97f2a51a-33c7-4869-8315-f5c266adac7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-236fc4-6c5fd847b7-xxv8l\" (UID: \"97f2a51a-33c7-4869-8315-f5c266adac7a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-236fc4-6c5fd847b7-xxv8l" Oct 30 18:37:00.594423 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:00.594325 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9gtn\" (UniqueName: \"kubernetes.io/projected/97f2a51a-33c7-4869-8315-f5c266adac7a-kube-api-access-g9gtn\") pod \"payload-pre-processing-e2e-shared-a-w0-236fc4-6c5fd847b7-xxv8l\" (UID: \"97f2a51a-33c7-4869-8315-f5c266adac7a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-236fc4-6c5fd847b7-xxv8l" Oct 30 18:37:00.698160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:00.695333 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-236fc4-88f5755fb-fskbl"] Oct 30 18:37:00.748158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:00.731262 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9gtn\" (UniqueName: \"kubernetes.io/projected/97f2a51a-33c7-4869-8315-f5c266adac7a-kube-api-access-g9gtn\") pod \"payload-pre-processing-e2e-shared-a-w0-236fc4-6c5fd847b7-xxv8l\" (UID: \"97f2a51a-33c7-4869-8315-f5c266adac7a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-236fc4-6c5fd847b7-xxv8l" Oct 30 18:37:00.748158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:00.731559 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97f2a51a-33c7-4869-8315-f5c266adac7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-236fc4-6c5fd847b7-xxv8l\" (UID: \"97f2a51a-33c7-4869-8315-f5c266adac7a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-236fc4-6c5fd847b7-xxv8l" Oct 30 18:37:00.748158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:00.740025 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-236fc4-88f5755fb-fskbl"] Oct 30 18:37:00.748158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:00.740184 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-236fc4-88f5755fb-fskbl" Oct 30 18:37:00.748158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:00.741070 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97f2a51a-33c7-4869-8315-f5c266adac7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-236fc4-6c5fd847b7-xxv8l\" (UID: \"97f2a51a-33c7-4869-8315-f5c266adac7a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-236fc4-6c5fd847b7-xxv8l" Oct 30 18:37:00.847202 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:00.843109 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9gtn\" (UniqueName: \"kubernetes.io/projected/97f2a51a-33c7-4869-8315-f5c266adac7a-kube-api-access-g9gtn\") pod \"payload-pre-processing-e2e-shared-a-w0-236fc4-6c5fd847b7-xxv8l\" (UID: \"97f2a51a-33c7-4869-8315-f5c266adac7a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-236fc4-6c5fd847b7-xxv8l" Oct 30 18:37:00.869189 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:00.850652 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-236fc4-6c5fd847b7-xxv8l" Oct 30 18:37:00.936661 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:00.936614 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4f4443c-7b80-47d0-8bbf-3e6ef11f437b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-236fc4-88f5755fb-fskbl\" (UID: \"e4f4443c-7b80-47d0-8bbf-3e6ef11f437b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-236fc4-88f5755fb-fskbl" Oct 30 18:37:00.936661 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:00.936664 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5c22\" (UniqueName: \"kubernetes.io/projected/e4f4443c-7b80-47d0-8bbf-3e6ef11f437b-kube-api-access-r5c22\") pod \"payload-processing-e2e-shared-a-w0-236fc4-88f5755fb-fskbl\" (UID: \"e4f4443c-7b80-47d0-8bbf-3e6ef11f437b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-236fc4-88f5755fb-fskbl" Oct 30 18:37:01.038839 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:01.038538 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4f4443c-7b80-47d0-8bbf-3e6ef11f437b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-236fc4-88f5755fb-fskbl\" (UID: \"e4f4443c-7b80-47d0-8bbf-3e6ef11f437b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-236fc4-88f5755fb-fskbl" Oct 30 18:37:01.039028 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:01.038970 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5c22\" (UniqueName: \"kubernetes.io/projected/e4f4443c-7b80-47d0-8bbf-3e6ef11f437b-kube-api-access-r5c22\") pod \"payload-processing-e2e-shared-a-w0-236fc4-88f5755fb-fskbl\" (UID: \"e4f4443c-7b80-47d0-8bbf-3e6ef11f437b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-236fc4-88f5755fb-fskbl" Oct 30 18:37:01.039765 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:01.039743 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4f4443c-7b80-47d0-8bbf-3e6ef11f437b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-236fc4-88f5755fb-fskbl\" (UID: \"e4f4443c-7b80-47d0-8bbf-3e6ef11f437b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-236fc4-88f5755fb-fskbl" Oct 30 18:37:01.060105 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:01.060082 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5c22\" (UniqueName: \"kubernetes.io/projected/e4f4443c-7b80-47d0-8bbf-3e6ef11f437b-kube-api-access-r5c22\") pod \"payload-processing-e2e-shared-a-w0-236fc4-88f5755fb-fskbl\" (UID: \"e4f4443c-7b80-47d0-8bbf-3e6ef11f437b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-236fc4-88f5755fb-fskbl" Oct 30 18:37:01.184337 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:01.184305 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-236fc4-88f5755fb-fskbl" Oct 30 18:37:01.192975 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:01.192952 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-236fc4-6c5fd847b7-xxv8l"] Oct 30 18:37:01.202418 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:37:01.202384 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97f2a51a_33c7_4869_8315_f5c266adac7a.slice/crio-fc03a88338214590050a5b2be26a0bd429ebf3936716a8c2c5d43996bff5eda3 WatchSource:0}: Error finding container fc03a88338214590050a5b2be26a0bd429ebf3936716a8c2c5d43996bff5eda3: Status 404 returned error can't find the container with id fc03a88338214590050a5b2be26a0bd429ebf3936716a8c2c5d43996bff5eda3 Oct 30 18:37:01.512771 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:37:01.512733 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4f4443c_7b80_47d0_8bbf_3e6ef11f437b.slice/crio-5c81304d529d9687312b2f6a25a169a5c11e549d683f944d07b7bcceddad696e WatchSource:0}: Error finding container 5c81304d529d9687312b2f6a25a169a5c11e549d683f944d07b7bcceddad696e: Status 404 returned error can't find the container with id 5c81304d529d9687312b2f6a25a169a5c11e549d683f944d07b7bcceddad696e Oct 30 18:37:01.513165 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:01.513126 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-236fc4-88f5755fb-fskbl"] Oct 30 18:37:02.217218 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:02.217180 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-236fc4-88f5755fb-fskbl" event={"ID":"e4f4443c-7b80-47d0-8bbf-3e6ef11f437b","Type":"ContainerStarted","Data":"07610df0022ae7c8f2f3cdddea980b64fac326104d01edae512e864f05bfd075"} Oct 30 18:37:02.217717 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:02.217271 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-236fc4-88f5755fb-fskbl" event={"ID":"e4f4443c-7b80-47d0-8bbf-3e6ef11f437b","Type":"ContainerStarted","Data":"5c81304d529d9687312b2f6a25a169a5c11e549d683f944d07b7bcceddad696e"} Oct 30 18:37:02.217717 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:02.217377 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-236fc4-88f5755fb-fskbl" Oct 30 18:37:02.219188 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:02.219158 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-236fc4-6c5fd847b7-xxv8l" event={"ID":"97f2a51a-33c7-4869-8315-f5c266adac7a","Type":"ContainerStarted","Data":"d615caf88a30f4ad70b0e2c68c2b7c267d67ab7f45e8558f11be071a3f15430b"} Oct 30 18:37:02.219347 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:02.219204 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-236fc4-6c5fd847b7-xxv8l" event={"ID":"97f2a51a-33c7-4869-8315-f5c266adac7a","Type":"ContainerStarted","Data":"fc03a88338214590050a5b2be26a0bd429ebf3936716a8c2c5d43996bff5eda3"} Oct 30 18:37:02.219347 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:02.219245 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-236fc4-6c5fd847b7-xxv8l" Oct 30 18:37:02.258015 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:02.257953 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-236fc4-88f5755fb-fskbl" podStartSLOduration=2.257936706 podStartE2EDuration="2.257936706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:02.256639832 +0000 UTC m=+2319.346702171" watchObservedRunningTime="2026-10-30 18:37:02.257936706 +0000 UTC m=+2319.347999048" Oct 30 18:37:02.299955 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:02.299899 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-236fc4-6c5fd847b7-xxv8l" podStartSLOduration=2.299880397 podStartE2EDuration="2.299880397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:02.294973574 +0000 UTC m=+2319.385035913" watchObservedRunningTime="2026-10-30 18:37:02.299880397 +0000 UTC m=+2319.389942736" Oct 30 18:37:07.558487 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:07.558447 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-1a2436/auth-test-model-1a2436-kserve-547bf9b957-hhcgp"] Oct 30 18:37:08.699041 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:08.698999 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv"] Oct 30 18:37:08.699534 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:08.699268 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" podUID="8d7355a2-0978-497c-8229-f5a0f6ae194d" containerName="istio-proxy" containerID="cri-o://a8c9f2a9f1eb89996d239e8291466951c267e05bacb6ffa947340ae80ec9d58d" gracePeriod=30 Oct 30 18:37:08.711103 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:08.711066 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp"] Oct 30 18:37:08.711363 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:08.711336 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" podUID="a2399863-51b1-4b7a-acdd-ea28c9074e95" containerName="istio-proxy" containerID="cri-o://c19fecf022f3b46b5ba9a5ce7d8cc916830517e625d4a976ed22624d6c8ab02d" gracePeriod=30 Oct 30 18:37:08.715495 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:08.715470 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4"] Oct 30 18:37:08.715670 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:08.715651 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" podUID="253b93c5-40cc-444a-8836-06044e3d6dd4" containerName="istio-proxy" containerID="cri-o://59139cd792b7b832f56a135c3705848e70368a7884613449f94a12139f104d07" gracePeriod=30 Oct 30 18:37:08.721915 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:08.721891 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn"] Oct 30 18:37:08.722066 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:08.722047 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" podUID="e2065cdb-c364-4e80-addd-1055ed222931" containerName="istio-proxy" containerID="cri-o://ee8ebd7c9d53ab796aaf6e03c26bb9dd23913ba8495b8b9d0f766971d875b8ab" gracePeriod=30 Oct 30 18:37:13.233405 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:13.233376 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-236fc4-6c5fd847b7-xxv8l" Oct 30 18:37:13.233827 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:13.233458 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-236fc4-88f5755fb-fskbl" Oct 30 18:37:14.057196 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.050603 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" Oct 30 18:37:14.196189 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.196163 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" Oct 30 18:37:14.197685 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.197655 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/253b93c5-40cc-444a-8836-06044e3d6dd4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/253b93c5-40cc-444a-8836-06044e3d6dd4-workload-certs\") pod \"253b93c5-40cc-444a-8836-06044e3d6dd4\" (UID: \"253b93c5-40cc-444a-8836-06044e3d6dd4\") " Oct 30 18:37:14.197827 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.197703 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/253b93c5-40cc-444a-8836-06044e3d6dd4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/253b93c5-40cc-444a-8836-06044e3d6dd4-workload-socket\") pod \"253b93c5-40cc-444a-8836-06044e3d6dd4\" (UID: \"253b93c5-40cc-444a-8836-06044e3d6dd4\") " Oct 30 18:37:14.197827 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.197754 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d7355a2-0978-497c-8229-f5a0f6ae194d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7355a2-0978-497c-8229-f5a0f6ae194d-workload-socket\") pod \"8d7355a2-0978-497c-8229-f5a0f6ae194d\" (UID: \"8d7355a2-0978-497c-8229-f5a0f6ae194d\") " Oct 30 18:37:14.197827 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.197785 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d7355a2-0978-497c-8229-f5a0f6ae194d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d7355a2-0978-497c-8229-f5a0f6ae194d-workload-certs\") pod \"8d7355a2-0978-497c-8229-f5a0f6ae194d\" (UID: \"8d7355a2-0978-497c-8229-f5a0f6ae194d\") " Oct 30 18:37:14.197998 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.197886 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d7355a2-0978-497c-8229-f5a0f6ae194d-istio-token\" (UniqueName: \"kubernetes.io/projected/8d7355a2-0978-497c-8229-f5a0f6ae194d-istio-token\") pod \"8d7355a2-0978-497c-8229-f5a0f6ae194d\" (UID: \"8d7355a2-0978-497c-8229-f5a0f6ae194d\") " Oct 30 18:37:14.197998 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.197916 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/253b93c5-40cc-444a-8836-06044e3d6dd4-kube-api-access-qt96c\" (UniqueName: \"kubernetes.io/projected/253b93c5-40cc-444a-8836-06044e3d6dd4-kube-api-access-qt96c\") pod \"253b93c5-40cc-444a-8836-06044e3d6dd4\" (UID: \"253b93c5-40cc-444a-8836-06044e3d6dd4\") " Oct 30 18:37:14.197998 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.197934 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/253b93c5-40cc-444a-8836-06044e3d6dd4-workload-certs" pod "253b93c5-40cc-444a-8836-06044e3d6dd4" (UID: "253b93c5-40cc-444a-8836-06044e3d6dd4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:14.198276 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.198051 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d7355a2-0978-497c-8229-f5a0f6ae194d-workload-certs" pod "8d7355a2-0978-497c-8229-f5a0f6ae194d" (UID: "8d7355a2-0978-497c-8229-f5a0f6ae194d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:14.198276 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.198246 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/253b93c5-40cc-444a-8836-06044e3d6dd4-workload-socket" pod "253b93c5-40cc-444a-8836-06044e3d6dd4" (UID: "253b93c5-40cc-444a-8836-06044e3d6dd4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:14.198395 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.198304 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d7355a2-0978-497c-8229-f5a0f6ae194d-kube-api-access-xwfqq\" (UniqueName: \"kubernetes.io/projected/8d7355a2-0978-497c-8229-f5a0f6ae194d-kube-api-access-xwfqq\") pod \"8d7355a2-0978-497c-8229-f5a0f6ae194d\" (UID: \"8d7355a2-0978-497c-8229-f5a0f6ae194d\") " Oct 30 18:37:14.198395 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.198347 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/253b93c5-40cc-444a-8836-06044e3d6dd4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/253b93c5-40cc-444a-8836-06044e3d6dd4-istio-envoy\") pod \"253b93c5-40cc-444a-8836-06044e3d6dd4\" (UID: \"253b93c5-40cc-444a-8836-06044e3d6dd4\") " Oct 30 18:37:14.198522 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.198396 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/253b93c5-40cc-444a-8836-06044e3d6dd4-istio-token\" (UniqueName: \"kubernetes.io/projected/253b93c5-40cc-444a-8836-06044e3d6dd4-istio-token\") pod \"253b93c5-40cc-444a-8836-06044e3d6dd4\" (UID: \"253b93c5-40cc-444a-8836-06044e3d6dd4\") " Oct 30 18:37:14.198522 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.198511 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d7355a2-0978-497c-8229-f5a0f6ae194d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7355a2-0978-497c-8229-f5a0f6ae194d-credential-socket\") pod \"8d7355a2-0978-497c-8229-f5a0f6ae194d\" (UID: \"8d7355a2-0978-497c-8229-f5a0f6ae194d\") " Oct 30 18:37:14.198637 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.198514 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d7355a2-0978-497c-8229-f5a0f6ae194d-workload-socket" pod "8d7355a2-0978-497c-8229-f5a0f6ae194d" (UID: "8d7355a2-0978-497c-8229-f5a0f6ae194d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:14.198701 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.198687 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d7355a2-0978-497c-8229-f5a0f6ae194d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d7355a2-0978-497c-8229-f5a0f6ae194d-istio-data\") pod \"8d7355a2-0978-497c-8229-f5a0f6ae194d\" (UID: \"8d7355a2-0978-497c-8229-f5a0f6ae194d\") " Oct 30 18:37:14.198777 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.198736 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/253b93c5-40cc-444a-8836-06044e3d6dd4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/253b93c5-40cc-444a-8836-06044e3d6dd4-istio-podinfo\") pod \"253b93c5-40cc-444a-8836-06044e3d6dd4\" (UID: \"253b93c5-40cc-444a-8836-06044e3d6dd4\") " Oct 30 18:37:14.198838 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.198773 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d7355a2-0978-497c-8229-f5a0f6ae194d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d7355a2-0978-497c-8229-f5a0f6ae194d-istiod-ca-cert\") pod \"8d7355a2-0978-497c-8229-f5a0f6ae194d\" (UID: \"8d7355a2-0978-497c-8229-f5a0f6ae194d\") " Oct 30 18:37:14.198838 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.198792 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d7355a2-0978-497c-8229-f5a0f6ae194d-credential-socket" pod "8d7355a2-0978-497c-8229-f5a0f6ae194d" (UID: "8d7355a2-0978-497c-8229-f5a0f6ae194d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:14.198838 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.198826 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/253b93c5-40cc-444a-8836-06044e3d6dd4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/253b93c5-40cc-444a-8836-06044e3d6dd4-istiod-ca-cert\") pod \"253b93c5-40cc-444a-8836-06044e3d6dd4\" (UID: \"253b93c5-40cc-444a-8836-06044e3d6dd4\") " Oct 30 18:37:14.199000 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.198880 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/253b93c5-40cc-444a-8836-06044e3d6dd4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/253b93c5-40cc-444a-8836-06044e3d6dd4-credential-socket\") pod \"253b93c5-40cc-444a-8836-06044e3d6dd4\" (UID: \"253b93c5-40cc-444a-8836-06044e3d6dd4\") " Oct 30 18:37:14.199000 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.198945 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d7355a2-0978-497c-8229-f5a0f6ae194d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d7355a2-0978-497c-8229-f5a0f6ae194d-istio-podinfo\") pod \"8d7355a2-0978-497c-8229-f5a0f6ae194d\" (UID: \"8d7355a2-0978-497c-8229-f5a0f6ae194d\") " Oct 30 18:37:14.199000 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.198979 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/253b93c5-40cc-444a-8836-06044e3d6dd4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/253b93c5-40cc-444a-8836-06044e3d6dd4-istio-data\") pod \"253b93c5-40cc-444a-8836-06044e3d6dd4\" (UID: \"253b93c5-40cc-444a-8836-06044e3d6dd4\") " Oct 30 18:37:14.199192 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.199007 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d7355a2-0978-497c-8229-f5a0f6ae194d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d7355a2-0978-497c-8229-f5a0f6ae194d-istio-envoy\") pod \"8d7355a2-0978-497c-8229-f5a0f6ae194d\" (UID: \"8d7355a2-0978-497c-8229-f5a0f6ae194d\") " Oct 30 18:37:14.199259 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.199200 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d7355a2-0978-497c-8229-f5a0f6ae194d-istio-data" pod "8d7355a2-0978-497c-8229-f5a0f6ae194d" (UID: "8d7355a2-0978-497c-8229-f5a0f6ae194d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:14.199343 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.199319 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/253b93c5-40cc-444a-8836-06044e3d6dd4-credential-socket" pod "253b93c5-40cc-444a-8836-06044e3d6dd4" (UID: "253b93c5-40cc-444a-8836-06044e3d6dd4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:14.199424 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.199402 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/253b93c5-40cc-444a-8836-06044e3d6dd4-istio-data" pod "253b93c5-40cc-444a-8836-06044e3d6dd4" (UID: "253b93c5-40cc-444a-8836-06044e3d6dd4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:14.199731 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.199708 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d7355a2-0978-497c-8229-f5a0f6ae194d-istiod-ca-cert" pod "8d7355a2-0978-497c-8229-f5a0f6ae194d" (UID: "8d7355a2-0978-497c-8229-f5a0f6ae194d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:14.200186 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.200118 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/253b93c5-40cc-444a-8836-06044e3d6dd4-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:14.200285 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.200195 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/253b93c5-40cc-444a-8836-06044e3d6dd4-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:14.200285 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.200212 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7355a2-0978-497c-8229-f5a0f6ae194d-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:14.200285 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.200227 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d7355a2-0978-497c-8229-f5a0f6ae194d-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:14.200285 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.200242 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7355a2-0978-497c-8229-f5a0f6ae194d-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:14.200285 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.200254 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d7355a2-0978-497c-8229-f5a0f6ae194d-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:14.200285 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.200269 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d7355a2-0978-497c-8229-f5a0f6ae194d-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:14.200285 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.200281 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/253b93c5-40cc-444a-8836-06044e3d6dd4-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:14.200667 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.200295 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/253b93c5-40cc-444a-8836-06044e3d6dd4-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:14.202172 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.202100 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/253b93c5-40cc-444a-8836-06044e3d6dd4-istio-envoy" pod "253b93c5-40cc-444a-8836-06044e3d6dd4" (UID: "253b93c5-40cc-444a-8836-06044e3d6dd4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:14.202368 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.202343 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/253b93c5-40cc-444a-8836-06044e3d6dd4-istiod-ca-cert" pod "253b93c5-40cc-444a-8836-06044e3d6dd4" (UID: "253b93c5-40cc-444a-8836-06044e3d6dd4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:14.202511 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.202487 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d7355a2-0978-497c-8229-f5a0f6ae194d-kube-api-access-xwfqq" pod "8d7355a2-0978-497c-8229-f5a0f6ae194d" (UID: "8d7355a2-0978-497c-8229-f5a0f6ae194d"). InnerVolumeSpecName "kube-api-access-xwfqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:14.202852 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.202828 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d7355a2-0978-497c-8229-f5a0f6ae194d-istio-envoy" pod "8d7355a2-0978-497c-8229-f5a0f6ae194d" (UID: "8d7355a2-0978-497c-8229-f5a0f6ae194d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:14.203367 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.203346 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/253b93c5-40cc-444a-8836-06044e3d6dd4-istio-podinfo" pod "253b93c5-40cc-444a-8836-06044e3d6dd4" (UID: "253b93c5-40cc-444a-8836-06044e3d6dd4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:37:14.203681 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.203651 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d7355a2-0978-497c-8229-f5a0f6ae194d-istio-podinfo" pod "8d7355a2-0978-497c-8229-f5a0f6ae194d" (UID: "8d7355a2-0978-497c-8229-f5a0f6ae194d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:37:14.203681 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.203669 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" Oct 30 18:37:14.204001 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.203977 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/253b93c5-40cc-444a-8836-06044e3d6dd4-kube-api-access-qt96c" pod "253b93c5-40cc-444a-8836-06044e3d6dd4" (UID: "253b93c5-40cc-444a-8836-06044e3d6dd4"). InnerVolumeSpecName "kube-api-access-qt96c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:14.204373 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.204351 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d7355a2-0978-497c-8229-f5a0f6ae194d-istio-token" pod "8d7355a2-0978-497c-8229-f5a0f6ae194d" (UID: "8d7355a2-0978-497c-8229-f5a0f6ae194d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:14.204473 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.204459 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/253b93c5-40cc-444a-8836-06044e3d6dd4-istio-token" pod "253b93c5-40cc-444a-8836-06044e3d6dd4" (UID: "253b93c5-40cc-444a-8836-06044e3d6dd4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:14.206033 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.206020 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" Oct 30 18:37:14.301147 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.301049 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2065cdb-c364-4e80-addd-1055ed222931-istio-token\" (UniqueName: \"kubernetes.io/projected/e2065cdb-c364-4e80-addd-1055ed222931-istio-token\") pod \"e2065cdb-c364-4e80-addd-1055ed222931\" (UID: \"e2065cdb-c364-4e80-addd-1055ed222931\") " Oct 30 18:37:14.301147 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.301102 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2065cdb-c364-4e80-addd-1055ed222931-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2065cdb-c364-4e80-addd-1055ed222931-credential-socket\") pod \"e2065cdb-c364-4e80-addd-1055ed222931\" (UID: \"e2065cdb-c364-4e80-addd-1055ed222931\") " Oct 30 18:37:14.301747 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.301189 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2065cdb-c364-4e80-addd-1055ed222931-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2065cdb-c364-4e80-addd-1055ed222931-istio-podinfo\") pod \"e2065cdb-c364-4e80-addd-1055ed222931\" (UID: \"e2065cdb-c364-4e80-addd-1055ed222931\") " Oct 30 18:37:14.301747 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.301267 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2065cdb-c364-4e80-addd-1055ed222931-kube-api-access-cd9c7\" (UniqueName: \"kubernetes.io/projected/e2065cdb-c364-4e80-addd-1055ed222931-kube-api-access-cd9c7\") pod \"e2065cdb-c364-4e80-addd-1055ed222931\" (UID: \"e2065cdb-c364-4e80-addd-1055ed222931\") " Oct 30 18:37:14.301747 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.301308 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2065cdb-c364-4e80-addd-1055ed222931-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2065cdb-c364-4e80-addd-1055ed222931-istiod-ca-cert\") pod \"e2065cdb-c364-4e80-addd-1055ed222931\" (UID: \"e2065cdb-c364-4e80-addd-1055ed222931\") " Oct 30 18:37:14.301747 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.301359 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2065cdb-c364-4e80-addd-1055ed222931-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2065cdb-c364-4e80-addd-1055ed222931-workload-socket\") pod \"e2065cdb-c364-4e80-addd-1055ed222931\" (UID: \"e2065cdb-c364-4e80-addd-1055ed222931\") " Oct 30 18:37:14.301747 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.301460 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2399863-51b1-4b7a-acdd-ea28c9074e95-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2399863-51b1-4b7a-acdd-ea28c9074e95-istio-envoy\") pod \"a2399863-51b1-4b7a-acdd-ea28c9074e95\" (UID: \"a2399863-51b1-4b7a-acdd-ea28c9074e95\") " Oct 30 18:37:14.301747 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.301492 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2399863-51b1-4b7a-acdd-ea28c9074e95-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2399863-51b1-4b7a-acdd-ea28c9074e95-credential-socket\") pod \"a2399863-51b1-4b7a-acdd-ea28c9074e95\" (UID: \"a2399863-51b1-4b7a-acdd-ea28c9074e95\") " Oct 30 18:37:14.301747 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.301521 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2399863-51b1-4b7a-acdd-ea28c9074e95-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2399863-51b1-4b7a-acdd-ea28c9074e95-workload-certs\") pod \"a2399863-51b1-4b7a-acdd-ea28c9074e95\" (UID: \"a2399863-51b1-4b7a-acdd-ea28c9074e95\") " Oct 30 18:37:14.301747 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.301549 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2399863-51b1-4b7a-acdd-ea28c9074e95-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2399863-51b1-4b7a-acdd-ea28c9074e95-istio-podinfo\") pod \"a2399863-51b1-4b7a-acdd-ea28c9074e95\" (UID: \"a2399863-51b1-4b7a-acdd-ea28c9074e95\") " Oct 30 18:37:14.301747 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.301592 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2399863-51b1-4b7a-acdd-ea28c9074e95-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2399863-51b1-4b7a-acdd-ea28c9074e95-istiod-ca-cert\") pod \"a2399863-51b1-4b7a-acdd-ea28c9074e95\" (UID: \"a2399863-51b1-4b7a-acdd-ea28c9074e95\") " Oct 30 18:37:14.301747 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.301634 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2065cdb-c364-4e80-addd-1055ed222931-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2065cdb-c364-4e80-addd-1055ed222931-istio-envoy\") pod \"e2065cdb-c364-4e80-addd-1055ed222931\" (UID: \"e2065cdb-c364-4e80-addd-1055ed222931\") " Oct 30 18:37:14.301747 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.301654 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2065cdb-c364-4e80-addd-1055ed222931-credential-socket" pod "e2065cdb-c364-4e80-addd-1055ed222931" (UID: "e2065cdb-c364-4e80-addd-1055ed222931"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:14.301747 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.301667 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2399863-51b1-4b7a-acdd-ea28c9074e95-istio-token\" (UniqueName: \"kubernetes.io/projected/a2399863-51b1-4b7a-acdd-ea28c9074e95-istio-token\") pod \"a2399863-51b1-4b7a-acdd-ea28c9074e95\" (UID: \"a2399863-51b1-4b7a-acdd-ea28c9074e95\") " Oct 30 18:37:14.302554 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.302126 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2399863-51b1-4b7a-acdd-ea28c9074e95-kube-api-access-d7588\" (UniqueName: \"kubernetes.io/projected/a2399863-51b1-4b7a-acdd-ea28c9074e95-kube-api-access-d7588\") pod \"a2399863-51b1-4b7a-acdd-ea28c9074e95\" (UID: \"a2399863-51b1-4b7a-acdd-ea28c9074e95\") " Oct 30 18:37:14.302554 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.302183 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2399863-51b1-4b7a-acdd-ea28c9074e95-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2399863-51b1-4b7a-acdd-ea28c9074e95-workload-socket\") pod \"a2399863-51b1-4b7a-acdd-ea28c9074e95\" (UID: \"a2399863-51b1-4b7a-acdd-ea28c9074e95\") " Oct 30 18:37:14.302554 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.302492 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2399863-51b1-4b7a-acdd-ea28c9074e95-istiod-ca-cert" pod "a2399863-51b1-4b7a-acdd-ea28c9074e95" (UID: "a2399863-51b1-4b7a-acdd-ea28c9074e95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:14.306227 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.305764 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2399863-51b1-4b7a-acdd-ea28c9074e95-credential-socket" pod "a2399863-51b1-4b7a-acdd-ea28c9074e95" (UID: "a2399863-51b1-4b7a-acdd-ea28c9074e95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:14.306227 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.306119 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2399863-51b1-4b7a-acdd-ea28c9074e95-workload-socket" pod "a2399863-51b1-4b7a-acdd-ea28c9074e95" (UID: "a2399863-51b1-4b7a-acdd-ea28c9074e95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:14.306227 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.306001 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2399863-51b1-4b7a-acdd-ea28c9074e95-workload-certs" pod "a2399863-51b1-4b7a-acdd-ea28c9074e95" (UID: "a2399863-51b1-4b7a-acdd-ea28c9074e95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:14.306492 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.306376 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2065cdb-c364-4e80-addd-1055ed222931-workload-socket" pod "e2065cdb-c364-4e80-addd-1055ed222931" (UID: "e2065cdb-c364-4e80-addd-1055ed222931"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:14.306643 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.306619 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2065cdb-c364-4e80-addd-1055ed222931-workload-certs" pod "e2065cdb-c364-4e80-addd-1055ed222931" (UID: "e2065cdb-c364-4e80-addd-1055ed222931"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:14.307545 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.307485 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2065cdb-c364-4e80-addd-1055ed222931-istiod-ca-cert" pod "e2065cdb-c364-4e80-addd-1055ed222931" (UID: "e2065cdb-c364-4e80-addd-1055ed222931"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:14.309578 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.309555 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2065cdb-c364-4e80-addd-1055ed222931-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2065cdb-c364-4e80-addd-1055ed222931-workload-certs\") pod \"e2065cdb-c364-4e80-addd-1055ed222931\" (UID: \"e2065cdb-c364-4e80-addd-1055ed222931\") " Oct 30 18:37:14.309666 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.309620 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2399863-51b1-4b7a-acdd-ea28c9074e95-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2399863-51b1-4b7a-acdd-ea28c9074e95-istio-data\") pod \"a2399863-51b1-4b7a-acdd-ea28c9074e95\" (UID: \"a2399863-51b1-4b7a-acdd-ea28c9074e95\") " Oct 30 18:37:14.309708 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.309663 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2065cdb-c364-4e80-addd-1055ed222931-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2065cdb-c364-4e80-addd-1055ed222931-istio-data\") pod \"e2065cdb-c364-4e80-addd-1055ed222931\" (UID: \"e2065cdb-c364-4e80-addd-1055ed222931\") " Oct 30 18:37:14.310356 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.310335 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2399863-51b1-4b7a-acdd-ea28c9074e95-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:14.310416 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.310364 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d7355a2-0978-497c-8229-f5a0f6ae194d-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:14.310416 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.310379 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt96c\" (UniqueName: \"kubernetes.io/projected/253b93c5-40cc-444a-8836-06044e3d6dd4-kube-api-access-qt96c\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:14.310416 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.310394 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwfqq\" (UniqueName: \"kubernetes.io/projected/8d7355a2-0978-497c-8229-f5a0f6ae194d-kube-api-access-xwfqq\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:14.310416 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.310409 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/253b93c5-40cc-444a-8836-06044e3d6dd4-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:14.310544 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.310422 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2065cdb-c364-4e80-addd-1055ed222931-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:14.310544 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.310434 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/253b93c5-40cc-444a-8836-06044e3d6dd4-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:14.310544 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.310447 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2065cdb-c364-4e80-addd-1055ed222931-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:14.310544 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.310461 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2065cdb-c364-4e80-addd-1055ed222931-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:14.310544 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.310466 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2065cdb-c364-4e80-addd-1055ed222931-istio-podinfo" pod "e2065cdb-c364-4e80-addd-1055ed222931" (UID: "e2065cdb-c364-4e80-addd-1055ed222931"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:37:14.310544 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.310473 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/253b93c5-40cc-444a-8836-06044e3d6dd4-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:14.310544 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.310511 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/253b93c5-40cc-444a-8836-06044e3d6dd4-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:14.310544 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.310527 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2065cdb-c364-4e80-addd-1055ed222931-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:14.310544 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.310540 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d7355a2-0978-497c-8229-f5a0f6ae194d-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:14.310855 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.310554 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d7355a2-0978-497c-8229-f5a0f6ae194d-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:14.310855 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.310568 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2399863-51b1-4b7a-acdd-ea28c9074e95-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:14.310855 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.310581 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2399863-51b1-4b7a-acdd-ea28c9074e95-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:14.310855 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.310627 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2399863-51b1-4b7a-acdd-ea28c9074e95-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:14.310855 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.310637 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2065cdb-c364-4e80-addd-1055ed222931-istio-data" pod "e2065cdb-c364-4e80-addd-1055ed222931" (UID: "e2065cdb-c364-4e80-addd-1055ed222931"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:14.310855 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.310686 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2399863-51b1-4b7a-acdd-ea28c9074e95-istio-data" pod "a2399863-51b1-4b7a-acdd-ea28c9074e95" (UID: "a2399863-51b1-4b7a-acdd-ea28c9074e95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:14.313006 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.312971 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2065cdb-c364-4e80-addd-1055ed222931-istio-token" pod "e2065cdb-c364-4e80-addd-1055ed222931" (UID: "e2065cdb-c364-4e80-addd-1055ed222931"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:14.313462 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.313358 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2399863-51b1-4b7a-acdd-ea28c9074e95-kube-api-access-d7588" pod "a2399863-51b1-4b7a-acdd-ea28c9074e95" (UID: "a2399863-51b1-4b7a-acdd-ea28c9074e95"). InnerVolumeSpecName "kube-api-access-d7588". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:14.313462 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.313436 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2399863-51b1-4b7a-acdd-ea28c9074e95-istio-token" pod "a2399863-51b1-4b7a-acdd-ea28c9074e95" (UID: "a2399863-51b1-4b7a-acdd-ea28c9074e95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:14.313462 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.313487 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2065cdb-c364-4e80-addd-1055ed222931-kube-api-access-cd9c7" pod "e2065cdb-c364-4e80-addd-1055ed222931" (UID: "e2065cdb-c364-4e80-addd-1055ed222931"). InnerVolumeSpecName "kube-api-access-cd9c7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:14.313725 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.313536 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2399863-51b1-4b7a-acdd-ea28c9074e95-istio-podinfo" pod "a2399863-51b1-4b7a-acdd-ea28c9074e95" (UID: "a2399863-51b1-4b7a-acdd-ea28c9074e95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:37:14.313725 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.313636 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2399863-51b1-4b7a-acdd-ea28c9074e95-istio-envoy" pod "a2399863-51b1-4b7a-acdd-ea28c9074e95" (UID: "a2399863-51b1-4b7a-acdd-ea28c9074e95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:14.314840 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.314816 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2065cdb-c364-4e80-addd-1055ed222931-istio-envoy" pod "e2065cdb-c364-4e80-addd-1055ed222931" (UID: "e2065cdb-c364-4e80-addd-1055ed222931"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:14.369659 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.369624 4442 generic.go:356] "Generic (PLEG): container finished" podID="a2399863-51b1-4b7a-acdd-ea28c9074e95" containerID="c19fecf022f3b46b5ba9a5ce7d8cc916830517e625d4a976ed22624d6c8ab02d" exitCode=0 Oct 30 18:37:14.369842 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.369712 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" event={"ID":"a2399863-51b1-4b7a-acdd-ea28c9074e95","Type":"ContainerDied","Data":"c19fecf022f3b46b5ba9a5ce7d8cc916830517e625d4a976ed22624d6c8ab02d"} Oct 30 18:37:14.369842 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.369784 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" event={"ID":"a2399863-51b1-4b7a-acdd-ea28c9074e95","Type":"ContainerDied","Data":"b30034b140beafada7f484d78db9c74ccf7ffa169ab577d20e94695913c5725d"} Oct 30 18:37:14.369842 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.369813 4442 scope.go:122] "RemoveContainer" containerID="c19fecf022f3b46b5ba9a5ce7d8cc916830517e625d4a976ed22624d6c8ab02d" Oct 30 18:37:14.370113 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.370000 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp" Oct 30 18:37:14.375844 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.375814 4442 generic.go:356] "Generic (PLEG): container finished" podID="253b93c5-40cc-444a-8836-06044e3d6dd4" containerID="59139cd792b7b832f56a135c3705848e70368a7884613449f94a12139f104d07" exitCode=0 Oct 30 18:37:14.376037 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.376015 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" event={"ID":"253b93c5-40cc-444a-8836-06044e3d6dd4","Type":"ContainerDied","Data":"59139cd792b7b832f56a135c3705848e70368a7884613449f94a12139f104d07"} Oct 30 18:37:14.376127 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.376074 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" event={"ID":"253b93c5-40cc-444a-8836-06044e3d6dd4","Type":"ContainerDied","Data":"548d78ae3300a3938433db70d475ea9adb481e36d1c602d7a27e7e80ba7d8305"} Oct 30 18:37:14.376216 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.376185 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4" Oct 30 18:37:14.384274 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.384257 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" Oct 30 18:37:14.384555 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.384281 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" event={"ID":"e2065cdb-c364-4e80-addd-1055ed222931","Type":"ContainerDied","Data":"ee8ebd7c9d53ab796aaf6e03c26bb9dd23913ba8495b8b9d0f766971d875b8ab"} Oct 30 18:37:14.384555 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.384344 4442 generic.go:356] "Generic (PLEG): container finished" podID="e2065cdb-c364-4e80-addd-1055ed222931" containerID="ee8ebd7c9d53ab796aaf6e03c26bb9dd23913ba8495b8b9d0f766971d875b8ab" exitCode=0 Oct 30 18:37:14.386200 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.384723 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn" event={"ID":"e2065cdb-c364-4e80-addd-1055ed222931","Type":"ContainerDied","Data":"102f688314cf4c3d64f0f676c08f49bd9a98e69074ce5e4e1cb3d6fd77366c3e"} Oct 30 18:37:14.392388 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.392363 4442 generic.go:356] "Generic (PLEG): container finished" podID="8d7355a2-0978-497c-8229-f5a0f6ae194d" containerID="a8c9f2a9f1eb89996d239e8291466951c267e05bacb6ffa947340ae80ec9d58d" exitCode=0 Oct 30 18:37:14.392504 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.392424 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" event={"ID":"8d7355a2-0978-497c-8229-f5a0f6ae194d","Type":"ContainerDied","Data":"a8c9f2a9f1eb89996d239e8291466951c267e05bacb6ffa947340ae80ec9d58d"} Oct 30 18:37:14.392504 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.392464 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" event={"ID":"8d7355a2-0978-497c-8229-f5a0f6ae194d","Type":"ContainerDied","Data":"3ab1f863cfb0e225fe85c36cb2ecdd12795718c8b791a37070903d254ac67513"} Oct 30 18:37:14.392634 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.392543 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv" Oct 30 18:37:14.393954 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.393933 4442 scope.go:122] "RemoveContainer" containerID="c19fecf022f3b46b5ba9a5ce7d8cc916830517e625d4a976ed22624d6c8ab02d" Oct 30 18:37:14.394345 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:37:14.394315 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c19fecf022f3b46b5ba9a5ce7d8cc916830517e625d4a976ed22624d6c8ab02d\": container with ID starting with c19fecf022f3b46b5ba9a5ce7d8cc916830517e625d4a976ed22624d6c8ab02d not found: ID does not exist" containerID="c19fecf022f3b46b5ba9a5ce7d8cc916830517e625d4a976ed22624d6c8ab02d" Oct 30 18:37:14.394450 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.394357 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c19fecf022f3b46b5ba9a5ce7d8cc916830517e625d4a976ed22624d6c8ab02d"} err="failed to get container status \"c19fecf022f3b46b5ba9a5ce7d8cc916830517e625d4a976ed22624d6c8ab02d\": rpc error: code = NotFound desc = could not find container \"c19fecf022f3b46b5ba9a5ce7d8cc916830517e625d4a976ed22624d6c8ab02d\": container with ID starting with c19fecf022f3b46b5ba9a5ce7d8cc916830517e625d4a976ed22624d6c8ab02d not found: ID does not exist" Oct 30 18:37:14.394450 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.394379 4442 scope.go:122] "RemoveContainer" containerID="59139cd792b7b832f56a135c3705848e70368a7884613449f94a12139f104d07" Oct 30 18:37:14.408119 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.408102 4442 scope.go:122] "RemoveContainer" containerID="59139cd792b7b832f56a135c3705848e70368a7884613449f94a12139f104d07" Oct 30 18:37:14.408447 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:37:14.408429 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59139cd792b7b832f56a135c3705848e70368a7884613449f94a12139f104d07\": container with ID starting with 59139cd792b7b832f56a135c3705848e70368a7884613449f94a12139f104d07 not found: ID does not exist" containerID="59139cd792b7b832f56a135c3705848e70368a7884613449f94a12139f104d07" Oct 30 18:37:14.408515 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.408459 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59139cd792b7b832f56a135c3705848e70368a7884613449f94a12139f104d07"} err="failed to get container status \"59139cd792b7b832f56a135c3705848e70368a7884613449f94a12139f104d07\": rpc error: code = NotFound desc = could not find container \"59139cd792b7b832f56a135c3705848e70368a7884613449f94a12139f104d07\": container with ID starting with 59139cd792b7b832f56a135c3705848e70368a7884613449f94a12139f104d07 not found: ID does not exist" Oct 30 18:37:14.408515 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.408476 4442 scope.go:122] "RemoveContainer" containerID="ee8ebd7c9d53ab796aaf6e03c26bb9dd23913ba8495b8b9d0f766971d875b8ab" Oct 30 18:37:14.411844 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.411819 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7588\" (UniqueName: \"kubernetes.io/projected/a2399863-51b1-4b7a-acdd-ea28c9074e95-kube-api-access-d7588\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:14.411940 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.411850 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2399863-51b1-4b7a-acdd-ea28c9074e95-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:14.411940 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.411866 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2065cdb-c364-4e80-addd-1055ed222931-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:14.411940 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.411879 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2065cdb-c364-4e80-addd-1055ed222931-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:14.411940 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.411894 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2065cdb-c364-4e80-addd-1055ed222931-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:14.411940 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.411906 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd9c7\" (UniqueName: \"kubernetes.io/projected/e2065cdb-c364-4e80-addd-1055ed222931-kube-api-access-cd9c7\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:14.411940 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.411919 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2399863-51b1-4b7a-acdd-ea28c9074e95-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:14.411940 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.411932 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2399863-51b1-4b7a-acdd-ea28c9074e95-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:14.412284 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.411944 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2065cdb-c364-4e80-addd-1055ed222931-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:14.412284 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.411958 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2399863-51b1-4b7a-acdd-ea28c9074e95-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:14.419844 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.419826 4442 scope.go:122] "RemoveContainer" containerID="ee8ebd7c9d53ab796aaf6e03c26bb9dd23913ba8495b8b9d0f766971d875b8ab" Oct 30 18:37:14.420233 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:37:14.420193 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee8ebd7c9d53ab796aaf6e03c26bb9dd23913ba8495b8b9d0f766971d875b8ab\": container with ID starting with ee8ebd7c9d53ab796aaf6e03c26bb9dd23913ba8495b8b9d0f766971d875b8ab not found: ID does not exist" containerID="ee8ebd7c9d53ab796aaf6e03c26bb9dd23913ba8495b8b9d0f766971d875b8ab" Oct 30 18:37:14.420344 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.420227 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee8ebd7c9d53ab796aaf6e03c26bb9dd23913ba8495b8b9d0f766971d875b8ab"} err="failed to get container status \"ee8ebd7c9d53ab796aaf6e03c26bb9dd23913ba8495b8b9d0f766971d875b8ab\": rpc error: code = NotFound desc = could not find container \"ee8ebd7c9d53ab796aaf6e03c26bb9dd23913ba8495b8b9d0f766971d875b8ab\": container with ID starting with ee8ebd7c9d53ab796aaf6e03c26bb9dd23913ba8495b8b9d0f766971d875b8ab not found: ID does not exist" Oct 30 18:37:14.420344 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.420250 4442 scope.go:122] "RemoveContainer" containerID="a8c9f2a9f1eb89996d239e8291466951c267e05bacb6ffa947340ae80ec9d58d" Oct 30 18:37:14.430275 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.430250 4442 scope.go:122] "RemoveContainer" containerID="a8c9f2a9f1eb89996d239e8291466951c267e05bacb6ffa947340ae80ec9d58d" Oct 30 18:37:14.430613 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:37:14.430588 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8c9f2a9f1eb89996d239e8291466951c267e05bacb6ffa947340ae80ec9d58d\": container with ID starting with a8c9f2a9f1eb89996d239e8291466951c267e05bacb6ffa947340ae80ec9d58d not found: ID does not exist" containerID="a8c9f2a9f1eb89996d239e8291466951c267e05bacb6ffa947340ae80ec9d58d" Oct 30 18:37:14.430689 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.430625 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8c9f2a9f1eb89996d239e8291466951c267e05bacb6ffa947340ae80ec9d58d"} err="failed to get container status \"a8c9f2a9f1eb89996d239e8291466951c267e05bacb6ffa947340ae80ec9d58d\": rpc error: code = NotFound desc = could not find container \"a8c9f2a9f1eb89996d239e8291466951c267e05bacb6ffa947340ae80ec9d58d\": container with ID starting with a8c9f2a9f1eb89996d239e8291466951c267e05bacb6ffa947340ae80ec9d58d not found: ID does not exist" Oct 30 18:37:14.456450 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.456399 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4"] Oct 30 18:37:14.463543 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.463506 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-kwkl4"] Oct 30 18:37:14.495760 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.495719 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv"] Oct 30 18:37:14.499650 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.499625 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-gzlfv"] Oct 30 18:37:14.553994 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.553905 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn"] Oct 30 18:37:14.560120 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.560088 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-xlrwn"] Oct 30 18:37:14.590602 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.590569 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp"] Oct 30 18:37:14.594815 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:14.594784 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-mqdpp"] Oct 30 18:37:15.065955 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:15.065920 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="253b93c5-40cc-444a-8836-06044e3d6dd4" path="/var/lib/kubelet/pods/253b93c5-40cc-444a-8836-06044e3d6dd4/volumes" Oct 30 18:37:15.066579 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:15.066555 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d7355a2-0978-497c-8229-f5a0f6ae194d" path="/var/lib/kubelet/pods/8d7355a2-0978-497c-8229-f5a0f6ae194d/volumes" Oct 30 18:37:15.067103 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:15.067082 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2399863-51b1-4b7a-acdd-ea28c9074e95" path="/var/lib/kubelet/pods/a2399863-51b1-4b7a-acdd-ea28c9074e95/volumes" Oct 30 18:37:15.067967 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:15.067944 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2065cdb-c364-4e80-addd-1055ed222931" path="/var/lib/kubelet/pods/e2065cdb-c364-4e80-addd-1055ed222931/volumes" Oct 30 18:37:17.562668 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:37:17.562628 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:37:22.586259 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:22.586184 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-1a2436/auth-test-model-1a2436-kserve-547bf9b957-hhcgp" podUID="b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2" containerName="main" containerID="cri-o://c30529165a0d21b495c05cc7652b928c884c1e9a19ddbb244b0686ad8d710b34" gracePeriod=45 Oct 30 18:37:22.863925 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:22.863899 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-1a2436/auth-test-model-1a2436-kserve-547bf9b957-hhcgp" Oct 30 18:37:22.943862 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:22.943826 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2-home\" (UniqueName: \"kubernetes.io/empty-dir/b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2-home\") pod \"b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2\" (UID: \"b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2\") " Oct 30 18:37:22.943862 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:22.943866 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2-tls-certs\" (UniqueName: \"kubernetes.io/secret/b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2-tls-certs\") pod \"b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2\" (UID: \"b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2\") " Oct 30 18:37:22.944119 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:22.944000 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2-tmp-dir\") pod \"b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2\" (UID: \"b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2\") " Oct 30 18:37:22.944119 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:22.944106 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2-kube-api-access-wvsmx\" (UniqueName: \"kubernetes.io/projected/b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2-kube-api-access-wvsmx\") pod \"b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2\" (UID: \"b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2\") " Oct 30 18:37:22.944278 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:22.944183 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2-home" pod "b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2" (UID: "b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:22.944278 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:22.944255 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2-tmp-dir" pod "b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2" (UID: "b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:22.944391 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:22.944335 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2-model-cache\") pod \"b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2\" (UID: \"b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2\") " Oct 30 18:37:22.944391 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:22.944367 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2-dshm\") pod \"b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2\" (UID: \"b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2\") " Oct 30 18:37:22.944620 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:22.944595 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2-model-cache" pod "b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2" (UID: "b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:22.945000 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:22.944935 4442 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2-model-cache\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:22.945000 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:22.944964 4442 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2-home\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:22.945000 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:22.944977 4442 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2-tmp-dir\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:22.946868 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:22.946844 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2-tls-certs" pod "b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2" (UID: "b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:22.947288 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:22.947268 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2-dshm" pod "b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2" (UID: "b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:22.953171 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:22.953146 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2-kube-api-access-wvsmx" pod "b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2" (UID: "b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2"). InnerVolumeSpecName "kube-api-access-wvsmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:23.046446 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.046414 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvsmx\" (UniqueName: \"kubernetes.io/projected/b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2-kube-api-access-wvsmx\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:23.046446 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.046443 4442 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2-dshm\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:23.046446 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.046453 4442 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2-tls-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:23.306270 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.306236 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-236fc4-c8b87c5ff-9knfc"] Oct 30 18:37:23.306880 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.306865 4442 cpu_manager.go:414] "containerMap: removing container" podUID="e2065cdb-c364-4e80-addd-1055ed222931" containerName="istio-proxy" Oct 30 18:37:23.306933 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.306881 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2065cdb-c364-4e80-addd-1055ed222931" containerName="istio-proxy" Oct 30 18:37:23.306933 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.306891 4442 cpu_manager.go:414] "containerMap: removing container" podUID="253b93c5-40cc-444a-8836-06044e3d6dd4" containerName="istio-proxy" Oct 30 18:37:23.306933 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.306896 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="253b93c5-40cc-444a-8836-06044e3d6dd4" containerName="istio-proxy" Oct 30 18:37:23.306933 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.306916 4442 cpu_manager.go:414] "containerMap: removing container" podUID="8d7355a2-0978-497c-8229-f5a0f6ae194d" containerName="istio-proxy" Oct 30 18:37:23.306933 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.306922 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d7355a2-0978-497c-8229-f5a0f6ae194d" containerName="istio-proxy" Oct 30 18:37:23.306933 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.306929 4442 cpu_manager.go:414] "containerMap: removing container" podUID="a2399863-51b1-4b7a-acdd-ea28c9074e95" containerName="istio-proxy" Oct 30 18:37:23.306933 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.306933 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2399863-51b1-4b7a-acdd-ea28c9074e95" containerName="istio-proxy" Oct 30 18:37:23.307179 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.306957 4442 cpu_manager.go:414] "containerMap: removing container" podUID="b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2" containerName="main" Oct 30 18:37:23.307179 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.306961 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2" containerName="main" Oct 30 18:37:23.307179 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.307049 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d7355a2-0978-497c-8229-f5a0f6ae194d" containerName="istio-proxy" Oct 30 18:37:23.307179 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.307058 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2399863-51b1-4b7a-acdd-ea28c9074e95" containerName="istio-proxy" Oct 30 18:37:23.307179 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.307068 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2" containerName="main" Oct 30 18:37:23.307179 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.307076 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2065cdb-c364-4e80-addd-1055ed222931" containerName="istio-proxy" Oct 30 18:37:23.307179 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.307081 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="253b93c5-40cc-444a-8836-06044e3d6dd4" containerName="istio-proxy" Oct 30 18:37:23.310715 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.310690 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-236fc4-c8b87c5ff-9knfc" Oct 30 18:37:23.333343 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.333306 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-236fc4-c8b87c5ff-9knfc"] Oct 30 18:37:23.338421 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.338393 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-236fc4-serving-cert\"" Oct 30 18:37:23.451559 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.451522 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dfjq\" (UniqueName: \"kubernetes.io/projected/320da9e0-4338-4b7e-b950-7e3cc76b2faa-kube-api-access-6dfjq\") pod \"maas-api-e2e-shared-a-w0-236fc4-c8b87c5ff-9knfc\" (UID: \"320da9e0-4338-4b7e-b950-7e3cc76b2faa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-236fc4-c8b87c5ff-9knfc" Oct 30 18:37:23.451752 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.451702 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/320da9e0-4338-4b7e-b950-7e3cc76b2faa-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-236fc4-c8b87c5ff-9knfc\" (UID: \"320da9e0-4338-4b7e-b950-7e3cc76b2faa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-236fc4-c8b87c5ff-9knfc" Oct 30 18:37:23.451823 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.451758 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/320da9e0-4338-4b7e-b950-7e3cc76b2faa-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-236fc4-c8b87c5ff-9knfc\" (UID: \"320da9e0-4338-4b7e-b950-7e3cc76b2faa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-236fc4-c8b87c5ff-9knfc" Oct 30 18:37:23.451823 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.451779 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/320da9e0-4338-4b7e-b950-7e3cc76b2faa-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-236fc4-c8b87c5ff-9knfc\" (UID: \"320da9e0-4338-4b7e-b950-7e3cc76b2faa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-236fc4-c8b87c5ff-9knfc" Oct 30 18:37:23.471537 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.471505 4442 generic.go:356] "Generic (PLEG): container finished" podID="b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2" containerID="c30529165a0d21b495c05cc7652b928c884c1e9a19ddbb244b0686ad8d710b34" exitCode=0 Oct 30 18:37:23.471695 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.471547 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-1a2436/auth-test-model-1a2436-kserve-547bf9b957-hhcgp" Oct 30 18:37:23.471695 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.471572 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-1a2436/auth-test-model-1a2436-kserve-547bf9b957-hhcgp" event={"ID":"b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2","Type":"ContainerDied","Data":"c30529165a0d21b495c05cc7652b928c884c1e9a19ddbb244b0686ad8d710b34"} Oct 30 18:37:23.471695 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.471608 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-1a2436/auth-test-model-1a2436-kserve-547bf9b957-hhcgp" event={"ID":"b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2","Type":"ContainerDied","Data":"23466f610518422da53b5ec066462e46823d211721a8848ad577be3ca376686d"} Oct 30 18:37:23.471695 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.471625 4442 scope.go:122] "RemoveContainer" containerID="c30529165a0d21b495c05cc7652b928c884c1e9a19ddbb244b0686ad8d710b34" Oct 30 18:37:23.482518 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.482500 4442 scope.go:122] "RemoveContainer" containerID="c30529165a0d21b495c05cc7652b928c884c1e9a19ddbb244b0686ad8d710b34" Oct 30 18:37:23.482835 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:37:23.482819 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c30529165a0d21b495c05cc7652b928c884c1e9a19ddbb244b0686ad8d710b34\": container with ID starting with c30529165a0d21b495c05cc7652b928c884c1e9a19ddbb244b0686ad8d710b34 not found: ID does not exist" containerID="c30529165a0d21b495c05cc7652b928c884c1e9a19ddbb244b0686ad8d710b34" Oct 30 18:37:23.482913 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.482843 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c30529165a0d21b495c05cc7652b928c884c1e9a19ddbb244b0686ad8d710b34"} err="failed to get container status \"c30529165a0d21b495c05cc7652b928c884c1e9a19ddbb244b0686ad8d710b34\": rpc error: code = NotFound desc = could not find container \"c30529165a0d21b495c05cc7652b928c884c1e9a19ddbb244b0686ad8d710b34\": container with ID starting with c30529165a0d21b495c05cc7652b928c884c1e9a19ddbb244b0686ad8d710b34 not found: ID does not exist" Oct 30 18:37:23.573027 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.566698 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/320da9e0-4338-4b7e-b950-7e3cc76b2faa-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-236fc4-c8b87c5ff-9knfc\" (UID: \"320da9e0-4338-4b7e-b950-7e3cc76b2faa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-236fc4-c8b87c5ff-9knfc" Oct 30 18:37:23.573027 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.566841 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/320da9e0-4338-4b7e-b950-7e3cc76b2faa-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-236fc4-c8b87c5ff-9knfc\" (UID: \"320da9e0-4338-4b7e-b950-7e3cc76b2faa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-236fc4-c8b87c5ff-9knfc" Oct 30 18:37:23.573027 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.566883 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/320da9e0-4338-4b7e-b950-7e3cc76b2faa-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-236fc4-c8b87c5ff-9knfc\" (UID: \"320da9e0-4338-4b7e-b950-7e3cc76b2faa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-236fc4-c8b87c5ff-9knfc" Oct 30 18:37:23.573027 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.567001 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dfjq\" (UniqueName: \"kubernetes.io/projected/320da9e0-4338-4b7e-b950-7e3cc76b2faa-kube-api-access-6dfjq\") pod \"maas-api-e2e-shared-a-w0-236fc4-c8b87c5ff-9knfc\" (UID: \"320da9e0-4338-4b7e-b950-7e3cc76b2faa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-236fc4-c8b87c5ff-9knfc" Oct 30 18:37:23.574627 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.574527 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/320da9e0-4338-4b7e-b950-7e3cc76b2faa-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-236fc4-c8b87c5ff-9knfc\" (UID: \"320da9e0-4338-4b7e-b950-7e3cc76b2faa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-236fc4-c8b87c5ff-9knfc" Oct 30 18:37:23.579000 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.576629 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/320da9e0-4338-4b7e-b950-7e3cc76b2faa-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-236fc4-c8b87c5ff-9knfc\" (UID: \"320da9e0-4338-4b7e-b950-7e3cc76b2faa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-236fc4-c8b87c5ff-9knfc" Oct 30 18:37:23.586160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.580273 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-1a2436/auth-test-model-1a2436-kserve-547bf9b957-hhcgp"] Oct 30 18:37:23.590182 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.587729 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/320da9e0-4338-4b7e-b950-7e3cc76b2faa-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-236fc4-c8b87c5ff-9knfc\" (UID: \"320da9e0-4338-4b7e-b950-7e3cc76b2faa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-236fc4-c8b87c5ff-9knfc" Oct 30 18:37:23.601806 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.601775 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-1a2436/auth-test-model-1a2436-kserve-547bf9b957-hhcgp"] Oct 30 18:37:23.602547 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.602526 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dfjq\" (UniqueName: \"kubernetes.io/projected/320da9e0-4338-4b7e-b950-7e3cc76b2faa-kube-api-access-6dfjq\") pod \"maas-api-e2e-shared-a-w0-236fc4-c8b87c5ff-9knfc\" (UID: \"320da9e0-4338-4b7e-b950-7e3cc76b2faa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-236fc4-c8b87c5ff-9knfc" Oct 30 18:37:23.626543 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.626511 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-236fc4-c8b87c5ff-9knfc" Oct 30 18:37:23.775443 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.775401 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn"] Oct 30 18:37:23.775648 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.775605 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" podUID="15bc30e5-adb7-45f8-abd5-d5fea229bda7" containerName="istio-proxy" containerID="cri-o://93e0da9fffce6f5476844d2bfe20733255f86ff8b7daafd1ff650e670961f716" gracePeriod=30 Oct 30 18:37:23.967126 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:37:23.967081 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod320da9e0_4338_4b7e_b950_7e3cc76b2faa.slice/crio-503ba738d9cc793c5704f6d3c3c8ad71a95e7a320178751138ff6b9759191a21 WatchSource:0}: Error finding container 503ba738d9cc793c5704f6d3c3c8ad71a95e7a320178751138ff6b9759191a21: Status 404 returned error can't find the container with id 503ba738d9cc793c5704f6d3c3c8ad71a95e7a320178751138ff6b9759191a21 Oct 30 18:37:23.970771 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:23.970449 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-236fc4-c8b87c5ff-9knfc"] Oct 30 18:37:24.485258 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:24.485222 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-236fc4-c8b87c5ff-9knfc" event={"ID":"320da9e0-4338-4b7e-b950-7e3cc76b2faa","Type":"ContainerStarted","Data":"66e6d4aaaef585f8861ad78a159705b7349c285543bf79410e0e852ecf041e47"} Oct 30 18:37:24.485439 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:24.485282 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-236fc4-c8b87c5ff-9knfc" event={"ID":"320da9e0-4338-4b7e-b950-7e3cc76b2faa","Type":"ContainerStarted","Data":"503ba738d9cc793c5704f6d3c3c8ad71a95e7a320178751138ff6b9759191a21"} Oct 30 18:37:24.485439 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:24.485374 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-236fc4-c8b87c5ff-9knfc" Oct 30 18:37:24.532981 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:24.532517 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-236fc4-c8b87c5ff-9knfc" podStartSLOduration=1.115535601 podStartE2EDuration="1.532495006s" totalImagesPullingTime="416.959405ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:24.525978253 +0000 UTC m=+2341.616040628" watchObservedRunningTime="2026-10-30 18:37:24.532495006 +0000 UTC m=+2341.622557346" Oct 30 18:37:25.070037 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:25.068770 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2" path="/var/lib/kubelet/pods/b9261f2f-05ba-44bf-a5a2-1b4c9a41b6f2/volumes" Oct 30 18:37:26.367383 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:26.367344 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-24b124/auth-test-model-24b124-kserve-745594bcc-6lt65"] Oct 30 18:37:26.624815 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:26.624722 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7628-b718f6"] Oct 30 18:37:26.631093 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:26.631069 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7628-b718f6" Oct 30 18:37:26.645946 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:26.645907 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7628-b718f6"] Oct 30 18:37:26.646256 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:26.646177 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-r2spd\"" Oct 30 18:37:26.725332 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:26.725294 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2kwt\" (UniqueName: \"kubernetes.io/projected/4f6ddd11-474f-43c6-8cea-9d6140bdcc65-kube-api-access-j2kwt\") pod \"test-curl-7628-b718f6\" (UID: \"4f6ddd11-474f-43c6-8cea-9d6140bdcc65\") " pod="opendatahub/test-curl-7628-b718f6" Oct 30 18:37:26.826723 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:26.826689 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2kwt\" (UniqueName: \"kubernetes.io/projected/4f6ddd11-474f-43c6-8cea-9d6140bdcc65-kube-api-access-j2kwt\") pod \"test-curl-7628-b718f6\" (UID: \"4f6ddd11-474f-43c6-8cea-9d6140bdcc65\") " pod="opendatahub/test-curl-7628-b718f6" Oct 30 18:37:26.847908 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:26.847870 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2kwt\" (UniqueName: \"kubernetes.io/projected/4f6ddd11-474f-43c6-8cea-9d6140bdcc65-kube-api-access-j2kwt\") pod \"test-curl-7628-b718f6\" (UID: \"4f6ddd11-474f-43c6-8cea-9d6140bdcc65\") " pod="opendatahub/test-curl-7628-b718f6" Oct 30 18:37:26.951364 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:26.951280 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7628-b718f6" Oct 30 18:37:27.263422 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:27.263388 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7628-b718f6"] Oct 30 18:37:27.263760 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:37:27.263741 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f6ddd11_474f_43c6_8cea_9d6140bdcc65.slice/crio-d274abf9f781e175e3d5a0d7d5e4b69bf2a4b12db8429d38de9ea2b2ad358e10 WatchSource:0}: Error finding container d274abf9f781e175e3d5a0d7d5e4b69bf2a4b12db8429d38de9ea2b2ad358e10: Status 404 returned error can't find the container with id d274abf9f781e175e3d5a0d7d5e4b69bf2a4b12db8429d38de9ea2b2ad358e10 Oct 30 18:37:27.556890 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:27.556778 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7628-b718f6" event={"ID":"4f6ddd11-474f-43c6-8cea-9d6140bdcc65","Type":"ContainerStarted","Data":"6b224ad0473994861ff292491e4c3a2bf0802543a1904b99d62af29fe3da37d4"} Oct 30 18:37:27.556890 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:27.556825 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7628-b718f6" event={"ID":"4f6ddd11-474f-43c6-8cea-9d6140bdcc65","Type":"ContainerStarted","Data":"d274abf9f781e175e3d5a0d7d5e4b69bf2a4b12db8429d38de9ea2b2ad358e10"} Oct 30 18:37:27.594441 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:27.594390 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7628-b718f6" podStartSLOduration=1.594376849 podStartE2EDuration="1.594376849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:27.592784687 +0000 UTC m=+2344.682847027" watchObservedRunningTime="2026-10-30 18:37:27.594376849 +0000 UTC m=+2344.684439188" Oct 30 18:37:28.481710 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:28.481662 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7628-b718f6"] Oct 30 18:37:28.484898 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:28.484867 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7628-b718f6"] Oct 30 18:37:28.632109 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:28.632064 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7628-b6f680"] Oct 30 18:37:28.633160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:28.633103 4442 cpu_manager.go:414] "containerMap: removing container" podUID="4f6ddd11-474f-43c6-8cea-9d6140bdcc65" containerName="test-curl-7628-b718f6" Oct 30 18:37:28.633160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:28.633148 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f6ddd11-474f-43c6-8cea-9d6140bdcc65" containerName="test-curl-7628-b718f6" Oct 30 18:37:28.633335 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:28.633320 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f6ddd11-474f-43c6-8cea-9d6140bdcc65" containerName="test-curl-7628-b718f6" Oct 30 18:37:28.637080 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:28.637064 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7628-b6f680" Oct 30 18:37:28.650250 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:28.650229 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7628-b6f680"] Oct 30 18:37:28.651966 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:28.651940 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nhpt\" (UniqueName: \"kubernetes.io/projected/18265c8d-6de9-4922-8b89-05b35823a21c-kube-api-access-6nhpt\") pod \"test-curl-7628-b6f680\" (UID: \"18265c8d-6de9-4922-8b89-05b35823a21c\") " pod="opendatahub/test-curl-7628-b6f680" Oct 30 18:37:28.753958 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:28.753855 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nhpt\" (UniqueName: \"kubernetes.io/projected/18265c8d-6de9-4922-8b89-05b35823a21c-kube-api-access-6nhpt\") pod \"test-curl-7628-b6f680\" (UID: \"18265c8d-6de9-4922-8b89-05b35823a21c\") " pod="opendatahub/test-curl-7628-b6f680" Oct 30 18:37:28.777022 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:28.776989 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nhpt\" (UniqueName: \"kubernetes.io/projected/18265c8d-6de9-4922-8b89-05b35823a21c-kube-api-access-6nhpt\") pod \"test-curl-7628-b6f680\" (UID: \"18265c8d-6de9-4922-8b89-05b35823a21c\") " pod="opendatahub/test-curl-7628-b6f680" Oct 30 18:37:28.950773 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:28.950494 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7628-b6f680" Oct 30 18:37:29.082172 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:29.082128 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" Oct 30 18:37:29.159170 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:29.159099 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15bc30e5-adb7-45f8-abd5-d5fea229bda7-kube-api-access-skqh8\" (UniqueName: \"kubernetes.io/projected/15bc30e5-adb7-45f8-abd5-d5fea229bda7-kube-api-access-skqh8\") pod \"15bc30e5-adb7-45f8-abd5-d5fea229bda7\" (UID: \"15bc30e5-adb7-45f8-abd5-d5fea229bda7\") " Oct 30 18:37:29.159341 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:29.159219 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15bc30e5-adb7-45f8-abd5-d5fea229bda7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15bc30e5-adb7-45f8-abd5-d5fea229bda7-istio-envoy\") pod \"15bc30e5-adb7-45f8-abd5-d5fea229bda7\" (UID: \"15bc30e5-adb7-45f8-abd5-d5fea229bda7\") " Oct 30 18:37:29.159341 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:29.159274 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15bc30e5-adb7-45f8-abd5-d5fea229bda7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15bc30e5-adb7-45f8-abd5-d5fea229bda7-workload-socket\") pod \"15bc30e5-adb7-45f8-abd5-d5fea229bda7\" (UID: \"15bc30e5-adb7-45f8-abd5-d5fea229bda7\") " Oct 30 18:37:29.159341 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:29.159300 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15bc30e5-adb7-45f8-abd5-d5fea229bda7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15bc30e5-adb7-45f8-abd5-d5fea229bda7-istio-data\") pod \"15bc30e5-adb7-45f8-abd5-d5fea229bda7\" (UID: \"15bc30e5-adb7-45f8-abd5-d5fea229bda7\") " Oct 30 18:37:29.159549 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:29.159355 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15bc30e5-adb7-45f8-abd5-d5fea229bda7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15bc30e5-adb7-45f8-abd5-d5fea229bda7-credential-socket\") pod \"15bc30e5-adb7-45f8-abd5-d5fea229bda7\" (UID: \"15bc30e5-adb7-45f8-abd5-d5fea229bda7\") " Oct 30 18:37:29.159549 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:29.159429 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15bc30e5-adb7-45f8-abd5-d5fea229bda7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15bc30e5-adb7-45f8-abd5-d5fea229bda7-istiod-ca-cert\") pod \"15bc30e5-adb7-45f8-abd5-d5fea229bda7\" (UID: \"15bc30e5-adb7-45f8-abd5-d5fea229bda7\") " Oct 30 18:37:29.159655 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:29.159560 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15bc30e5-adb7-45f8-abd5-d5fea229bda7-istio-token\" (UniqueName: \"kubernetes.io/projected/15bc30e5-adb7-45f8-abd5-d5fea229bda7-istio-token\") pod \"15bc30e5-adb7-45f8-abd5-d5fea229bda7\" (UID: \"15bc30e5-adb7-45f8-abd5-d5fea229bda7\") " Oct 30 18:37:29.159655 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:29.159604 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15bc30e5-adb7-45f8-abd5-d5fea229bda7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15bc30e5-adb7-45f8-abd5-d5fea229bda7-istio-podinfo\") pod \"15bc30e5-adb7-45f8-abd5-d5fea229bda7\" (UID: \"15bc30e5-adb7-45f8-abd5-d5fea229bda7\") " Oct 30 18:37:29.159655 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:29.159625 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15bc30e5-adb7-45f8-abd5-d5fea229bda7-workload-socket" pod "15bc30e5-adb7-45f8-abd5-d5fea229bda7" (UID: "15bc30e5-adb7-45f8-abd5-d5fea229bda7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:29.159655 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:29.159641 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15bc30e5-adb7-45f8-abd5-d5fea229bda7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15bc30e5-adb7-45f8-abd5-d5fea229bda7-workload-certs\") pod \"15bc30e5-adb7-45f8-abd5-d5fea229bda7\" (UID: \"15bc30e5-adb7-45f8-abd5-d5fea229bda7\") " Oct 30 18:37:29.159655 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:29.159636 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15bc30e5-adb7-45f8-abd5-d5fea229bda7-istio-data" pod "15bc30e5-adb7-45f8-abd5-d5fea229bda7" (UID: "15bc30e5-adb7-45f8-abd5-d5fea229bda7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:29.159910 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:29.159888 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15bc30e5-adb7-45f8-abd5-d5fea229bda7-credential-socket" pod "15bc30e5-adb7-45f8-abd5-d5fea229bda7" (UID: "15bc30e5-adb7-45f8-abd5-d5fea229bda7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:29.159978 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:29.159959 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15bc30e5-adb7-45f8-abd5-d5fea229bda7-istiod-ca-cert" pod "15bc30e5-adb7-45f8-abd5-d5fea229bda7" (UID: "15bc30e5-adb7-45f8-abd5-d5fea229bda7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:29.160329 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:29.160302 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15bc30e5-adb7-45f8-abd5-d5fea229bda7-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:29.160448 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:29.160333 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15bc30e5-adb7-45f8-abd5-d5fea229bda7-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:29.160448 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:29.160347 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15bc30e5-adb7-45f8-abd5-d5fea229bda7-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:29.160448 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:29.160360 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15bc30e5-adb7-45f8-abd5-d5fea229bda7-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:29.160642 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:29.160617 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15bc30e5-adb7-45f8-abd5-d5fea229bda7-workload-certs" pod "15bc30e5-adb7-45f8-abd5-d5fea229bda7" (UID: "15bc30e5-adb7-45f8-abd5-d5fea229bda7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:29.162661 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:29.162604 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15bc30e5-adb7-45f8-abd5-d5fea229bda7-istio-envoy" pod "15bc30e5-adb7-45f8-abd5-d5fea229bda7" (UID: "15bc30e5-adb7-45f8-abd5-d5fea229bda7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:29.162880 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:29.162852 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15bc30e5-adb7-45f8-abd5-d5fea229bda7-istio-token" pod "15bc30e5-adb7-45f8-abd5-d5fea229bda7" (UID: "15bc30e5-adb7-45f8-abd5-d5fea229bda7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:29.163179 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:29.163154 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15bc30e5-adb7-45f8-abd5-d5fea229bda7-kube-api-access-skqh8" pod "15bc30e5-adb7-45f8-abd5-d5fea229bda7" (UID: "15bc30e5-adb7-45f8-abd5-d5fea229bda7"). InnerVolumeSpecName "kube-api-access-skqh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:29.163269 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:29.163198 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15bc30e5-adb7-45f8-abd5-d5fea229bda7-istio-podinfo" pod "15bc30e5-adb7-45f8-abd5-d5fea229bda7" (UID: "15bc30e5-adb7-45f8-abd5-d5fea229bda7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:37:29.271088 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:29.261713 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15bc30e5-adb7-45f8-abd5-d5fea229bda7-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:29.271088 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:29.261749 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15bc30e5-adb7-45f8-abd5-d5fea229bda7-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:29.271088 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:29.261762 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15bc30e5-adb7-45f8-abd5-d5fea229bda7-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:29.271088 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:29.261775 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skqh8\" (UniqueName: \"kubernetes.io/projected/15bc30e5-adb7-45f8-abd5-d5fea229bda7-kube-api-access-skqh8\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:29.271088 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:29.261791 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15bc30e5-adb7-45f8-abd5-d5fea229bda7-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:29.281378 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:37:29.281352 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18265c8d_6de9_4922_8b89_05b35823a21c.slice/crio-39026c705e6cf5d479ac77dcd96afd4e3825dc686942137c86d6d9dd9f17e334 WatchSource:0}: Error finding container 39026c705e6cf5d479ac77dcd96afd4e3825dc686942137c86d6d9dd9f17e334: Status 404 returned error can't find the container with id 39026c705e6cf5d479ac77dcd96afd4e3825dc686942137c86d6d9dd9f17e334 Oct 30 18:37:29.281791 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:29.281771 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7628-b6f680"] Oct 30 18:37:29.639340 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:29.638230 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7628-b6f680" event={"ID":"18265c8d-6de9-4922-8b89-05b35823a21c","Type":"ContainerStarted","Data":"ba5544281d7a068a41f76337d10de8e101b793ebad364325cdb5716cfe038ff7"} Oct 30 18:37:29.639340 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:29.638285 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7628-b6f680" event={"ID":"18265c8d-6de9-4922-8b89-05b35823a21c","Type":"ContainerStarted","Data":"39026c705e6cf5d479ac77dcd96afd4e3825dc686942137c86d6d9dd9f17e334"} Oct 30 18:37:29.645174 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:29.641702 4442 generic.go:356] "Generic (PLEG): container finished" podID="15bc30e5-adb7-45f8-abd5-d5fea229bda7" containerID="93e0da9fffce6f5476844d2bfe20733255f86ff8b7daafd1ff650e670961f716" exitCode=0 Oct 30 18:37:29.645174 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:29.641982 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7628-b718f6" podUID="4f6ddd11-474f-43c6-8cea-9d6140bdcc65" containerName="test-curl-7628-b718f6" containerID="cri-o://6b224ad0473994861ff292491e4c3a2bf0802543a1904b99d62af29fe3da37d4" gracePeriod=2 Oct 30 18:37:29.645174 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:29.642359 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" Oct 30 18:37:29.650037 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:29.650007 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" event={"ID":"15bc30e5-adb7-45f8-abd5-d5fea229bda7","Type":"ContainerDied","Data":"93e0da9fffce6f5476844d2bfe20733255f86ff8b7daafd1ff650e670961f716"} Oct 30 18:37:29.650259 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:29.650242 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn" event={"ID":"15bc30e5-adb7-45f8-abd5-d5fea229bda7","Type":"ContainerDied","Data":"bb6e0c7b97cab0149640daf0eeec4aa4f770215c1b8870e2d5390d2fd522b188"} Oct 30 18:37:29.650465 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:29.650396 4442 scope.go:122] "RemoveContainer" containerID="93e0da9fffce6f5476844d2bfe20733255f86ff8b7daafd1ff650e670961f716" Oct 30 18:37:29.678380 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:29.677278 4442 scope.go:122] "RemoveContainer" containerID="93e0da9fffce6f5476844d2bfe20733255f86ff8b7daafd1ff650e670961f716" Oct 30 18:37:29.678380 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:37:29.677740 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93e0da9fffce6f5476844d2bfe20733255f86ff8b7daafd1ff650e670961f716\": container with ID starting with 93e0da9fffce6f5476844d2bfe20733255f86ff8b7daafd1ff650e670961f716 not found: ID does not exist" containerID="93e0da9fffce6f5476844d2bfe20733255f86ff8b7daafd1ff650e670961f716" Oct 30 18:37:29.678380 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:29.677776 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93e0da9fffce6f5476844d2bfe20733255f86ff8b7daafd1ff650e670961f716"} err="failed to get container status \"93e0da9fffce6f5476844d2bfe20733255f86ff8b7daafd1ff650e670961f716\": rpc error: code = NotFound desc = could not find container \"93e0da9fffce6f5476844d2bfe20733255f86ff8b7daafd1ff650e670961f716\": container with ID starting with 93e0da9fffce6f5476844d2bfe20733255f86ff8b7daafd1ff650e670961f716 not found: ID does not exist" Oct 30 18:37:29.691191 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:29.689275 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7628-b6f680" podStartSLOduration=1.689258819 podStartE2EDuration="1.689258819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:29.685320403 +0000 UTC m=+2346.775382743" watchObservedRunningTime="2026-10-30 18:37:29.689258819 +0000 UTC m=+2346.779321157" Oct 30 18:37:29.725290 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:29.725257 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn"] Oct 30 18:37:29.730773 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:29.730744 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-l72wn"] Oct 30 18:37:30.498167 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:30.498105 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-236fc4-c8b87c5ff-9knfc" Oct 30 18:37:31.067258 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:31.067227 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15bc30e5-adb7-45f8-abd5-d5fea229bda7" path="/var/lib/kubelet/pods/15bc30e5-adb7-45f8-abd5-d5fea229bda7/volumes" Oct 30 18:37:31.882235 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:31.882208 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7628-b718f6" Oct 30 18:37:31.936334 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:31.936244 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f6ddd11-474f-43c6-8cea-9d6140bdcc65-kube-api-access-j2kwt\" (UniqueName: \"kubernetes.io/projected/4f6ddd11-474f-43c6-8cea-9d6140bdcc65-kube-api-access-j2kwt\") pod \"4f6ddd11-474f-43c6-8cea-9d6140bdcc65\" (UID: \"4f6ddd11-474f-43c6-8cea-9d6140bdcc65\") " Oct 30 18:37:31.938833 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:31.938801 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f6ddd11-474f-43c6-8cea-9d6140bdcc65-kube-api-access-j2kwt" pod "4f6ddd11-474f-43c6-8cea-9d6140bdcc65" (UID: "4f6ddd11-474f-43c6-8cea-9d6140bdcc65"). InnerVolumeSpecName "kube-api-access-j2kwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:32.037370 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:32.037335 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2kwt\" (UniqueName: \"kubernetes.io/projected/4f6ddd11-474f-43c6-8cea-9d6140bdcc65-kube-api-access-j2kwt\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:32.684686 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:32.684640 4442 generic.go:356] "Generic (PLEG): container finished" podID="4f6ddd11-474f-43c6-8cea-9d6140bdcc65" containerID="6b224ad0473994861ff292491e4c3a2bf0802543a1904b99d62af29fe3da37d4" exitCode=137 Oct 30 18:37:32.685088 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:32.684885 4442 scope.go:122] "RemoveContainer" containerID="6b224ad0473994861ff292491e4c3a2bf0802543a1904b99d62af29fe3da37d4" Oct 30 18:37:32.685088 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:32.685028 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7628-b718f6" Oct 30 18:37:32.697613 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:32.697576 4442 scope.go:122] "RemoveContainer" containerID="6b224ad0473994861ff292491e4c3a2bf0802543a1904b99d62af29fe3da37d4" Oct 30 18:37:32.698018 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:37:32.697992 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b224ad0473994861ff292491e4c3a2bf0802543a1904b99d62af29fe3da37d4\": container with ID starting with 6b224ad0473994861ff292491e4c3a2bf0802543a1904b99d62af29fe3da37d4 not found: ID does not exist" containerID="6b224ad0473994861ff292491e4c3a2bf0802543a1904b99d62af29fe3da37d4" Oct 30 18:37:32.698108 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:32.698031 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b224ad0473994861ff292491e4c3a2bf0802543a1904b99d62af29fe3da37d4"} err="failed to get container status \"6b224ad0473994861ff292491e4c3a2bf0802543a1904b99d62af29fe3da37d4\": rpc error: code = NotFound desc = could not find container \"6b224ad0473994861ff292491e4c3a2bf0802543a1904b99d62af29fe3da37d4\": container with ID starting with 6b224ad0473994861ff292491e4c3a2bf0802543a1904b99d62af29fe3da37d4 not found: ID does not exist" Oct 30 18:37:32.708663 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:37:32.708635 4442 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7628-b718f6\" is forbidden: User \"system:node:ip-10-0-138-176.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-176.ec2.internal' and this object" podUID="4f6ddd11-474f-43c6-8cea-9d6140bdcc65" pod="opendatahub/test-curl-7628-b718f6" Oct 30 18:37:33.065677 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:33.065587 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f6ddd11-474f-43c6-8cea-9d6140bdcc65" path="/var/lib/kubelet/pods/4f6ddd11-474f-43c6-8cea-9d6140bdcc65/volumes" Oct 30 18:37:33.073872 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:37:33.073838 4442 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7628-b718f6\" is forbidden: User \"system:node:ip-10-0-138-176.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-176.ec2.internal' and this object" podUID="4f6ddd11-474f-43c6-8cea-9d6140bdcc65" pod="opendatahub/test-curl-7628-b718f6" Oct 30 18:37:41.396585 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:41.396545 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-24b124/auth-test-model-24b124-kserve-745594bcc-6lt65" podUID="8f10e975-c54c-4949-a213-40fc7d4ffe1f" containerName="main" containerID="cri-o://ebdf55d0953228961c3e84c63e5b1ec5984a59228e5b8eb99f3300d90686753d" gracePeriod=45 Oct 30 18:37:42.115215 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:42.108636 4442 generic.go:356] "Generic (PLEG): container finished" podID="8f10e975-c54c-4949-a213-40fc7d4ffe1f" containerID="ebdf55d0953228961c3e84c63e5b1ec5984a59228e5b8eb99f3300d90686753d" exitCode=0 Oct 30 18:37:42.115215 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:42.108846 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-24b124/auth-test-model-24b124-kserve-745594bcc-6lt65" event={"ID":"8f10e975-c54c-4949-a213-40fc7d4ffe1f","Type":"ContainerDied","Data":"ebdf55d0953228961c3e84c63e5b1ec5984a59228e5b8eb99f3300d90686753d"} Oct 30 18:37:42.240159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:42.234414 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-24b124/auth-test-model-24b124-kserve-745594bcc-6lt65" Oct 30 18:37:42.254167 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:42.252976 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f10e975-c54c-4949-a213-40fc7d4ffe1f-kube-api-access-8dpz4\" (UniqueName: \"kubernetes.io/projected/8f10e975-c54c-4949-a213-40fc7d4ffe1f-kube-api-access-8dpz4\") pod \"8f10e975-c54c-4949-a213-40fc7d4ffe1f\" (UID: \"8f10e975-c54c-4949-a213-40fc7d4ffe1f\") " Oct 30 18:37:42.254167 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:42.253036 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f10e975-c54c-4949-a213-40fc7d4ffe1f-home\" (UniqueName: \"kubernetes.io/empty-dir/8f10e975-c54c-4949-a213-40fc7d4ffe1f-home\") pod \"8f10e975-c54c-4949-a213-40fc7d4ffe1f\" (UID: \"8f10e975-c54c-4949-a213-40fc7d4ffe1f\") " Oct 30 18:37:42.254167 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:42.253067 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f10e975-c54c-4949-a213-40fc7d4ffe1f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f10e975-c54c-4949-a213-40fc7d4ffe1f-dshm\") pod \"8f10e975-c54c-4949-a213-40fc7d4ffe1f\" (UID: \"8f10e975-c54c-4949-a213-40fc7d4ffe1f\") " Oct 30 18:37:42.254167 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:42.253168 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f10e975-c54c-4949-a213-40fc7d4ffe1f-tls-certs\" (UniqueName: \"kubernetes.io/secret/8f10e975-c54c-4949-a213-40fc7d4ffe1f-tls-certs\") pod \"8f10e975-c54c-4949-a213-40fc7d4ffe1f\" (UID: \"8f10e975-c54c-4949-a213-40fc7d4ffe1f\") " Oct 30 18:37:42.254167 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:42.253237 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f10e975-c54c-4949-a213-40fc7d4ffe1f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f10e975-c54c-4949-a213-40fc7d4ffe1f-tmp-dir\") pod \"8f10e975-c54c-4949-a213-40fc7d4ffe1f\" (UID: \"8f10e975-c54c-4949-a213-40fc7d4ffe1f\") " Oct 30 18:37:42.254167 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:42.253313 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f10e975-c54c-4949-a213-40fc7d4ffe1f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f10e975-c54c-4949-a213-40fc7d4ffe1f-model-cache\") pod \"8f10e975-c54c-4949-a213-40fc7d4ffe1f\" (UID: \"8f10e975-c54c-4949-a213-40fc7d4ffe1f\") " Oct 30 18:37:42.265156 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:42.264436 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f10e975-c54c-4949-a213-40fc7d4ffe1f-tmp-dir" pod "8f10e975-c54c-4949-a213-40fc7d4ffe1f" (UID: "8f10e975-c54c-4949-a213-40fc7d4ffe1f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:42.265156 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:42.264711 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f10e975-c54c-4949-a213-40fc7d4ffe1f-home" pod "8f10e975-c54c-4949-a213-40fc7d4ffe1f" (UID: "8f10e975-c54c-4949-a213-40fc7d4ffe1f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:42.265156 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:42.264812 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f10e975-c54c-4949-a213-40fc7d4ffe1f-model-cache" pod "8f10e975-c54c-4949-a213-40fc7d4ffe1f" (UID: "8f10e975-c54c-4949-a213-40fc7d4ffe1f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:42.299271 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:42.293683 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f10e975-c54c-4949-a213-40fc7d4ffe1f-kube-api-access-8dpz4" pod "8f10e975-c54c-4949-a213-40fc7d4ffe1f" (UID: "8f10e975-c54c-4949-a213-40fc7d4ffe1f"). InnerVolumeSpecName "kube-api-access-8dpz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:42.299271 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:42.293782 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f10e975-c54c-4949-a213-40fc7d4ffe1f-tls-certs" pod "8f10e975-c54c-4949-a213-40fc7d4ffe1f" (UID: "8f10e975-c54c-4949-a213-40fc7d4ffe1f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:42.299271 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:42.293839 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f10e975-c54c-4949-a213-40fc7d4ffe1f-dshm" pod "8f10e975-c54c-4949-a213-40fc7d4ffe1f" (UID: "8f10e975-c54c-4949-a213-40fc7d4ffe1f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:42.364165 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:42.357239 4442 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f10e975-c54c-4949-a213-40fc7d4ffe1f-tmp-dir\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:42.364165 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:42.357278 4442 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f10e975-c54c-4949-a213-40fc7d4ffe1f-model-cache\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:42.364165 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:42.357292 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dpz4\" (UniqueName: \"kubernetes.io/projected/8f10e975-c54c-4949-a213-40fc7d4ffe1f-kube-api-access-8dpz4\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:42.364165 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:42.357305 4442 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f10e975-c54c-4949-a213-40fc7d4ffe1f-home\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:42.364165 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:42.357322 4442 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f10e975-c54c-4949-a213-40fc7d4ffe1f-dshm\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:42.364165 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:42.357337 4442 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f10e975-c54c-4949-a213-40fc7d4ffe1f-tls-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:42.849920 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:42.842239 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-24b124/auth-test-model-24b124-kserve-745594bcc-6lt65" podUID="8f10e975-c54c-4949-a213-40fc7d4ffe1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.199:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:37:43.172895 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:43.171269 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-24b124/auth-test-model-24b124-kserve-745594bcc-6lt65" Oct 30 18:37:43.175772 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:43.175740 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-24b124/auth-test-model-24b124-kserve-745594bcc-6lt65" event={"ID":"8f10e975-c54c-4949-a213-40fc7d4ffe1f","Type":"ContainerDied","Data":"4496126b3c8874fcf0222fdbe6a3223fdbb48234001a1233733e7c91884227a5"} Oct 30 18:37:43.176031 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:43.176010 4442 scope.go:122] "RemoveContainer" containerID="ebdf55d0953228961c3e84c63e5b1ec5984a59228e5b8eb99f3300d90686753d" Oct 30 18:37:43.275346 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:43.275308 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-24b124/auth-test-model-24b124-kserve-745594bcc-6lt65"] Oct 30 18:37:43.297142 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:43.297110 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-24b124/auth-test-model-24b124-kserve-745594bcc-6lt65"] Oct 30 18:37:45.074216 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:45.074182 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f10e975-c54c-4949-a213-40fc7d4ffe1f" path="/var/lib/kubelet/pods/8f10e975-c54c-4949-a213-40fc7d4ffe1f/volumes" Oct 30 18:37:46.044271 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:46.039988 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18308ff8-6wbzx"] Oct 30 18:37:46.044271 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:46.040938 4442 cpu_manager.go:414] "containerMap: removing container" podUID="15bc30e5-adb7-45f8-abd5-d5fea229bda7" containerName="istio-proxy" Oct 30 18:37:46.044271 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:46.040960 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15bc30e5-adb7-45f8-abd5-d5fea229bda7" containerName="istio-proxy" Oct 30 18:37:46.044271 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:46.041001 4442 cpu_manager.go:414] "containerMap: removing container" podUID="8f10e975-c54c-4949-a213-40fc7d4ffe1f" containerName="main" Oct 30 18:37:46.044271 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:46.041009 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f10e975-c54c-4949-a213-40fc7d4ffe1f" containerName="main" Oct 30 18:37:46.044271 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:46.041160 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f10e975-c54c-4949-a213-40fc7d4ffe1f" containerName="main" Oct 30 18:37:46.044271 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:46.041185 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="15bc30e5-adb7-45f8-abd5-d5fea229bda7" containerName="istio-proxy" Oct 30 18:37:46.059826 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:46.059800 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18308ff8-6wbzx" Oct 30 18:37:46.062210 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:46.062184 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18308ff8-6wbzx"] Oct 30 18:37:46.076638 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:46.076616 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:37:46.087854 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:46.087821 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:37:46.119274 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:46.119248 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24975938-abe1-4fdf-9f9b-c0d37f84c394-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18308ff8-6wbzx\" (UID: \"24975938-abe1-4fdf-9f9b-c0d37f84c394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18308ff8-6wbzx" Oct 30 18:37:46.119443 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:46.119287 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl7mj\" (UniqueName: \"kubernetes.io/projected/24975938-abe1-4fdf-9f9b-c0d37f84c394-kube-api-access-hl7mj\") pod \"maas-api-revoke-sub-18308ff8-6wbzx\" (UID: \"24975938-abe1-4fdf-9f9b-c0d37f84c394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18308ff8-6wbzx" Oct 30 18:37:46.119529 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:46.119459 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24975938-abe1-4fdf-9f9b-c0d37f84c394-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18308ff8-6wbzx\" (UID: \"24975938-abe1-4fdf-9f9b-c0d37f84c394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18308ff8-6wbzx" Oct 30 18:37:46.220629 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:46.220591 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24975938-abe1-4fdf-9f9b-c0d37f84c394-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18308ff8-6wbzx\" (UID: \"24975938-abe1-4fdf-9f9b-c0d37f84c394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18308ff8-6wbzx" Oct 30 18:37:46.220823 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:46.220640 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl7mj\" (UniqueName: \"kubernetes.io/projected/24975938-abe1-4fdf-9f9b-c0d37f84c394-kube-api-access-hl7mj\") pod \"maas-api-revoke-sub-18308ff8-6wbzx\" (UID: \"24975938-abe1-4fdf-9f9b-c0d37f84c394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18308ff8-6wbzx" Oct 30 18:37:46.220823 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:46.220763 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24975938-abe1-4fdf-9f9b-c0d37f84c394-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18308ff8-6wbzx\" (UID: \"24975938-abe1-4fdf-9f9b-c0d37f84c394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18308ff8-6wbzx" Oct 30 18:37:46.221401 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:46.221381 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24975938-abe1-4fdf-9f9b-c0d37f84c394-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18308ff8-6wbzx\" (UID: \"24975938-abe1-4fdf-9f9b-c0d37f84c394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18308ff8-6wbzx" Oct 30 18:37:46.232278 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:46.232251 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24975938-abe1-4fdf-9f9b-c0d37f84c394-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18308ff8-6wbzx\" (UID: \"24975938-abe1-4fdf-9f9b-c0d37f84c394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18308ff8-6wbzx" Oct 30 18:37:46.258319 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:46.258257 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl7mj\" (UniqueName: \"kubernetes.io/projected/24975938-abe1-4fdf-9f9b-c0d37f84c394-kube-api-access-hl7mj\") pod \"maas-api-revoke-sub-18308ff8-6wbzx\" (UID: \"24975938-abe1-4fdf-9f9b-c0d37f84c394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18308ff8-6wbzx" Oct 30 18:37:46.380518 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:46.380485 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18308ff8-6wbzx" Oct 30 18:37:46.712651 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:46.712295 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18308ff8-6wbzx"] Oct 30 18:37:46.712932 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:37:46.712892 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24975938_abe1_4fdf_9f9b_c0d37f84c394.slice/crio-7e079dc5a6cf1057c262cb88a4530618c1eeb79cfa95b7bd90e8e0061421c23d WatchSource:0}: Error finding container 7e079dc5a6cf1057c262cb88a4530618c1eeb79cfa95b7bd90e8e0061421c23d: Status 404 returned error can't find the container with id 7e079dc5a6cf1057c262cb88a4530618c1eeb79cfa95b7bd90e8e0061421c23d Oct 30 18:37:46.863797 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:46.863759 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk"] Oct 30 18:37:46.869564 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:46.869537 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" Oct 30 18:37:46.891402 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:46.891360 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk"] Oct 30 18:37:46.895830 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:46.895432 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-2107d4-openshift-default-dockercfg-94jvk\"" Oct 30 18:37:46.958107 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:46.957216 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2be17bb-1852-432e-a24d-24d762b27661-istio-podinfo\") pod \"e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk\" (UID: \"c2be17bb-1852-432e-a24d-24d762b27661\") " pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" Oct 30 18:37:46.958107 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:46.957382 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mrw8\" (UniqueName: \"kubernetes.io/projected/c2be17bb-1852-432e-a24d-24d762b27661-kube-api-access-8mrw8\") pod \"e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk\" (UID: \"c2be17bb-1852-432e-a24d-24d762b27661\") " pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" Oct 30 18:37:46.958107 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:46.957442 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2be17bb-1852-432e-a24d-24d762b27661-istiod-ca-cert\") pod \"e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk\" (UID: \"c2be17bb-1852-432e-a24d-24d762b27661\") " pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" Oct 30 18:37:46.958107 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:46.957468 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2be17bb-1852-432e-a24d-24d762b27661-credential-socket\") pod \"e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk\" (UID: \"c2be17bb-1852-432e-a24d-24d762b27661\") " pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" Oct 30 18:37:46.958107 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:46.957495 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2be17bb-1852-432e-a24d-24d762b27661-istio-data\") pod \"e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk\" (UID: \"c2be17bb-1852-432e-a24d-24d762b27661\") " pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" Oct 30 18:37:46.958107 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:46.957542 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2be17bb-1852-432e-a24d-24d762b27661-istio-envoy\") pod \"e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk\" (UID: \"c2be17bb-1852-432e-a24d-24d762b27661\") " pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" Oct 30 18:37:46.958107 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:46.957592 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2be17bb-1852-432e-a24d-24d762b27661-istio-token\") pod \"e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk\" (UID: \"c2be17bb-1852-432e-a24d-24d762b27661\") " pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" Oct 30 18:37:46.958107 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:46.957625 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2be17bb-1852-432e-a24d-24d762b27661-workload-socket\") pod \"e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk\" (UID: \"c2be17bb-1852-432e-a24d-24d762b27661\") " pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" Oct 30 18:37:46.958107 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:46.957700 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2be17bb-1852-432e-a24d-24d762b27661-workload-certs\") pod \"e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk\" (UID: \"c2be17bb-1852-432e-a24d-24d762b27661\") " pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" Oct 30 18:37:47.068207 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:47.065975 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mrw8\" (UniqueName: \"kubernetes.io/projected/c2be17bb-1852-432e-a24d-24d762b27661-kube-api-access-8mrw8\") pod \"e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk\" (UID: \"c2be17bb-1852-432e-a24d-24d762b27661\") " pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" Oct 30 18:37:47.068207 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:47.066064 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2be17bb-1852-432e-a24d-24d762b27661-istiod-ca-cert\") pod \"e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk\" (UID: \"c2be17bb-1852-432e-a24d-24d762b27661\") " pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" Oct 30 18:37:47.068207 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:47.066094 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2be17bb-1852-432e-a24d-24d762b27661-credential-socket\") pod \"e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk\" (UID: \"c2be17bb-1852-432e-a24d-24d762b27661\") " pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" Oct 30 18:37:47.068207 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:47.066124 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2be17bb-1852-432e-a24d-24d762b27661-istio-data\") pod \"e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk\" (UID: \"c2be17bb-1852-432e-a24d-24d762b27661\") " pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" Oct 30 18:37:47.068207 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:47.066192 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2be17bb-1852-432e-a24d-24d762b27661-istio-envoy\") pod \"e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk\" (UID: \"c2be17bb-1852-432e-a24d-24d762b27661\") " pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" Oct 30 18:37:47.068207 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:47.066245 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2be17bb-1852-432e-a24d-24d762b27661-istio-token\") pod \"e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk\" (UID: \"c2be17bb-1852-432e-a24d-24d762b27661\") " pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" Oct 30 18:37:47.068207 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:47.066281 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2be17bb-1852-432e-a24d-24d762b27661-workload-socket\") pod \"e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk\" (UID: \"c2be17bb-1852-432e-a24d-24d762b27661\") " pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" Oct 30 18:37:47.068207 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:47.066360 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2be17bb-1852-432e-a24d-24d762b27661-workload-certs\") pod \"e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk\" (UID: \"c2be17bb-1852-432e-a24d-24d762b27661\") " pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" Oct 30 18:37:47.068207 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:47.066411 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2be17bb-1852-432e-a24d-24d762b27661-istio-podinfo\") pod \"e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk\" (UID: \"c2be17bb-1852-432e-a24d-24d762b27661\") " pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" Oct 30 18:37:47.173183 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:47.074706 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2be17bb-1852-432e-a24d-24d762b27661-istio-data\") pod \"e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk\" (UID: \"c2be17bb-1852-432e-a24d-24d762b27661\") " pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" Oct 30 18:37:47.173183 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:47.080046 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2be17bb-1852-432e-a24d-24d762b27661-workload-socket\") pod \"e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk\" (UID: \"c2be17bb-1852-432e-a24d-24d762b27661\") " pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" Oct 30 18:37:47.173183 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:47.080332 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2be17bb-1852-432e-a24d-24d762b27661-credential-socket\") pod \"e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk\" (UID: \"c2be17bb-1852-432e-a24d-24d762b27661\") " pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" Oct 30 18:37:47.173183 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:47.080883 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2be17bb-1852-432e-a24d-24d762b27661-workload-certs\") pod \"e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk\" (UID: \"c2be17bb-1852-432e-a24d-24d762b27661\") " pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" Oct 30 18:37:47.173183 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:47.100704 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2be17bb-1852-432e-a24d-24d762b27661-istio-podinfo\") pod \"e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk\" (UID: \"c2be17bb-1852-432e-a24d-24d762b27661\") " pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" Oct 30 18:37:47.173183 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:47.116321 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2be17bb-1852-432e-a24d-24d762b27661-istiod-ca-cert\") pod \"e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk\" (UID: \"c2be17bb-1852-432e-a24d-24d762b27661\") " pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" Oct 30 18:37:47.173183 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:47.123426 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mrw8\" (UniqueName: \"kubernetes.io/projected/c2be17bb-1852-432e-a24d-24d762b27661-kube-api-access-8mrw8\") pod \"e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk\" (UID: \"c2be17bb-1852-432e-a24d-24d762b27661\") " pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" Oct 30 18:37:47.173183 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:47.123941 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2be17bb-1852-432e-a24d-24d762b27661-istio-token\") pod \"e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk\" (UID: \"c2be17bb-1852-432e-a24d-24d762b27661\") " pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" Oct 30 18:37:47.173183 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:47.145333 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2be17bb-1852-432e-a24d-24d762b27661-istio-envoy\") pod \"e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk\" (UID: \"c2be17bb-1852-432e-a24d-24d762b27661\") " pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" Oct 30 18:37:47.201453 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:47.199286 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" Oct 30 18:37:47.271994 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:47.271961 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18308ff8-6wbzx" event={"ID":"24975938-abe1-4fdf-9f9b-c0d37f84c394","Type":"ContainerStarted","Data":"7e079dc5a6cf1057c262cb88a4530618c1eeb79cfa95b7bd90e8e0061421c23d"} Oct 30 18:37:47.586572 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:37:47.586070 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:37:47.608688 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:47.608662 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk"] Oct 30 18:37:47.614242 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:47.614214 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:37:47.614386 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:47.614374 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:37:47.614464 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:47.614454 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:37:48.283118 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:48.283070 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" event={"ID":"c2be17bb-1852-432e-a24d-24d762b27661","Type":"ContainerStarted","Data":"5898fae1e077c4b36892e8caf35c10bcb3ca6c3a9ffe4ff6b724a2c19ca0413b"} Oct 30 18:37:48.283619 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:48.283190 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" event={"ID":"c2be17bb-1852-432e-a24d-24d762b27661","Type":"ContainerStarted","Data":"69e0e9d4bd0a42a95bfc128355b3f10c16beb93f3de33cdd16242d75b8957235"} Oct 30 18:37:48.284890 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:48.284861 4442 generic.go:356] "Generic (PLEG): container finished" podID="24975938-abe1-4fdf-9f9b-c0d37f84c394" containerID="1eb00d811c0953636c500831ce53df75d45680785a7fe2beba6e08d628bc36c2" exitCode=0 Oct 30 18:37:48.285008 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:48.284902 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18308ff8-6wbzx" event={"ID":"24975938-abe1-4fdf-9f9b-c0d37f84c394","Type":"ContainerDied","Data":"1eb00d811c0953636c500831ce53df75d45680785a7fe2beba6e08d628bc36c2"} Oct 30 18:37:48.335653 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:48.335589 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" podStartSLOduration=2.335570174 podStartE2EDuration="2.335570174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:48.334888033 +0000 UTC m=+2365.424950372" watchObservedRunningTime="2026-10-30 18:37:48.335570174 +0000 UTC m=+2365.425632515" Oct 30 18:37:49.200576 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:49.200533 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" Oct 30 18:37:49.206970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:49.206943 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" Oct 30 18:37:49.292076 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:49.292045 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" Oct 30 18:37:49.293342 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:49.293323 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" Oct 30 18:37:49.524196 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:49.524169 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18308ff8-6wbzx" Oct 30 18:37:49.614872 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:49.614837 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24975938-abe1-4fdf-9f9b-c0d37f84c394-kube-api-access-hl7mj\" (UniqueName: \"kubernetes.io/projected/24975938-abe1-4fdf-9f9b-c0d37f84c394-kube-api-access-hl7mj\") pod \"24975938-abe1-4fdf-9f9b-c0d37f84c394\" (UID: \"24975938-abe1-4fdf-9f9b-c0d37f84c394\") " Oct 30 18:37:49.615070 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:49.614924 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24975938-abe1-4fdf-9f9b-c0d37f84c394-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24975938-abe1-4fdf-9f9b-c0d37f84c394-maas-api-xks-ca\") pod \"24975938-abe1-4fdf-9f9b-c0d37f84c394\" (UID: \"24975938-abe1-4fdf-9f9b-c0d37f84c394\") " Oct 30 18:37:49.615070 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:49.615041 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24975938-abe1-4fdf-9f9b-c0d37f84c394-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24975938-abe1-4fdf-9f9b-c0d37f84c394-maas-api-openshift-ca\") pod \"24975938-abe1-4fdf-9f9b-c0d37f84c394\" (UID: \"24975938-abe1-4fdf-9f9b-c0d37f84c394\") " Oct 30 18:37:49.616047 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:49.616015 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24975938-abe1-4fdf-9f9b-c0d37f84c394-maas-api-openshift-ca" pod "24975938-abe1-4fdf-9f9b-c0d37f84c394" (UID: "24975938-abe1-4fdf-9f9b-c0d37f84c394"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:49.618386 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:49.618360 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24975938-abe1-4fdf-9f9b-c0d37f84c394-maas-api-xks-ca" pod "24975938-abe1-4fdf-9f9b-c0d37f84c394" (UID: "24975938-abe1-4fdf-9f9b-c0d37f84c394"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:49.618386 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:49.618363 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24975938-abe1-4fdf-9f9b-c0d37f84c394-kube-api-access-hl7mj" pod "24975938-abe1-4fdf-9f9b-c0d37f84c394" (UID: "24975938-abe1-4fdf-9f9b-c0d37f84c394"). InnerVolumeSpecName "kube-api-access-hl7mj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:49.717467 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:49.717426 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24975938-abe1-4fdf-9f9b-c0d37f84c394-maas-api-xks-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:49.717467 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:49.717462 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24975938-abe1-4fdf-9f9b-c0d37f84c394-maas-api-openshift-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:49.717467 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:49.717474 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl7mj\" (UniqueName: \"kubernetes.io/projected/24975938-abe1-4fdf-9f9b-c0d37f84c394-kube-api-access-hl7mj\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:50.300957 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:50.300928 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18308ff8-6wbzx" Oct 30 18:37:50.301392 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:50.300954 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18308ff8-6wbzx" event={"ID":"24975938-abe1-4fdf-9f9b-c0d37f84c394","Type":"ContainerDied","Data":"7e079dc5a6cf1057c262cb88a4530618c1eeb79cfa95b7bd90e8e0061421c23d"} Oct 30 18:37:50.301392 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:50.301005 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e079dc5a6cf1057c262cb88a4530618c1eeb79cfa95b7bd90e8e0061421c23d" Oct 30 18:37:52.317693 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:52.317661 4442 generic.go:356] "Generic (PLEG): container finished" podID="bfbb6b47-1274-4d3d-8845-627898246065" containerID="09643b155ae61d3db212a12fe01e8b94e6d8211a3180916351b902bcce2dad97" exitCode=0 Oct 30 18:37:52.318177 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:52.317737 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" event={"ID":"bfbb6b47-1274-4d3d-8845-627898246065","Type":"ContainerDied","Data":"09643b155ae61d3db212a12fe01e8b94e6d8211a3180916351b902bcce2dad97"} Oct 30 18:37:52.318234 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:52.318196 4442 scope.go:122] "RemoveContainer" containerID="09643b155ae61d3db212a12fe01e8b94e6d8211a3180916351b902bcce2dad97" Oct 30 18:37:52.319855 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:52.319825 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:37:52.319949 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:52.319895 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:37:52.319949 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:52.319926 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:37:53.352157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:53.351845 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" event={"ID":"bfbb6b47-1274-4d3d-8845-627898246065","Type":"ContainerStarted","Data":"1ff4a0cd501cafc0d3c80ef1c5a352e444fdb8bd53b65bda59ea6122c68c33b9"} Oct 30 18:37:53.956558 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:53.956519 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" Oct 30 18:37:53.957678 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:53.957647 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 30 18:37:53.957804 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:53.957708 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" podUID="bfbb6b47-1274-4d3d-8845-627898246065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 30 18:37:54.957581 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:54.957225 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 30 18:37:54.957581 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:54.957284 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" podUID="bfbb6b47-1274-4d3d-8845-627898246065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 30 18:37:55.956554 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:55.956508 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 30 18:37:55.956778 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:55.956595 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" podUID="bfbb6b47-1274-4d3d-8845-627898246065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 30 18:37:56.407502 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:56.407470 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c46f6e24-rv2dx"] Oct 30 18:37:56.408142 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:56.408115 4442 cpu_manager.go:414] "containerMap: removing container" podUID="24975938-abe1-4fdf-9f9b-c0d37f84c394" containerName="revoke-keys" Oct 30 18:37:56.408212 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:56.408143 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24975938-abe1-4fdf-9f9b-c0d37f84c394" containerName="revoke-keys" Oct 30 18:37:56.408297 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:56.408272 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="24975938-abe1-4fdf-9f9b-c0d37f84c394" containerName="revoke-keys" Oct 30 18:37:56.410788 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:56.410771 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46f6e24-rv2dx" Oct 30 18:37:56.419329 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:56.419304 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b737eec0-6c8e-4bf2-9a62-286bae2d2b57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c46f6e24-rv2dx\" (UID: \"b737eec0-6c8e-4bf2-9a62-286bae2d2b57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46f6e24-rv2dx" Oct 30 18:37:56.419550 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:56.419526 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b737eec0-6c8e-4bf2-9a62-286bae2d2b57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c46f6e24-rv2dx\" (UID: \"b737eec0-6c8e-4bf2-9a62-286bae2d2b57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46f6e24-rv2dx" Oct 30 18:37:56.419633 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:56.419580 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2lf6\" (UniqueName: \"kubernetes.io/projected/b737eec0-6c8e-4bf2-9a62-286bae2d2b57-kube-api-access-b2lf6\") pod \"maas-api-revoke-sub-c46f6e24-rv2dx\" (UID: \"b737eec0-6c8e-4bf2-9a62-286bae2d2b57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46f6e24-rv2dx" Oct 30 18:37:56.429495 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:56.429472 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:37:56.467971 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:56.467938 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:37:56.474698 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:56.474658 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c46f6e24-rv2dx"] Oct 30 18:37:56.520937 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:56.520905 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2lf6\" (UniqueName: \"kubernetes.io/projected/b737eec0-6c8e-4bf2-9a62-286bae2d2b57-kube-api-access-b2lf6\") pod \"maas-api-revoke-sub-c46f6e24-rv2dx\" (UID: \"b737eec0-6c8e-4bf2-9a62-286bae2d2b57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46f6e24-rv2dx" Oct 30 18:37:56.521111 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:56.520951 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b737eec0-6c8e-4bf2-9a62-286bae2d2b57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c46f6e24-rv2dx\" (UID: \"b737eec0-6c8e-4bf2-9a62-286bae2d2b57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46f6e24-rv2dx" Oct 30 18:37:56.521111 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:56.521088 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b737eec0-6c8e-4bf2-9a62-286bae2d2b57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c46f6e24-rv2dx\" (UID: \"b737eec0-6c8e-4bf2-9a62-286bae2d2b57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46f6e24-rv2dx" Oct 30 18:37:56.521786 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:56.521766 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b737eec0-6c8e-4bf2-9a62-286bae2d2b57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c46f6e24-rv2dx\" (UID: \"b737eec0-6c8e-4bf2-9a62-286bae2d2b57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46f6e24-rv2dx" Oct 30 18:37:56.537159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:56.530660 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b737eec0-6c8e-4bf2-9a62-286bae2d2b57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c46f6e24-rv2dx\" (UID: \"b737eec0-6c8e-4bf2-9a62-286bae2d2b57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46f6e24-rv2dx" Oct 30 18:37:56.564386 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:56.564344 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2lf6\" (UniqueName: \"kubernetes.io/projected/b737eec0-6c8e-4bf2-9a62-286bae2d2b57-kube-api-access-b2lf6\") pod \"maas-api-revoke-sub-c46f6e24-rv2dx\" (UID: \"b737eec0-6c8e-4bf2-9a62-286bae2d2b57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46f6e24-rv2dx" Oct 30 18:37:56.745213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:56.729298 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46f6e24-rv2dx" Oct 30 18:37:56.956298 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:56.956256 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 30 18:37:56.956513 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:56.956335 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" podUID="bfbb6b47-1274-4d3d-8845-627898246065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 30 18:37:57.018830 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:57.018728 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7622-99ea9a"] Oct 30 18:37:57.047158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:57.034460 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7622-99ea9a" Oct 30 18:37:57.081185 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:57.081119 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7622-99ea9a"] Oct 30 18:37:57.081328 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:57.081210 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c46f6e24-rv2dx"] Oct 30 18:37:57.154180 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:57.153284 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" Oct 30 18:37:57.170862 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:57.170732 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftjpb\" (UniqueName: \"kubernetes.io/projected/d9073e4f-b7e9-461b-b42b-4cb370bce4bf-kube-api-access-ftjpb\") pod \"test-curl-7622-99ea9a\" (UID: \"d9073e4f-b7e9-461b-b42b-4cb370bce4bf\") " pod="opendatahub/test-curl-7622-99ea9a" Oct 30 18:37:57.272521 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:57.272428 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftjpb\" (UniqueName: \"kubernetes.io/projected/d9073e4f-b7e9-461b-b42b-4cb370bce4bf-kube-api-access-ftjpb\") pod \"test-curl-7622-99ea9a\" (UID: \"d9073e4f-b7e9-461b-b42b-4cb370bce4bf\") " pod="opendatahub/test-curl-7622-99ea9a" Oct 30 18:37:57.293764 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:57.293729 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftjpb\" (UniqueName: \"kubernetes.io/projected/d9073e4f-b7e9-461b-b42b-4cb370bce4bf-kube-api-access-ftjpb\") pod \"test-curl-7622-99ea9a\" (UID: \"d9073e4f-b7e9-461b-b42b-4cb370bce4bf\") " pod="opendatahub/test-curl-7622-99ea9a" Oct 30 18:37:57.393736 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:57.393710 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7622-99ea9a" Oct 30 18:37:57.402656 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:57.402629 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46f6e24-rv2dx" event={"ID":"b737eec0-6c8e-4bf2-9a62-286bae2d2b57","Type":"ContainerStarted","Data":"2e1be95fe9f94be8efb241cd6b9369dae30c4c47567ac216d2170bbbfe04a47f"} Oct 30 18:37:57.754523 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:57.754481 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7622-99ea9a"] Oct 30 18:37:57.759738 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:37:57.759692 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9073e4f_b7e9_461b_b42b_4cb370bce4bf.slice/crio-0f2ed37b0768b0b2f19df0c140706bcc3d18d8c8a705dc1e25b045a00bd77ce0 WatchSource:0}: Error finding container 0f2ed37b0768b0b2f19df0c140706bcc3d18d8c8a705dc1e25b045a00bd77ce0: Status 404 returned error can't find the container with id 0f2ed37b0768b0b2f19df0c140706bcc3d18d8c8a705dc1e25b045a00bd77ce0 Oct 30 18:37:57.957235 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:57.957187 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 30 18:37:57.957424 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:57.957241 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" podUID="bfbb6b47-1274-4d3d-8845-627898246065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 30 18:37:58.428161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:58.423672 4442 generic.go:356] "Generic (PLEG): container finished" podID="b737eec0-6c8e-4bf2-9a62-286bae2d2b57" containerID="6d63d0f032e217f6f9e3159177cc2e1e205ecad10913b1a284a4b606619aa0ec" exitCode=0 Oct 30 18:37:58.428557 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:58.428527 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46f6e24-rv2dx" event={"ID":"b737eec0-6c8e-4bf2-9a62-286bae2d2b57","Type":"ContainerDied","Data":"6d63d0f032e217f6f9e3159177cc2e1e205ecad10913b1a284a4b606619aa0ec"} Oct 30 18:37:58.437185 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:58.433748 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7622-99ea9a" event={"ID":"d9073e4f-b7e9-461b-b42b-4cb370bce4bf","Type":"ContainerStarted","Data":"485dc325078aa9864fb00964e2b407a186f89f6e258a6465b3951750340ab15a"} Oct 30 18:37:58.437185 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:58.433814 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7622-99ea9a" event={"ID":"d9073e4f-b7e9-461b-b42b-4cb370bce4bf","Type":"ContainerStarted","Data":"0f2ed37b0768b0b2f19df0c140706bcc3d18d8c8a705dc1e25b045a00bd77ce0"} Oct 30 18:37:58.511176 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:58.511031 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7622-99ea9a" podStartSLOduration=2.511016451 podStartE2EDuration="2.511016451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:58.50696028 +0000 UTC m=+2375.597022643" watchObservedRunningTime="2026-10-30 18:37:58.511016451 +0000 UTC m=+2375.601078789" Oct 30 18:37:59.129064 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:59.128709 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" Oct 30 18:37:59.130998 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:59.130971 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" Oct 30 18:37:59.491650 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:59.491616 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7622-99ea9a"] Oct 30 18:37:59.501268 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:59.501239 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7622-99ea9a"] Oct 30 18:37:59.627747 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:59.627705 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46f6e24-rv2dx" Oct 30 18:37:59.712853 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:59.712775 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b737eec0-6c8e-4bf2-9a62-286bae2d2b57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b737eec0-6c8e-4bf2-9a62-286bae2d2b57-maas-api-xks-ca\") pod \"b737eec0-6c8e-4bf2-9a62-286bae2d2b57\" (UID: \"b737eec0-6c8e-4bf2-9a62-286bae2d2b57\") " Oct 30 18:37:59.712853 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:59.712839 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b737eec0-6c8e-4bf2-9a62-286bae2d2b57-kube-api-access-b2lf6\" (UniqueName: \"kubernetes.io/projected/b737eec0-6c8e-4bf2-9a62-286bae2d2b57-kube-api-access-b2lf6\") pod \"b737eec0-6c8e-4bf2-9a62-286bae2d2b57\" (UID: \"b737eec0-6c8e-4bf2-9a62-286bae2d2b57\") " Oct 30 18:37:59.713155 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:59.712874 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b737eec0-6c8e-4bf2-9a62-286bae2d2b57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b737eec0-6c8e-4bf2-9a62-286bae2d2b57-maas-api-openshift-ca\") pod \"b737eec0-6c8e-4bf2-9a62-286bae2d2b57\" (UID: \"b737eec0-6c8e-4bf2-9a62-286bae2d2b57\") " Oct 30 18:37:59.713491 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:59.713398 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b737eec0-6c8e-4bf2-9a62-286bae2d2b57-maas-api-openshift-ca" pod "b737eec0-6c8e-4bf2-9a62-286bae2d2b57" (UID: "b737eec0-6c8e-4bf2-9a62-286bae2d2b57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:59.713780 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:59.713758 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b737eec0-6c8e-4bf2-9a62-286bae2d2b57-maas-api-openshift-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:59.715431 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:59.715406 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b737eec0-6c8e-4bf2-9a62-286bae2d2b57-maas-api-xks-ca" pod "b737eec0-6c8e-4bf2-9a62-286bae2d2b57" (UID: "b737eec0-6c8e-4bf2-9a62-286bae2d2b57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:59.715855 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:59.715838 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b737eec0-6c8e-4bf2-9a62-286bae2d2b57-kube-api-access-b2lf6" pod "b737eec0-6c8e-4bf2-9a62-286bae2d2b57" (UID: "b737eec0-6c8e-4bf2-9a62-286bae2d2b57"). InnerVolumeSpecName "kube-api-access-b2lf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:59.751214 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:59.751182 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-434dd2db-xz4ll"] Oct 30 18:37:59.751980 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:59.751958 4442 cpu_manager.go:414] "containerMap: removing container" podUID="d9073e4f-b7e9-461b-b42b-4cb370bce4bf" containerName="test-curl-7622-99ea9a" Oct 30 18:37:59.751980 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:59.751976 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9073e4f-b7e9-461b-b42b-4cb370bce4bf" containerName="test-curl-7622-99ea9a" Oct 30 18:37:59.752200 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:59.751999 4442 cpu_manager.go:414] "containerMap: removing container" podUID="b737eec0-6c8e-4bf2-9a62-286bae2d2b57" containerName="revoke-keys" Oct 30 18:37:59.752200 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:59.752009 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b737eec0-6c8e-4bf2-9a62-286bae2d2b57" containerName="revoke-keys" Oct 30 18:37:59.752285 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:59.752220 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9073e4f-b7e9-461b-b42b-4cb370bce4bf" containerName="test-curl-7622-99ea9a" Oct 30 18:37:59.752285 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:59.752241 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="b737eec0-6c8e-4bf2-9a62-286bae2d2b57" containerName="revoke-keys" Oct 30 18:37:59.755010 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:59.754990 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-434dd2db-xz4ll" Oct 30 18:37:59.778351 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:59.778320 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-434dd2db-xz4ll"] Oct 30 18:37:59.815613 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:59.815571 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d35f52d-1113-4f7a-8ac6-036152da4093-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-434dd2db-xz4ll\" (UID: \"5d35f52d-1113-4f7a-8ac6-036152da4093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-434dd2db-xz4ll" Oct 30 18:37:59.815813 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:59.815662 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d35f52d-1113-4f7a-8ac6-036152da4093-maas-api-xks-ca\") pod \"maas-api-revoke-sub-434dd2db-xz4ll\" (UID: \"5d35f52d-1113-4f7a-8ac6-036152da4093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-434dd2db-xz4ll" Oct 30 18:37:59.815813 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:59.815775 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wnh4\" (UniqueName: \"kubernetes.io/projected/5d35f52d-1113-4f7a-8ac6-036152da4093-kube-api-access-5wnh4\") pod \"maas-api-revoke-sub-434dd2db-xz4ll\" (UID: \"5d35f52d-1113-4f7a-8ac6-036152da4093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-434dd2db-xz4ll" Oct 30 18:37:59.815962 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:59.815944 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b737eec0-6c8e-4bf2-9a62-286bae2d2b57-maas-api-xks-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:59.816023 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:59.815967 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2lf6\" (UniqueName: \"kubernetes.io/projected/b737eec0-6c8e-4bf2-9a62-286bae2d2b57-kube-api-access-b2lf6\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:37:59.916799 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:59.916772 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d35f52d-1113-4f7a-8ac6-036152da4093-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-434dd2db-xz4ll\" (UID: \"5d35f52d-1113-4f7a-8ac6-036152da4093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-434dd2db-xz4ll" Oct 30 18:37:59.916979 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:59.916821 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d35f52d-1113-4f7a-8ac6-036152da4093-maas-api-xks-ca\") pod \"maas-api-revoke-sub-434dd2db-xz4ll\" (UID: \"5d35f52d-1113-4f7a-8ac6-036152da4093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-434dd2db-xz4ll" Oct 30 18:37:59.916979 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:59.916895 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wnh4\" (UniqueName: \"kubernetes.io/projected/5d35f52d-1113-4f7a-8ac6-036152da4093-kube-api-access-5wnh4\") pod \"maas-api-revoke-sub-434dd2db-xz4ll\" (UID: \"5d35f52d-1113-4f7a-8ac6-036152da4093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-434dd2db-xz4ll" Oct 30 18:37:59.917743 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:59.917712 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d35f52d-1113-4f7a-8ac6-036152da4093-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-434dd2db-xz4ll\" (UID: \"5d35f52d-1113-4f7a-8ac6-036152da4093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-434dd2db-xz4ll" Oct 30 18:37:59.919680 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:59.919658 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d35f52d-1113-4f7a-8ac6-036152da4093-maas-api-xks-ca\") pod \"maas-api-revoke-sub-434dd2db-xz4ll\" (UID: \"5d35f52d-1113-4f7a-8ac6-036152da4093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-434dd2db-xz4ll" Oct 30 18:37:59.945157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:37:59.940976 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wnh4\" (UniqueName: \"kubernetes.io/projected/5d35f52d-1113-4f7a-8ac6-036152da4093-kube-api-access-5wnh4\") pod \"maas-api-revoke-sub-434dd2db-xz4ll\" (UID: \"5d35f52d-1113-4f7a-8ac6-036152da4093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-434dd2db-xz4ll" Oct 30 18:38:00.077477 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:00.077402 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-434dd2db-xz4ll" Oct 30 18:38:00.388456 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:38:00.388425 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d35f52d_1113_4f7a_8ac6_036152da4093.slice/crio-4843ad2a4617959f67fedafd57bc6c749234cc87497c58c7ebd5c37b7f6dfee1 WatchSource:0}: Error finding container 4843ad2a4617959f67fedafd57bc6c749234cc87497c58c7ebd5c37b7f6dfee1: Status 404 returned error can't find the container with id 4843ad2a4617959f67fedafd57bc6c749234cc87497c58c7ebd5c37b7f6dfee1 Oct 30 18:38:00.390625 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:00.390595 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-434dd2db-xz4ll"] Oct 30 18:38:00.452479 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:00.452450 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46f6e24-rv2dx" Oct 30 18:38:00.452479 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:00.452466 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46f6e24-rv2dx" event={"ID":"b737eec0-6c8e-4bf2-9a62-286bae2d2b57","Type":"ContainerDied","Data":"2e1be95fe9f94be8efb241cd6b9369dae30c4c47567ac216d2170bbbfe04a47f"} Oct 30 18:38:00.452749 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:00.452516 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e1be95fe9f94be8efb241cd6b9369dae30c4c47567ac216d2170bbbfe04a47f" Oct 30 18:38:00.454186 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:00.454162 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-434dd2db-xz4ll" event={"ID":"5d35f52d-1113-4f7a-8ac6-036152da4093","Type":"ContainerStarted","Data":"4843ad2a4617959f67fedafd57bc6c749234cc87497c58c7ebd5c37b7f6dfee1"} Oct 30 18:38:00.454395 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:00.454361 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7622-99ea9a" podUID="d9073e4f-b7e9-461b-b42b-4cb370bce4bf" containerName="test-curl-7622-99ea9a" containerID="cri-o://485dc325078aa9864fb00964e2b407a186f89f6e258a6465b3951750340ab15a" gracePeriod=2 Oct 30 18:38:00.467533 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:38:00.467510 4442 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7622-99ea9a\" is forbidden: User \"system:node:ip-10-0-138-176.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-176.ec2.internal' and this object" podUID="d9073e4f-b7e9-461b-b42b-4cb370bce4bf" pod="opendatahub/test-curl-7622-99ea9a" Oct 30 18:38:00.498375 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:38:00.498337 4442 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7622-99ea9a\" is forbidden: User \"system:node:ip-10-0-138-176.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-176.ec2.internal' and this object" podUID="d9073e4f-b7e9-461b-b42b-4cb370bce4bf" pod="opendatahub/test-curl-7622-99ea9a" Oct 30 18:38:00.928387 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:00.928357 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp"] Oct 30 18:38:00.932779 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:00.932755 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" Oct 30 18:38:00.963587 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:00.963553 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp"] Oct 30 18:38:01.033944 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:01.033858 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/425bc56e-e6f2-400d-a6dc-a277201acb8f-istio-data\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp\" (UID: \"425bc56e-e6f2-400d-a6dc-a277201acb8f\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" Oct 30 18:38:01.034084 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:01.033965 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n85w5\" (UniqueName: \"kubernetes.io/projected/425bc56e-e6f2-400d-a6dc-a277201acb8f-kube-api-access-n85w5\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp\" (UID: \"425bc56e-e6f2-400d-a6dc-a277201acb8f\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" Oct 30 18:38:01.034084 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:01.034007 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/425bc56e-e6f2-400d-a6dc-a277201acb8f-istiod-ca-cert\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp\" (UID: \"425bc56e-e6f2-400d-a6dc-a277201acb8f\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" Oct 30 18:38:01.034230 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:01.034101 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/425bc56e-e6f2-400d-a6dc-a277201acb8f-istio-podinfo\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp\" (UID: \"425bc56e-e6f2-400d-a6dc-a277201acb8f\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" Oct 30 18:38:01.034230 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:01.034221 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/425bc56e-e6f2-400d-a6dc-a277201acb8f-workload-certs\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp\" (UID: \"425bc56e-e6f2-400d-a6dc-a277201acb8f\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" Oct 30 18:38:01.034304 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:01.034257 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/425bc56e-e6f2-400d-a6dc-a277201acb8f-credential-socket\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp\" (UID: \"425bc56e-e6f2-400d-a6dc-a277201acb8f\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" Oct 30 18:38:01.034304 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:01.034274 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/425bc56e-e6f2-400d-a6dc-a277201acb8f-istio-envoy\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp\" (UID: \"425bc56e-e6f2-400d-a6dc-a277201acb8f\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" Oct 30 18:38:01.034406 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:01.034380 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/425bc56e-e6f2-400d-a6dc-a277201acb8f-workload-socket\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp\" (UID: \"425bc56e-e6f2-400d-a6dc-a277201acb8f\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" Oct 30 18:38:01.034450 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:01.034432 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/425bc56e-e6f2-400d-a6dc-a277201acb8f-istio-token\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp\" (UID: \"425bc56e-e6f2-400d-a6dc-a277201acb8f\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" Oct 30 18:38:01.059652 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:38:01.059616 4442 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7622-99ea9a\" is forbidden: User \"system:node:ip-10-0-138-176.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-176.ec2.internal' and this object" podUID="d9073e4f-b7e9-461b-b42b-4cb370bce4bf" pod="opendatahub/test-curl-7622-99ea9a" Oct 30 18:38:01.072854 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:01.072835 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-c095fa-openshift-default-dockercfg-kvfnl\"" Oct 30 18:38:01.136239 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:01.136208 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/425bc56e-e6f2-400d-a6dc-a277201acb8f-credential-socket\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp\" (UID: \"425bc56e-e6f2-400d-a6dc-a277201acb8f\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" Oct 30 18:38:01.136239 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:01.136240 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/425bc56e-e6f2-400d-a6dc-a277201acb8f-istio-envoy\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp\" (UID: \"425bc56e-e6f2-400d-a6dc-a277201acb8f\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" Oct 30 18:38:01.136449 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:01.136332 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/425bc56e-e6f2-400d-a6dc-a277201acb8f-workload-socket\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp\" (UID: \"425bc56e-e6f2-400d-a6dc-a277201acb8f\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" Oct 30 18:38:01.136449 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:01.136391 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/425bc56e-e6f2-400d-a6dc-a277201acb8f-istio-token\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp\" (UID: \"425bc56e-e6f2-400d-a6dc-a277201acb8f\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" Oct 30 18:38:01.136538 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:01.136474 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/425bc56e-e6f2-400d-a6dc-a277201acb8f-istio-data\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp\" (UID: \"425bc56e-e6f2-400d-a6dc-a277201acb8f\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" Oct 30 18:38:01.136597 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:01.136566 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n85w5\" (UniqueName: \"kubernetes.io/projected/425bc56e-e6f2-400d-a6dc-a277201acb8f-kube-api-access-n85w5\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp\" (UID: \"425bc56e-e6f2-400d-a6dc-a277201acb8f\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" Oct 30 18:38:01.136658 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:01.136613 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/425bc56e-e6f2-400d-a6dc-a277201acb8f-istiod-ca-cert\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp\" (UID: \"425bc56e-e6f2-400d-a6dc-a277201acb8f\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" Oct 30 18:38:01.136658 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:01.136649 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/425bc56e-e6f2-400d-a6dc-a277201acb8f-credential-socket\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp\" (UID: \"425bc56e-e6f2-400d-a6dc-a277201acb8f\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" Oct 30 18:38:01.136796 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:01.136696 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/425bc56e-e6f2-400d-a6dc-a277201acb8f-istio-podinfo\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp\" (UID: \"425bc56e-e6f2-400d-a6dc-a277201acb8f\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" Oct 30 18:38:01.136861 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:01.136791 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/425bc56e-e6f2-400d-a6dc-a277201acb8f-workload-socket\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp\" (UID: \"425bc56e-e6f2-400d-a6dc-a277201acb8f\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" Oct 30 18:38:01.136861 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:01.136815 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/425bc56e-e6f2-400d-a6dc-a277201acb8f-workload-certs\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp\" (UID: \"425bc56e-e6f2-400d-a6dc-a277201acb8f\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" Oct 30 18:38:01.136982 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:01.136855 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/425bc56e-e6f2-400d-a6dc-a277201acb8f-istio-data\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp\" (UID: \"425bc56e-e6f2-400d-a6dc-a277201acb8f\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" Oct 30 18:38:01.137267 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:01.137244 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/425bc56e-e6f2-400d-a6dc-a277201acb8f-istiod-ca-cert\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp\" (UID: \"425bc56e-e6f2-400d-a6dc-a277201acb8f\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" Oct 30 18:38:01.137383 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:01.137344 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/425bc56e-e6f2-400d-a6dc-a277201acb8f-workload-certs\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp\" (UID: \"425bc56e-e6f2-400d-a6dc-a277201acb8f\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" Oct 30 18:38:01.138719 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:01.138700 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/425bc56e-e6f2-400d-a6dc-a277201acb8f-istio-envoy\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp\" (UID: \"425bc56e-e6f2-400d-a6dc-a277201acb8f\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" Oct 30 18:38:01.139215 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:01.139196 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/425bc56e-e6f2-400d-a6dc-a277201acb8f-istio-podinfo\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp\" (UID: \"425bc56e-e6f2-400d-a6dc-a277201acb8f\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" Oct 30 18:38:01.179142 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:01.179111 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/425bc56e-e6f2-400d-a6dc-a277201acb8f-istio-token\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp\" (UID: \"425bc56e-e6f2-400d-a6dc-a277201acb8f\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" Oct 30 18:38:01.190851 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:01.190824 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n85w5\" (UniqueName: \"kubernetes.io/projected/425bc56e-e6f2-400d-a6dc-a277201acb8f-kube-api-access-n85w5\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp\" (UID: \"425bc56e-e6f2-400d-a6dc-a277201acb8f\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" Oct 30 18:38:01.247798 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:01.247767 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" Oct 30 18:38:01.536044 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:01.533245 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-434dd2db-xz4ll" event={"ID":"5d35f52d-1113-4f7a-8ac6-036152da4093","Type":"ContainerStarted","Data":"d620e9087948f13c8f520b24a7c89ec665c6f08a91745ff3381b9afc2f8a1809"} Oct 30 18:38:01.592532 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:01.592428 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-434dd2db-xz4ll" podStartSLOduration=2.592406392 podStartE2EDuration="2.592406392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:01.586028015 +0000 UTC m=+2378.676090354" watchObservedRunningTime="2026-10-30 18:38:01.592406392 +0000 UTC m=+2378.682468732" Oct 30 18:38:01.777066 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:01.777041 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp"] Oct 30 18:38:01.778763 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:38:01.778730 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod425bc56e_e6f2_400d_a6dc_a277201acb8f.slice/crio-e5cc57b12a6798a437e0d328c53f352deb7f2b933ba0c0a788a938e4606bcc2f WatchSource:0}: Error finding container e5cc57b12a6798a437e0d328c53f352deb7f2b933ba0c0a788a938e4606bcc2f: Status 404 returned error can't find the container with id e5cc57b12a6798a437e0d328c53f352deb7f2b933ba0c0a788a938e4606bcc2f Oct 30 18:38:01.782320 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:01.782283 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:38:01.782410 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:01.782374 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:38:01.782451 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:01.782411 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:38:02.545893 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:02.545854 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" event={"ID":"425bc56e-e6f2-400d-a6dc-a277201acb8f","Type":"ContainerStarted","Data":"529b13f2ac933b33601c939e4f23541c0eb541e41c6d2fa2fefc6ff9bcc9924f"} Oct 30 18:38:02.546308 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:02.545935 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" event={"ID":"425bc56e-e6f2-400d-a6dc-a277201acb8f","Type":"ContainerStarted","Data":"e5cc57b12a6798a437e0d328c53f352deb7f2b933ba0c0a788a938e4606bcc2f"} Oct 30 18:38:02.547980 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:02.547946 4442 generic.go:356] "Generic (PLEG): container finished" podID="d9073e4f-b7e9-461b-b42b-4cb370bce4bf" containerID="485dc325078aa9864fb00964e2b407a186f89f6e258a6465b3951750340ab15a" exitCode=137 Oct 30 18:38:02.550888 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:02.550863 4442 generic.go:356] "Generic (PLEG): container finished" podID="5d35f52d-1113-4f7a-8ac6-036152da4093" containerID="d620e9087948f13c8f520b24a7c89ec665c6f08a91745ff3381b9afc2f8a1809" exitCode=0 Oct 30 18:38:02.550992 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:02.550930 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-434dd2db-xz4ll" event={"ID":"5d35f52d-1113-4f7a-8ac6-036152da4093","Type":"ContainerDied","Data":"d620e9087948f13c8f520b24a7c89ec665c6f08a91745ff3381b9afc2f8a1809"} Oct 30 18:38:02.666169 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:02.666089 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" podStartSLOduration=2.666073485 podStartE2EDuration="2.666073485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:02.617233627 +0000 UTC m=+2379.707295967" watchObservedRunningTime="2026-10-30 18:38:02.666073485 +0000 UTC m=+2379.756135832" Oct 30 18:38:02.686597 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:02.686566 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7622-99ea9a" Oct 30 18:38:02.784222 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:02.783305 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9073e4f-b7e9-461b-b42b-4cb370bce4bf-kube-api-access-ftjpb\" (UniqueName: \"kubernetes.io/projected/d9073e4f-b7e9-461b-b42b-4cb370bce4bf-kube-api-access-ftjpb\") pod \"d9073e4f-b7e9-461b-b42b-4cb370bce4bf\" (UID: \"d9073e4f-b7e9-461b-b42b-4cb370bce4bf\") " Oct 30 18:38:02.804147 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:02.801492 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9073e4f-b7e9-461b-b42b-4cb370bce4bf-kube-api-access-ftjpb" pod "d9073e4f-b7e9-461b-b42b-4cb370bce4bf" (UID: "d9073e4f-b7e9-461b-b42b-4cb370bce4bf"). InnerVolumeSpecName "kube-api-access-ftjpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:02.886308 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:02.886273 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftjpb\" (UniqueName: \"kubernetes.io/projected/d9073e4f-b7e9-461b-b42b-4cb370bce4bf-kube-api-access-ftjpb\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:38:03.034542 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:03.034466 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2107d4-7588cd98bc-nt76m"] Oct 30 18:38:03.037802 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:03.037784 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2107d4-7588cd98bc-nt76m" Oct 30 18:38:03.055203 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:03.055159 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-2107d4\"" Oct 30 18:38:03.055359 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:03.055341 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-2107d4-dockercfg-jmmv4\"" Oct 30 18:38:03.069814 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:03.069782 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9073e4f-b7e9-461b-b42b-4cb370bce4bf" path="/var/lib/kubelet/pods/d9073e4f-b7e9-461b-b42b-4cb370bce4bf/volumes" Oct 30 18:38:03.070545 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:03.070525 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2107d4-7588cd98bc-nt76m"] Oct 30 18:38:03.203157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:03.192401 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d147fcbd-d2ce-4a51-a8f1-531db7bcf8fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-2107d4-7588cd98bc-nt76m\" (UID: \"d147fcbd-d2ce-4a51-a8f1-531db7bcf8fc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2107d4-7588cd98bc-nt76m" Oct 30 18:38:03.203157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:03.192913 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjvrz\" (UniqueName: \"kubernetes.io/projected/d147fcbd-d2ce-4a51-a8f1-531db7bcf8fc-kube-api-access-vjvrz\") pod \"payload-pre-processing-e2e-extmod-a-2107d4-7588cd98bc-nt76m\" (UID: \"d147fcbd-d2ce-4a51-a8f1-531db7bcf8fc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2107d4-7588cd98bc-nt76m" Oct 30 18:38:03.262441 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:03.261343 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" Oct 30 18:38:03.299568 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:03.299485 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" Oct 30 18:38:03.300538 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:03.300513 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d147fcbd-d2ce-4a51-a8f1-531db7bcf8fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-2107d4-7588cd98bc-nt76m\" (UID: \"d147fcbd-d2ce-4a51-a8f1-531db7bcf8fc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2107d4-7588cd98bc-nt76m" Oct 30 18:38:03.300815 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:03.300793 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjvrz\" (UniqueName: \"kubernetes.io/projected/d147fcbd-d2ce-4a51-a8f1-531db7bcf8fc-kube-api-access-vjvrz\") pod \"payload-pre-processing-e2e-extmod-a-2107d4-7588cd98bc-nt76m\" (UID: \"d147fcbd-d2ce-4a51-a8f1-531db7bcf8fc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2107d4-7588cd98bc-nt76m" Oct 30 18:38:03.302633 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:03.302607 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d147fcbd-d2ce-4a51-a8f1-531db7bcf8fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-2107d4-7588cd98bc-nt76m\" (UID: \"d147fcbd-d2ce-4a51-a8f1-531db7bcf8fc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2107d4-7588cd98bc-nt76m" Oct 30 18:38:03.308926 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:03.308901 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2107d4-7bf5b8d588-hz7nk"] Oct 30 18:38:03.314231 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:03.314198 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2107d4-7bf5b8d588-hz7nk" Oct 30 18:38:03.319251 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:03.319214 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2107d4-7bf5b8d588-hz7nk"] Oct 30 18:38:03.328180 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:03.324221 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjvrz\" (UniqueName: \"kubernetes.io/projected/d147fcbd-d2ce-4a51-a8f1-531db7bcf8fc-kube-api-access-vjvrz\") pod \"payload-pre-processing-e2e-extmod-a-2107d4-7588cd98bc-nt76m\" (UID: \"d147fcbd-d2ce-4a51-a8f1-531db7bcf8fc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2107d4-7588cd98bc-nt76m" Oct 30 18:38:03.352673 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:03.352643 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2107d4-7588cd98bc-nt76m" Oct 30 18:38:03.402582 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:03.402116 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/017e9c94-87b9-4143-bd73-311bb6edbed9-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-2107d4-7bf5b8d588-hz7nk\" (UID: \"017e9c94-87b9-4143-bd73-311bb6edbed9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2107d4-7bf5b8d588-hz7nk" Oct 30 18:38:03.402582 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:03.402274 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzc5p\" (UniqueName: \"kubernetes.io/projected/017e9c94-87b9-4143-bd73-311bb6edbed9-kube-api-access-bzc5p\") pod \"payload-processing-e2e-extmod-a-2107d4-7bf5b8d588-hz7nk\" (UID: \"017e9c94-87b9-4143-bd73-311bb6edbed9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2107d4-7bf5b8d588-hz7nk" Oct 30 18:38:03.503823 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:03.503785 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/017e9c94-87b9-4143-bd73-311bb6edbed9-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-2107d4-7bf5b8d588-hz7nk\" (UID: \"017e9c94-87b9-4143-bd73-311bb6edbed9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2107d4-7bf5b8d588-hz7nk" Oct 30 18:38:03.504035 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:03.503879 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzc5p\" (UniqueName: \"kubernetes.io/projected/017e9c94-87b9-4143-bd73-311bb6edbed9-kube-api-access-bzc5p\") pod \"payload-processing-e2e-extmod-a-2107d4-7bf5b8d588-hz7nk\" (UID: \"017e9c94-87b9-4143-bd73-311bb6edbed9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2107d4-7bf5b8d588-hz7nk" Oct 30 18:38:03.504567 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:03.504545 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/017e9c94-87b9-4143-bd73-311bb6edbed9-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-2107d4-7bf5b8d588-hz7nk\" (UID: \"017e9c94-87b9-4143-bd73-311bb6edbed9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2107d4-7bf5b8d588-hz7nk" Oct 30 18:38:03.557693 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:03.557610 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzc5p\" (UniqueName: \"kubernetes.io/projected/017e9c94-87b9-4143-bd73-311bb6edbed9-kube-api-access-bzc5p\") pod \"payload-processing-e2e-extmod-a-2107d4-7bf5b8d588-hz7nk\" (UID: \"017e9c94-87b9-4143-bd73-311bb6edbed9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2107d4-7bf5b8d588-hz7nk" Oct 30 18:38:03.566509 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:03.566481 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7622-99ea9a" Oct 30 18:38:03.566694 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:03.566527 4442 scope.go:122] "RemoveContainer" containerID="485dc325078aa9864fb00964e2b407a186f89f6e258a6465b3951750340ab15a" Oct 30 18:38:03.568189 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:03.567200 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" Oct 30 18:38:03.568513 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:03.568490 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" Oct 30 18:38:03.579585 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:38:03.579553 4442 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7622-99ea9a\" is forbidden: User \"system:node:ip-10-0-138-176.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-176.ec2.internal' and this object" podUID="d9073e4f-b7e9-461b-b42b-4cb370bce4bf" pod="opendatahub/test-curl-7622-99ea9a" Oct 30 18:38:03.592619 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:38:03.592581 4442 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7622-99ea9a\" is forbidden: User \"system:node:ip-10-0-138-176.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-176.ec2.internal' and this object" podUID="d9073e4f-b7e9-461b-b42b-4cb370bce4bf" pod="opendatahub/test-curl-7622-99ea9a" Oct 30 18:38:03.654494 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:03.654235 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2107d4-7bf5b8d588-hz7nk" Oct 30 18:38:03.665201 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:03.664757 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2107d4-7588cd98bc-nt76m"] Oct 30 18:38:03.676289 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:38:03.676247 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd147fcbd_d2ce_4a51_a8f1_531db7bcf8fc.slice/crio-c6bf26dbfaac263e38ee2ceae53091f40135eb106a703154a8e41179c174e37a WatchSource:0}: Error finding container c6bf26dbfaac263e38ee2ceae53091f40135eb106a703154a8e41179c174e37a: Status 404 returned error can't find the container with id c6bf26dbfaac263e38ee2ceae53091f40135eb106a703154a8e41179c174e37a Oct 30 18:38:03.816092 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:03.815936 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-434dd2db-xz4ll" Oct 30 18:38:03.870083 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:38:03.869735 4442 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7622-99ea9a\" is forbidden: User \"system:node:ip-10-0-138-176.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-176.ec2.internal' and this object" podUID="d9073e4f-b7e9-461b-b42b-4cb370bce4bf" pod="opendatahub/test-curl-7622-99ea9a" Oct 30 18:38:03.930791 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:03.930760 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d35f52d-1113-4f7a-8ac6-036152da4093-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d35f52d-1113-4f7a-8ac6-036152da4093-maas-api-openshift-ca\") pod \"5d35f52d-1113-4f7a-8ac6-036152da4093\" (UID: \"5d35f52d-1113-4f7a-8ac6-036152da4093\") " Oct 30 18:38:03.930964 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:03.930933 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d35f52d-1113-4f7a-8ac6-036152da4093-kube-api-access-5wnh4\" (UniqueName: \"kubernetes.io/projected/5d35f52d-1113-4f7a-8ac6-036152da4093-kube-api-access-5wnh4\") pod \"5d35f52d-1113-4f7a-8ac6-036152da4093\" (UID: \"5d35f52d-1113-4f7a-8ac6-036152da4093\") " Oct 30 18:38:03.931051 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:03.931024 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d35f52d-1113-4f7a-8ac6-036152da4093-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d35f52d-1113-4f7a-8ac6-036152da4093-maas-api-xks-ca\") pod \"5d35f52d-1113-4f7a-8ac6-036152da4093\" (UID: \"5d35f52d-1113-4f7a-8ac6-036152da4093\") " Oct 30 18:38:03.931204 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:03.931180 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d35f52d-1113-4f7a-8ac6-036152da4093-maas-api-openshift-ca" pod "5d35f52d-1113-4f7a-8ac6-036152da4093" (UID: "5d35f52d-1113-4f7a-8ac6-036152da4093"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:03.931735 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:03.931641 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d35f52d-1113-4f7a-8ac6-036152da4093-maas-api-openshift-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:38:03.933346 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:03.933315 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d35f52d-1113-4f7a-8ac6-036152da4093-maas-api-xks-ca" pod "5d35f52d-1113-4f7a-8ac6-036152da4093" (UID: "5d35f52d-1113-4f7a-8ac6-036152da4093"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:03.933453 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:03.933391 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d35f52d-1113-4f7a-8ac6-036152da4093-kube-api-access-5wnh4" pod "5d35f52d-1113-4f7a-8ac6-036152da4093" (UID: "5d35f52d-1113-4f7a-8ac6-036152da4093"). InnerVolumeSpecName "kube-api-access-5wnh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:04.033701 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:04.033668 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d35f52d-1113-4f7a-8ac6-036152da4093-maas-api-xks-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:38:04.033701 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:04.033700 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wnh4\" (UniqueName: \"kubernetes.io/projected/5d35f52d-1113-4f7a-8ac6-036152da4093-kube-api-access-5wnh4\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:38:04.040255 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:04.040223 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2107d4-7bf5b8d588-hz7nk"] Oct 30 18:38:04.045047 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:38:04.045023 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod017e9c94_87b9_4143_bd73_311bb6edbed9.slice/crio-9a22a225a7c094903a78c9b13e550146070e46f658cb76b14b72305384cc80fa WatchSource:0}: Error finding container 9a22a225a7c094903a78c9b13e550146070e46f658cb76b14b72305384cc80fa: Status 404 returned error can't find the container with id 9a22a225a7c094903a78c9b13e550146070e46f658cb76b14b72305384cc80fa Oct 30 18:38:04.576952 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:04.576855 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2107d4-7588cd98bc-nt76m" event={"ID":"d147fcbd-d2ce-4a51-a8f1-531db7bcf8fc","Type":"ContainerStarted","Data":"7b49cf6f38e1fa31410e1fcc6ed89a4499a6a0c27b0b4473a619bdfe99821767"} Oct 30 18:38:04.577436 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:04.576954 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2107d4-7588cd98bc-nt76m" event={"ID":"d147fcbd-d2ce-4a51-a8f1-531db7bcf8fc","Type":"ContainerStarted","Data":"c6bf26dbfaac263e38ee2ceae53091f40135eb106a703154a8e41179c174e37a"} Oct 30 18:38:04.577436 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:04.577126 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2107d4-7588cd98bc-nt76m" Oct 30 18:38:04.578760 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:04.578726 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2107d4-7bf5b8d588-hz7nk" event={"ID":"017e9c94-87b9-4143-bd73-311bb6edbed9","Type":"ContainerStarted","Data":"8d4456cf1413d30fb074e5d95ebc5e0dad05102ad7f3e9eae250d3ffadcb86c0"} Oct 30 18:38:04.578886 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:04.578776 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2107d4-7bf5b8d588-hz7nk" event={"ID":"017e9c94-87b9-4143-bd73-311bb6edbed9","Type":"ContainerStarted","Data":"9a22a225a7c094903a78c9b13e550146070e46f658cb76b14b72305384cc80fa"} Oct 30 18:38:04.578886 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:04.578801 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-2107d4-7bf5b8d588-hz7nk" Oct 30 18:38:04.581318 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:04.581283 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-434dd2db-xz4ll" event={"ID":"5d35f52d-1113-4f7a-8ac6-036152da4093","Type":"ContainerDied","Data":"4843ad2a4617959f67fedafd57bc6c749234cc87497c58c7ebd5c37b7f6dfee1"} Oct 30 18:38:04.581420 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:04.581324 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-434dd2db-xz4ll" Oct 30 18:38:04.581420 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:04.581342 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4843ad2a4617959f67fedafd57bc6c749234cc87497c58c7ebd5c37b7f6dfee1" Oct 30 18:38:04.590354 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:38:04.590327 4442 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7622-99ea9a\" is forbidden: User \"system:node:ip-10-0-138-176.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-176.ec2.internal' and this object" podUID="d9073e4f-b7e9-461b-b42b-4cb370bce4bf" pod="opendatahub/test-curl-7622-99ea9a" Oct 30 18:38:04.645535 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:04.645467 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2107d4-7588cd98bc-nt76m" podStartSLOduration=1.645448092 podStartE2EDuration="1.645448092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:04.642869912 +0000 UTC m=+2381.732932255" watchObservedRunningTime="2026-10-30 18:38:04.645448092 +0000 UTC m=+2381.735510431" Oct 30 18:38:04.693020 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:04.692971 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-2107d4-7bf5b8d588-hz7nk" podStartSLOduration=1.692955923 podStartE2EDuration="1.692955923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:04.688324859 +0000 UTC m=+2381.778387198" watchObservedRunningTime="2026-10-30 18:38:04.692955923 +0000 UTC m=+2381.783018262" Oct 30 18:38:04.716087 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:38:04.716046 4442 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7622-99ea9a\" is forbidden: User \"system:node:ip-10-0-138-176.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-176.ec2.internal' and this object" podUID="d9073e4f-b7e9-461b-b42b-4cb370bce4bf" pod="opendatahub/test-curl-7622-99ea9a" Oct 30 18:38:07.120686 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:07.120656 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c095fa-5d479bd6f9-rgj6w"] Oct 30 18:38:07.121313 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:07.121298 4442 cpu_manager.go:414] "containerMap: removing container" podUID="5d35f52d-1113-4f7a-8ac6-036152da4093" containerName="revoke-keys" Oct 30 18:38:07.121313 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:07.121313 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d35f52d-1113-4f7a-8ac6-036152da4093" containerName="revoke-keys" Oct 30 18:38:07.121449 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:07.121440 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d35f52d-1113-4f7a-8ac6-036152da4093" containerName="revoke-keys" Oct 30 18:38:07.124035 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:07.124020 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c095fa-5d479bd6f9-rgj6w" Oct 30 18:38:07.139552 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:07.139519 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-c095fa\"" Oct 30 18:38:07.139670 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:07.139650 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-c095fa-dockercfg-ftjxd\"" Oct 30 18:38:07.157631 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:07.157603 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c095fa-5d479bd6f9-rgj6w"] Oct 30 18:38:07.178469 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:07.178448 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c095fa-7ff45cdc78-lvpxv"] Oct 30 18:38:07.181807 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:07.181787 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c095fa-7ff45cdc78-lvpxv" Oct 30 18:38:07.306155 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:07.300927 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/979a6818-4b68-4229-86e5-c11fdd1586bd-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-c095fa-7ff45cdc78-lvpxv\" (UID: \"979a6818-4b68-4229-86e5-c11fdd1586bd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c095fa-7ff45cdc78-lvpxv" Oct 30 18:38:07.306155 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:07.301065 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7l8k\" (UniqueName: \"kubernetes.io/projected/979a6818-4b68-4229-86e5-c11fdd1586bd-kube-api-access-w7l8k\") pod \"payload-processing-e2e-shared-b-w0-c095fa-7ff45cdc78-lvpxv\" (UID: \"979a6818-4b68-4229-86e5-c11fdd1586bd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c095fa-7ff45cdc78-lvpxv" Oct 30 18:38:07.306155 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:07.301181 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79fng\" (UniqueName: \"kubernetes.io/projected/b9e66197-f5b9-4ea3-8fa6-d52647e45327-kube-api-access-79fng\") pod \"payload-pre-processing-e2e-shared-b-w0-c095fa-5d479bd6f9-rgj6w\" (UID: \"b9e66197-f5b9-4ea3-8fa6-d52647e45327\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c095fa-5d479bd6f9-rgj6w" Oct 30 18:38:07.306155 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:07.301284 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9e66197-f5b9-4ea3-8fa6-d52647e45327-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-c095fa-5d479bd6f9-rgj6w\" (UID: \"b9e66197-f5b9-4ea3-8fa6-d52647e45327\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c095fa-5d479bd6f9-rgj6w" Oct 30 18:38:07.381300 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:07.380191 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c095fa-7ff45cdc78-lvpxv"] Oct 30 18:38:07.406161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:07.403044 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/979a6818-4b68-4229-86e5-c11fdd1586bd-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-c095fa-7ff45cdc78-lvpxv\" (UID: \"979a6818-4b68-4229-86e5-c11fdd1586bd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c095fa-7ff45cdc78-lvpxv" Oct 30 18:38:07.406161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:07.403261 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7l8k\" (UniqueName: \"kubernetes.io/projected/979a6818-4b68-4229-86e5-c11fdd1586bd-kube-api-access-w7l8k\") pod \"payload-processing-e2e-shared-b-w0-c095fa-7ff45cdc78-lvpxv\" (UID: \"979a6818-4b68-4229-86e5-c11fdd1586bd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c095fa-7ff45cdc78-lvpxv" Oct 30 18:38:07.406161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:07.403398 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79fng\" (UniqueName: \"kubernetes.io/projected/b9e66197-f5b9-4ea3-8fa6-d52647e45327-kube-api-access-79fng\") pod \"payload-pre-processing-e2e-shared-b-w0-c095fa-5d479bd6f9-rgj6w\" (UID: \"b9e66197-f5b9-4ea3-8fa6-d52647e45327\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c095fa-5d479bd6f9-rgj6w" Oct 30 18:38:07.406161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:07.403512 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9e66197-f5b9-4ea3-8fa6-d52647e45327-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-c095fa-5d479bd6f9-rgj6w\" (UID: \"b9e66197-f5b9-4ea3-8fa6-d52647e45327\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c095fa-5d479bd6f9-rgj6w" Oct 30 18:38:07.406161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:07.404104 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/979a6818-4b68-4229-86e5-c11fdd1586bd-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-c095fa-7ff45cdc78-lvpxv\" (UID: \"979a6818-4b68-4229-86e5-c11fdd1586bd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c095fa-7ff45cdc78-lvpxv" Oct 30 18:38:07.406161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:07.404589 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9e66197-f5b9-4ea3-8fa6-d52647e45327-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-c095fa-5d479bd6f9-rgj6w\" (UID: \"b9e66197-f5b9-4ea3-8fa6-d52647e45327\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c095fa-5d479bd6f9-rgj6w" Oct 30 18:38:07.424433 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:07.424396 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7l8k\" (UniqueName: \"kubernetes.io/projected/979a6818-4b68-4229-86e5-c11fdd1586bd-kube-api-access-w7l8k\") pod \"payload-processing-e2e-shared-b-w0-c095fa-7ff45cdc78-lvpxv\" (UID: \"979a6818-4b68-4229-86e5-c11fdd1586bd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c095fa-7ff45cdc78-lvpxv" Oct 30 18:38:07.425420 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:07.425399 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79fng\" (UniqueName: \"kubernetes.io/projected/b9e66197-f5b9-4ea3-8fa6-d52647e45327-kube-api-access-79fng\") pod \"payload-pre-processing-e2e-shared-b-w0-c095fa-5d479bd6f9-rgj6w\" (UID: \"b9e66197-f5b9-4ea3-8fa6-d52647e45327\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c095fa-5d479bd6f9-rgj6w" Oct 30 18:38:07.437373 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:07.437343 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c095fa-5d479bd6f9-rgj6w" Oct 30 18:38:07.647777 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:07.647681 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c095fa-7ff45cdc78-lvpxv" Oct 30 18:38:07.744636 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:38:07.744594 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9e66197_f5b9_4ea3_8fa6_d52647e45327.slice/crio-1c0744bdeea36e128c76abd3c346e7177f14d4e00da6e69218243d1435017c99 WatchSource:0}: Error finding container 1c0744bdeea36e128c76abd3c346e7177f14d4e00da6e69218243d1435017c99: Status 404 returned error can't find the container with id 1c0744bdeea36e128c76abd3c346e7177f14d4e00da6e69218243d1435017c99 Oct 30 18:38:07.749529 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:07.749125 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c095fa-5d479bd6f9-rgj6w"] Oct 30 18:38:07.976466 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:38:07.976428 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod979a6818_4b68_4229_86e5_c11fdd1586bd.slice/crio-0db5ed6135a93b7c247f96ee4126ec30453e3874f80cb595a2da2e5d51767f68 WatchSource:0}: Error finding container 0db5ed6135a93b7c247f96ee4126ec30453e3874f80cb595a2da2e5d51767f68: Status 404 returned error can't find the container with id 0db5ed6135a93b7c247f96ee4126ec30453e3874f80cb595a2da2e5d51767f68 Oct 30 18:38:07.983237 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:07.983210 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c095fa-7ff45cdc78-lvpxv"] Oct 30 18:38:08.619060 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:08.619027 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c095fa-7ff45cdc78-lvpxv" event={"ID":"979a6818-4b68-4229-86e5-c11fdd1586bd","Type":"ContainerStarted","Data":"4e28fba4ba4a8187c95d2d0115df7b6eef83539475cce4a4d812515bd5a4f43f"} Oct 30 18:38:08.619541 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:08.619089 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c095fa-7ff45cdc78-lvpxv" event={"ID":"979a6818-4b68-4229-86e5-c11fdd1586bd","Type":"ContainerStarted","Data":"0db5ed6135a93b7c247f96ee4126ec30453e3874f80cb595a2da2e5d51767f68"} Oct 30 18:38:08.619541 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:08.619118 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c095fa-7ff45cdc78-lvpxv" Oct 30 18:38:08.621150 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:08.621111 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c095fa-5d479bd6f9-rgj6w" event={"ID":"b9e66197-f5b9-4ea3-8fa6-d52647e45327","Type":"ContainerStarted","Data":"b8439f6f2605eed5e5db2887dfbddd74528a4337fe7ac304ea75cf9b7bea2e30"} Oct 30 18:38:08.621270 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:08.621164 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c095fa-5d479bd6f9-rgj6w" event={"ID":"b9e66197-f5b9-4ea3-8fa6-d52647e45327","Type":"ContainerStarted","Data":"1c0744bdeea36e128c76abd3c346e7177f14d4e00da6e69218243d1435017c99"} Oct 30 18:38:08.621270 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:08.621263 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c095fa-5d479bd6f9-rgj6w" Oct 30 18:38:08.679277 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:08.679230 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c095fa-7ff45cdc78-lvpxv" podStartSLOduration=1.6792138140000001 podStartE2EDuration="1.679213814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:08.676472469 +0000 UTC m=+2385.766534810" watchObservedRunningTime="2026-10-30 18:38:08.679213814 +0000 UTC m=+2385.769276153" Oct 30 18:38:08.726397 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:08.726347 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c095fa-5d479bd6f9-rgj6w" podStartSLOduration=1.726333256 podStartE2EDuration="1.726333256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:08.720228994 +0000 UTC m=+2385.810291337" watchObservedRunningTime="2026-10-30 18:38:08.726333256 +0000 UTC m=+2385.816395597" Oct 30 18:38:15.283545 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:15.283511 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh"] Oct 30 18:38:15.288411 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:15.288386 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh" Oct 30 18:38:15.307568 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:15.307543 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-4c75d7-openshift-default-dockercfg-hm865\"" Oct 30 18:38:15.361820 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:15.361027 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh"] Oct 30 18:38:15.462157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:15.450298 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86ac6565-97e8-466f-9ba8-f82843763246-istio-envoy\") pod \"e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh\" (UID: \"86ac6565-97e8-466f-9ba8-f82843763246\") " pod="openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh" Oct 30 18:38:15.462157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:15.450387 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86ac6565-97e8-466f-9ba8-f82843763246-istio-data\") pod \"e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh\" (UID: \"86ac6565-97e8-466f-9ba8-f82843763246\") " pod="openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh" Oct 30 18:38:15.462157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:15.450430 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86ac6565-97e8-466f-9ba8-f82843763246-istio-token\") pod \"e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh\" (UID: \"86ac6565-97e8-466f-9ba8-f82843763246\") " pod="openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh" Oct 30 18:38:15.462157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:15.450468 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86ac6565-97e8-466f-9ba8-f82843763246-istio-podinfo\") pod \"e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh\" (UID: \"86ac6565-97e8-466f-9ba8-f82843763246\") " pod="openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh" Oct 30 18:38:15.462157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:15.450555 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86ac6565-97e8-466f-9ba8-f82843763246-istiod-ca-cert\") pod \"e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh\" (UID: \"86ac6565-97e8-466f-9ba8-f82843763246\") " pod="openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh" Oct 30 18:38:15.462157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:15.450636 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ac6565-97e8-466f-9ba8-f82843763246-credential-socket\") pod \"e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh\" (UID: \"86ac6565-97e8-466f-9ba8-f82843763246\") " pod="openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh" Oct 30 18:38:15.462157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:15.450678 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86ac6565-97e8-466f-9ba8-f82843763246-workload-certs\") pod \"e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh\" (UID: \"86ac6565-97e8-466f-9ba8-f82843763246\") " pod="openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh" Oct 30 18:38:15.462157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:15.450719 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5klxh\" (UniqueName: \"kubernetes.io/projected/86ac6565-97e8-466f-9ba8-f82843763246-kube-api-access-5klxh\") pod \"e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh\" (UID: \"86ac6565-97e8-466f-9ba8-f82843763246\") " pod="openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh" Oct 30 18:38:15.462157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:15.450788 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ac6565-97e8-466f-9ba8-f82843763246-workload-socket\") pod \"e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh\" (UID: \"86ac6565-97e8-466f-9ba8-f82843763246\") " pod="openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh" Oct 30 18:38:15.565799 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:15.555117 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86ac6565-97e8-466f-9ba8-f82843763246-istiod-ca-cert\") pod \"e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh\" (UID: \"86ac6565-97e8-466f-9ba8-f82843763246\") " pod="openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh" Oct 30 18:38:15.565799 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:15.557655 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ac6565-97e8-466f-9ba8-f82843763246-credential-socket\") pod \"e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh\" (UID: \"86ac6565-97e8-466f-9ba8-f82843763246\") " pod="openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh" Oct 30 18:38:15.565799 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:15.557707 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86ac6565-97e8-466f-9ba8-f82843763246-workload-certs\") pod \"e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh\" (UID: \"86ac6565-97e8-466f-9ba8-f82843763246\") " pod="openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh" Oct 30 18:38:15.565799 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:15.557752 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5klxh\" (UniqueName: \"kubernetes.io/projected/86ac6565-97e8-466f-9ba8-f82843763246-kube-api-access-5klxh\") pod \"e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh\" (UID: \"86ac6565-97e8-466f-9ba8-f82843763246\") " pod="openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh" Oct 30 18:38:15.565799 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:15.557845 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ac6565-97e8-466f-9ba8-f82843763246-workload-socket\") pod \"e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh\" (UID: \"86ac6565-97e8-466f-9ba8-f82843763246\") " pod="openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh" Oct 30 18:38:15.565799 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:15.557923 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86ac6565-97e8-466f-9ba8-f82843763246-istio-envoy\") pod \"e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh\" (UID: \"86ac6565-97e8-466f-9ba8-f82843763246\") " pod="openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh" Oct 30 18:38:15.565799 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:15.558003 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86ac6565-97e8-466f-9ba8-f82843763246-istio-data\") pod \"e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh\" (UID: \"86ac6565-97e8-466f-9ba8-f82843763246\") " pod="openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh" Oct 30 18:38:15.565799 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:15.558046 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86ac6565-97e8-466f-9ba8-f82843763246-istio-token\") pod \"e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh\" (UID: \"86ac6565-97e8-466f-9ba8-f82843763246\") " pod="openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh" Oct 30 18:38:15.565799 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:15.558089 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86ac6565-97e8-466f-9ba8-f82843763246-istio-podinfo\") pod \"e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh\" (UID: \"86ac6565-97e8-466f-9ba8-f82843763246\") " pod="openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh" Oct 30 18:38:15.565799 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:15.560359 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86ac6565-97e8-466f-9ba8-f82843763246-istiod-ca-cert\") pod \"e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh\" (UID: \"86ac6565-97e8-466f-9ba8-f82843763246\") " pod="openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh" Oct 30 18:38:15.565799 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:15.560682 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ac6565-97e8-466f-9ba8-f82843763246-credential-socket\") pod \"e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh\" (UID: \"86ac6565-97e8-466f-9ba8-f82843763246\") " pod="openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh" Oct 30 18:38:15.565799 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:15.560963 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86ac6565-97e8-466f-9ba8-f82843763246-workload-certs\") pod \"e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh\" (UID: \"86ac6565-97e8-466f-9ba8-f82843763246\") " pod="openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh" Oct 30 18:38:15.565799 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:15.561145 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ac6565-97e8-466f-9ba8-f82843763246-workload-socket\") pod \"e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh\" (UID: \"86ac6565-97e8-466f-9ba8-f82843763246\") " pod="openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh" Oct 30 18:38:15.565799 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:15.561276 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86ac6565-97e8-466f-9ba8-f82843763246-istio-data\") pod \"e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh\" (UID: \"86ac6565-97e8-466f-9ba8-f82843763246\") " pod="openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh" Oct 30 18:38:15.608161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:15.607718 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86ac6565-97e8-466f-9ba8-f82843763246-istio-podinfo\") pod \"e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh\" (UID: \"86ac6565-97e8-466f-9ba8-f82843763246\") " pod="openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh" Oct 30 18:38:15.622589 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:15.615216 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86ac6565-97e8-466f-9ba8-f82843763246-istio-envoy\") pod \"e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh\" (UID: \"86ac6565-97e8-466f-9ba8-f82843763246\") " pod="openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh" Oct 30 18:38:15.646363 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:15.645787 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86ac6565-97e8-466f-9ba8-f82843763246-istio-token\") pod \"e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh\" (UID: \"86ac6565-97e8-466f-9ba8-f82843763246\") " pod="openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh" Oct 30 18:38:15.646363 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:15.645793 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-2107d4-7bf5b8d588-hz7nk" Oct 30 18:38:15.649335 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:15.648454 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2107d4-7588cd98bc-nt76m" Oct 30 18:38:15.657328 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:15.657260 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5klxh\" (UniqueName: \"kubernetes.io/projected/86ac6565-97e8-466f-9ba8-f82843763246-kube-api-access-5klxh\") pod \"e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh\" (UID: \"86ac6565-97e8-466f-9ba8-f82843763246\") " pod="openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh" Oct 30 18:38:15.903950 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:15.903912 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh" Oct 30 18:38:16.244976 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:38:16.244943 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86ac6565_97e8_466f_9ba8_f82843763246.slice/crio-6c1c66a2985f99cb33a2823174cbf6afb205b80d182b922e6a3e4ade4a5edc89 WatchSource:0}: Error finding container 6c1c66a2985f99cb33a2823174cbf6afb205b80d182b922e6a3e4ade4a5edc89: Status 404 returned error can't find the container with id 6c1c66a2985f99cb33a2823174cbf6afb205b80d182b922e6a3e4ade4a5edc89 Oct 30 18:38:16.247772 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:16.247744 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh"] Oct 30 18:38:16.248082 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:16.248047 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:38:16.248183 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:16.248125 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:38:16.248269 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:16.248192 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:38:16.695581 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:16.695540 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh" event={"ID":"86ac6565-97e8-466f-9ba8-f82843763246","Type":"ContainerStarted","Data":"5894eef59ba11524541f2a14694ac3bd7d8c6aaefe60195f8b4d235b645ff724"} Oct 30 18:38:16.696108 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:16.695628 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh" event={"ID":"86ac6565-97e8-466f-9ba8-f82843763246","Type":"ContainerStarted","Data":"6c1c66a2985f99cb33a2823174cbf6afb205b80d182b922e6a3e4ade4a5edc89"} Oct 30 18:38:16.749642 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:16.749589 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh" podStartSLOduration=1.749572958 podStartE2EDuration="1.749572958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:16.74645155 +0000 UTC m=+2393.836513888" watchObservedRunningTime="2026-10-30 18:38:16.749572958 +0000 UTC m=+2393.839635298" Oct 30 18:38:16.908857 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:16.904562 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh" Oct 30 18:38:17.561916 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:38:17.561878 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:38:17.910453 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:17.910424 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh" Oct 30 18:38:18.712785 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:18.712753 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh" Oct 30 18:38:18.713790 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:18.713766 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh" Oct 30 18:38:19.635184 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:19.633500 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c095fa-7ff45cdc78-lvpxv" Oct 30 18:38:19.636272 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:19.636251 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c095fa-5d479bd6f9-rgj6w" Oct 30 18:38:23.612023 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:23.611999 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52rn6_516c6b6c-fc40-49cf-93b5-1c4471b02cf0/ovn-acl-logging/0.log" Oct 30 18:38:23.612525 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:23.612432 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52rn6_516c6b6c-fc40-49cf-93b5-1c4471b02cf0/ovn-acl-logging/0.log" Oct 30 18:38:31.483976 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:31.483933 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2107d4-84dfc46fbf-wz2h2"] Oct 30 18:38:31.490443 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:31.490420 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2107d4-84dfc46fbf-wz2h2" Oct 30 18:38:31.512482 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:31.512437 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-2107d4-serving-cert\"" Oct 30 18:38:31.513247 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:31.513217 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2107d4-84dfc46fbf-wz2h2"] Oct 30 18:38:31.643545 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:31.643505 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83d795bd-5ce3-4648-8823-be54dddc5580-maas-api-tls\") pod \"maas-api-e2e-extmod-a-2107d4-84dfc46fbf-wz2h2\" (UID: \"83d795bd-5ce3-4648-8823-be54dddc5580\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2107d4-84dfc46fbf-wz2h2" Oct 30 18:38:31.643739 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:31.643683 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83d795bd-5ce3-4648-8823-be54dddc5580-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-2107d4-84dfc46fbf-wz2h2\" (UID: \"83d795bd-5ce3-4648-8823-be54dddc5580\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2107d4-84dfc46fbf-wz2h2" Oct 30 18:38:31.643866 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:31.643839 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvd8q\" (UniqueName: \"kubernetes.io/projected/83d795bd-5ce3-4648-8823-be54dddc5580-kube-api-access-zvd8q\") pod \"maas-api-e2e-extmod-a-2107d4-84dfc46fbf-wz2h2\" (UID: \"83d795bd-5ce3-4648-8823-be54dddc5580\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2107d4-84dfc46fbf-wz2h2" Oct 30 18:38:31.643968 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:31.643883 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83d795bd-5ce3-4648-8823-be54dddc5580-metrics-certs\") pod \"maas-api-e2e-extmod-a-2107d4-84dfc46fbf-wz2h2\" (UID: \"83d795bd-5ce3-4648-8823-be54dddc5580\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2107d4-84dfc46fbf-wz2h2" Oct 30 18:38:31.747093 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:31.746986 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83d795bd-5ce3-4648-8823-be54dddc5580-maas-api-tls\") pod \"maas-api-e2e-extmod-a-2107d4-84dfc46fbf-wz2h2\" (UID: \"83d795bd-5ce3-4648-8823-be54dddc5580\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2107d4-84dfc46fbf-wz2h2" Oct 30 18:38:31.747596 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:31.747573 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83d795bd-5ce3-4648-8823-be54dddc5580-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-2107d4-84dfc46fbf-wz2h2\" (UID: \"83d795bd-5ce3-4648-8823-be54dddc5580\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2107d4-84dfc46fbf-wz2h2" Oct 30 18:38:31.748808 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:31.748785 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvd8q\" (UniqueName: \"kubernetes.io/projected/83d795bd-5ce3-4648-8823-be54dddc5580-kube-api-access-zvd8q\") pod \"maas-api-e2e-extmod-a-2107d4-84dfc46fbf-wz2h2\" (UID: \"83d795bd-5ce3-4648-8823-be54dddc5580\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2107d4-84dfc46fbf-wz2h2" Oct 30 18:38:31.749404 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:31.749385 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83d795bd-5ce3-4648-8823-be54dddc5580-metrics-certs\") pod \"maas-api-e2e-extmod-a-2107d4-84dfc46fbf-wz2h2\" (UID: \"83d795bd-5ce3-4648-8823-be54dddc5580\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2107d4-84dfc46fbf-wz2h2" Oct 30 18:38:31.749935 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:31.748579 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83d795bd-5ce3-4648-8823-be54dddc5580-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-2107d4-84dfc46fbf-wz2h2\" (UID: \"83d795bd-5ce3-4648-8823-be54dddc5580\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2107d4-84dfc46fbf-wz2h2" Oct 30 18:38:31.753150 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:31.753104 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83d795bd-5ce3-4648-8823-be54dddc5580-metrics-certs\") pod \"maas-api-e2e-extmod-a-2107d4-84dfc46fbf-wz2h2\" (UID: \"83d795bd-5ce3-4648-8823-be54dddc5580\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2107d4-84dfc46fbf-wz2h2" Oct 30 18:38:31.753292 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:31.753268 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83d795bd-5ce3-4648-8823-be54dddc5580-maas-api-tls\") pod \"maas-api-e2e-extmod-a-2107d4-84dfc46fbf-wz2h2\" (UID: \"83d795bd-5ce3-4648-8823-be54dddc5580\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2107d4-84dfc46fbf-wz2h2" Oct 30 18:38:31.770294 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:31.770262 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvd8q\" (UniqueName: \"kubernetes.io/projected/83d795bd-5ce3-4648-8823-be54dddc5580-kube-api-access-zvd8q\") pod \"maas-api-e2e-extmod-a-2107d4-84dfc46fbf-wz2h2\" (UID: \"83d795bd-5ce3-4648-8823-be54dddc5580\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2107d4-84dfc46fbf-wz2h2" Oct 30 18:38:31.811396 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:31.811362 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2107d4-84dfc46fbf-wz2h2" Oct 30 18:38:32.128095 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:38:32.128044 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83d795bd_5ce3_4648_8823_be54dddc5580.slice/crio-91350d52495f1ff61da61ca51a678ad5829a972b559dea0a43ba5885d9682873 WatchSource:0}: Error finding container 91350d52495f1ff61da61ca51a678ad5829a972b559dea0a43ba5885d9682873: Status 404 returned error can't find the container with id 91350d52495f1ff61da61ca51a678ad5829a972b559dea0a43ba5885d9682873 Oct 30 18:38:32.128258 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:32.128154 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2107d4-84dfc46fbf-wz2h2"] Oct 30 18:38:32.859226 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:32.859104 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2107d4-84dfc46fbf-wz2h2" event={"ID":"83d795bd-5ce3-4648-8823-be54dddc5580","Type":"ContainerStarted","Data":"fd9e8d8fc7b6e887d23f7d65009a58aff70607e056ddf705848e94d6b25190ac"} Oct 30 18:38:32.859226 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:32.859188 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2107d4-84dfc46fbf-wz2h2" event={"ID":"83d795bd-5ce3-4648-8823-be54dddc5580","Type":"ContainerStarted","Data":"91350d52495f1ff61da61ca51a678ad5829a972b559dea0a43ba5885d9682873"} Oct 30 18:38:32.859226 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:32.859218 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2107d4-84dfc46fbf-wz2h2" Oct 30 18:38:32.906943 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:32.906892 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2107d4-84dfc46fbf-wz2h2" podStartSLOduration=1.588070157 podStartE2EDuration="1.906878533s" totalImagesPullingTime="318.808376ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:32.90445368 +0000 UTC m=+2409.994516018" watchObservedRunningTime="2026-10-30 18:38:32.906878533 +0000 UTC m=+2409.996940872" Oct 30 18:38:38.873071 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:38.873045 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2107d4-84dfc46fbf-wz2h2" Oct 30 18:38:40.258985 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:40.258951 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7628-b6f680"] Oct 30 18:38:40.259778 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:40.259747 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7628-b6f680" podUID="18265c8d-6de9-4922-8b89-05b35823a21c" containerName="test-curl-7628-b6f680" containerID="cri-o://ba5544281d7a068a41f76337d10de8e101b793ebad364325cdb5716cfe038ff7" gracePeriod=2 Oct 30 18:38:40.265650 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:40.265624 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7628-b6f680"] Oct 30 18:38:40.273312 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:38:40.273275 4442 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7628-b6f680\" is forbidden: User \"system:node:ip-10-0-138-176.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-176.ec2.internal' and this object" podUID="18265c8d-6de9-4922-8b89-05b35823a21c" pod="opendatahub/test-curl-7628-b6f680" Oct 30 18:38:42.456092 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:42.456054 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq"] Oct 30 18:38:42.460359 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:42.457162 4442 cpu_manager.go:414] "containerMap: removing container" podUID="18265c8d-6de9-4922-8b89-05b35823a21c" containerName="test-curl-7628-b6f680" Oct 30 18:38:42.460359 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:42.457188 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18265c8d-6de9-4922-8b89-05b35823a21c" containerName="test-curl-7628-b6f680" Oct 30 18:38:42.460359 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:42.457424 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="18265c8d-6de9-4922-8b89-05b35823a21c" containerName="test-curl-7628-b6f680" Oct 30 18:38:42.460359 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:42.462203 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" Oct 30 18:38:42.482530 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:42.482496 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-1c4bcd-openshift-default-dockercfg-2ckdj\"" Oct 30 18:38:42.501277 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:42.501246 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq"] Oct 30 18:38:42.567020 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:42.566987 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7628-b6f680" Oct 30 18:38:42.625168 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:42.620969 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-istio-envoy\") pod \"e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq\" (UID: \"b0040e62-2a9b-4af5-82bd-730b0f5aba4f\") " pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" Oct 30 18:38:42.625168 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:42.621051 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndtd7\" (UniqueName: \"kubernetes.io/projected/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-kube-api-access-ndtd7\") pod \"e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq\" (UID: \"b0040e62-2a9b-4af5-82bd-730b0f5aba4f\") " pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" Oct 30 18:38:42.625168 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:42.621086 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-workload-certs\") pod \"e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq\" (UID: \"b0040e62-2a9b-4af5-82bd-730b0f5aba4f\") " pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" Oct 30 18:38:42.625168 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:42.621193 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-istio-token\") pod \"e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq\" (UID: \"b0040e62-2a9b-4af5-82bd-730b0f5aba4f\") " pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" Oct 30 18:38:42.625168 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:42.621238 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-istio-data\") pod \"e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq\" (UID: \"b0040e62-2a9b-4af5-82bd-730b0f5aba4f\") " pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" Oct 30 18:38:42.625168 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:42.621282 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-credential-socket\") pod \"e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq\" (UID: \"b0040e62-2a9b-4af5-82bd-730b0f5aba4f\") " pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" Oct 30 18:38:42.625168 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:42.621342 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-istio-podinfo\") pod \"e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq\" (UID: \"b0040e62-2a9b-4af5-82bd-730b0f5aba4f\") " pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" Oct 30 18:38:42.625168 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:42.621465 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-workload-socket\") pod \"e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq\" (UID: \"b0040e62-2a9b-4af5-82bd-730b0f5aba4f\") " pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" Oct 30 18:38:42.625168 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:42.621501 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-istiod-ca-cert\") pod \"e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq\" (UID: \"b0040e62-2a9b-4af5-82bd-730b0f5aba4f\") " pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" Oct 30 18:38:42.729161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:42.725697 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18265c8d-6de9-4922-8b89-05b35823a21c-kube-api-access-6nhpt\" (UniqueName: \"kubernetes.io/projected/18265c8d-6de9-4922-8b89-05b35823a21c-kube-api-access-6nhpt\") pod \"18265c8d-6de9-4922-8b89-05b35823a21c\" (UID: \"18265c8d-6de9-4922-8b89-05b35823a21c\") " Oct 30 18:38:42.729161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:42.726042 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-istiod-ca-cert\") pod \"e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq\" (UID: \"b0040e62-2a9b-4af5-82bd-730b0f5aba4f\") " pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" Oct 30 18:38:42.729161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:42.726162 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-istio-envoy\") pod \"e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq\" (UID: \"b0040e62-2a9b-4af5-82bd-730b0f5aba4f\") " pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" Oct 30 18:38:42.729161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:42.726224 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndtd7\" (UniqueName: \"kubernetes.io/projected/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-kube-api-access-ndtd7\") pod \"e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq\" (UID: \"b0040e62-2a9b-4af5-82bd-730b0f5aba4f\") " pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" Oct 30 18:38:42.729161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:42.726257 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-workload-certs\") pod \"e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq\" (UID: \"b0040e62-2a9b-4af5-82bd-730b0f5aba4f\") " pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" Oct 30 18:38:42.729161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:42.726335 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-istio-token\") pod \"e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq\" (UID: \"b0040e62-2a9b-4af5-82bd-730b0f5aba4f\") " pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" Oct 30 18:38:42.729161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:42.726372 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-istio-data\") pod \"e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq\" (UID: \"b0040e62-2a9b-4af5-82bd-730b0f5aba4f\") " pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" Oct 30 18:38:42.729161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:42.726413 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-credential-socket\") pod \"e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq\" (UID: \"b0040e62-2a9b-4af5-82bd-730b0f5aba4f\") " pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" Oct 30 18:38:42.729161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:42.726469 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-istio-podinfo\") pod \"e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq\" (UID: \"b0040e62-2a9b-4af5-82bd-730b0f5aba4f\") " pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" Oct 30 18:38:42.729161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:42.726592 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-workload-socket\") pod \"e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq\" (UID: \"b0040e62-2a9b-4af5-82bd-730b0f5aba4f\") " pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" Oct 30 18:38:42.729161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:42.727063 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-workload-socket\") pod \"e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq\" (UID: \"b0040e62-2a9b-4af5-82bd-730b0f5aba4f\") " pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" Oct 30 18:38:42.741109 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:42.740190 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-istiod-ca-cert\") pod \"e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq\" (UID: \"b0040e62-2a9b-4af5-82bd-730b0f5aba4f\") " pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" Oct 30 18:38:42.741109 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:42.740501 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-istio-data\") pod \"e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq\" (UID: \"b0040e62-2a9b-4af5-82bd-730b0f5aba4f\") " pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" Oct 30 18:38:42.741109 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:42.740784 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-credential-socket\") pod \"e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq\" (UID: \"b0040e62-2a9b-4af5-82bd-730b0f5aba4f\") " pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" Oct 30 18:38:42.757162 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:42.741694 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-workload-certs\") pod \"e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq\" (UID: \"b0040e62-2a9b-4af5-82bd-730b0f5aba4f\") " pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" Oct 30 18:38:42.757540 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:42.757086 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-istio-podinfo\") pod \"e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq\" (UID: \"b0040e62-2a9b-4af5-82bd-730b0f5aba4f\") " pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" Oct 30 18:38:42.775165 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:42.765348 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18265c8d-6de9-4922-8b89-05b35823a21c-kube-api-access-6nhpt" pod "18265c8d-6de9-4922-8b89-05b35823a21c" (UID: "18265c8d-6de9-4922-8b89-05b35823a21c"). InnerVolumeSpecName "kube-api-access-6nhpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:42.799216 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:42.782521 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-istio-envoy\") pod \"e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq\" (UID: \"b0040e62-2a9b-4af5-82bd-730b0f5aba4f\") " pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" Oct 30 18:38:42.803975 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:42.803894 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndtd7\" (UniqueName: \"kubernetes.io/projected/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-kube-api-access-ndtd7\") pod \"e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq\" (UID: \"b0040e62-2a9b-4af5-82bd-730b0f5aba4f\") " pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" Oct 30 18:38:42.818652 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:42.818577 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-istio-token\") pod \"e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq\" (UID: \"b0040e62-2a9b-4af5-82bd-730b0f5aba4f\") " pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" Oct 30 18:38:42.829686 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:42.829651 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nhpt\" (UniqueName: \"kubernetes.io/projected/18265c8d-6de9-4922-8b89-05b35823a21c-kube-api-access-6nhpt\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:38:42.965570 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:42.965533 4442 generic.go:356] "Generic (PLEG): container finished" podID="18265c8d-6de9-4922-8b89-05b35823a21c" containerID="ba5544281d7a068a41f76337d10de8e101b793ebad364325cdb5716cfe038ff7" exitCode=137 Oct 30 18:38:42.965761 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:42.965588 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7628-b6f680" Oct 30 18:38:42.965761 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:42.965638 4442 scope.go:122] "RemoveContainer" containerID="ba5544281d7a068a41f76337d10de8e101b793ebad364325cdb5716cfe038ff7" Oct 30 18:38:42.980510 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:42.980491 4442 scope.go:122] "RemoveContainer" containerID="ba5544281d7a068a41f76337d10de8e101b793ebad364325cdb5716cfe038ff7" Oct 30 18:38:42.980838 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:38:42.980817 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba5544281d7a068a41f76337d10de8e101b793ebad364325cdb5716cfe038ff7\": container with ID starting with ba5544281d7a068a41f76337d10de8e101b793ebad364325cdb5716cfe038ff7 not found: ID does not exist" containerID="ba5544281d7a068a41f76337d10de8e101b793ebad364325cdb5716cfe038ff7" Oct 30 18:38:42.980902 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:42.980847 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba5544281d7a068a41f76337d10de8e101b793ebad364325cdb5716cfe038ff7"} err="failed to get container status \"ba5544281d7a068a41f76337d10de8e101b793ebad364325cdb5716cfe038ff7\": rpc error: code = NotFound desc = could not find container \"ba5544281d7a068a41f76337d10de8e101b793ebad364325cdb5716cfe038ff7\": container with ID starting with ba5544281d7a068a41f76337d10de8e101b793ebad364325cdb5716cfe038ff7 not found: ID does not exist" Oct 30 18:38:42.989870 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:38:42.989823 4442 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7628-b6f680\" is forbidden: User \"system:node:ip-10-0-138-176.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-176.ec2.internal' and this object" podUID="18265c8d-6de9-4922-8b89-05b35823a21c" pod="opendatahub/test-curl-7628-b6f680" Oct 30 18:38:43.075204 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:38:43.075075 4442 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7628-b6f680\" is forbidden: User \"system:node:ip-10-0-138-176.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-176.ec2.internal' and this object" podUID="18265c8d-6de9-4922-8b89-05b35823a21c" pod="opendatahub/test-curl-7628-b6f680" Oct 30 18:38:43.079458 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:43.079427 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18265c8d-6de9-4922-8b89-05b35823a21c" path="/var/lib/kubelet/pods/18265c8d-6de9-4922-8b89-05b35823a21c/volumes" Oct 30 18:38:43.083335 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:43.083315 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" Oct 30 18:38:43.404244 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:38:43.404212 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0040e62_2a9b_4af5_82bd_730b0f5aba4f.slice/crio-c67c9577d1e4c574f84d99aa258353094fd0ce87905b854b62cc7c93f8388a95 WatchSource:0}: Error finding container c67c9577d1e4c574f84d99aa258353094fd0ce87905b854b62cc7c93f8388a95: Status 404 returned error can't find the container with id c67c9577d1e4c574f84d99aa258353094fd0ce87905b854b62cc7c93f8388a95 Oct 30 18:38:43.405367 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:43.405337 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq"] Oct 30 18:38:43.407365 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:43.407328 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:38:43.407432 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:43.407418 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:38:43.407479 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:43.407464 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:38:43.980473 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:43.980439 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" event={"ID":"b0040e62-2a9b-4af5-82bd-730b0f5aba4f","Type":"ContainerStarted","Data":"da4ca7073b149a7b14639f368ab7d085b7db275859c059436f4cffa352298e1f"} Oct 30 18:38:43.981221 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:43.980524 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" event={"ID":"b0040e62-2a9b-4af5-82bd-730b0f5aba4f","Type":"ContainerStarted","Data":"c67c9577d1e4c574f84d99aa258353094fd0ce87905b854b62cc7c93f8388a95"} Oct 30 18:38:44.063770 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:44.063704 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" podStartSLOduration=2.06368533 podStartE2EDuration="2.06368533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:44.056306955 +0000 UTC m=+2421.146369295" watchObservedRunningTime="2026-10-30 18:38:44.06368533 +0000 UTC m=+2421.153747671" Oct 30 18:38:44.084398 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:44.084366 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" Oct 30 18:38:44.092812 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:44.092784 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" Oct 30 18:38:44.989362 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:44.989321 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" Oct 30 18:38:44.990591 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:44.990567 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" Oct 30 18:38:47.561826 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:38:47.561787 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:38:48.816888 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:48.816856 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1c4bcd-76ffbffdcf-q4ttx"] Oct 30 18:38:48.821383 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:48.821365 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1c4bcd-76ffbffdcf-q4ttx" Oct 30 18:38:48.851823 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:48.847563 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-1c4bcd\"" Oct 30 18:38:48.857590 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:48.857558 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-1c4bcd-dockercfg-4mzmv\"" Oct 30 18:38:48.859633 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:48.859604 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1c4bcd-76ffbffdcf-q4ttx"] Oct 30 18:38:48.947568 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:48.904530 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98ldz\" (UniqueName: \"kubernetes.io/projected/f1e71600-36ee-4faa-872d-da1203e65bdc-kube-api-access-98ldz\") pod \"payload-pre-processing-e2e-extmod-a-1c4bcd-76ffbffdcf-q4ttx\" (UID: \"f1e71600-36ee-4faa-872d-da1203e65bdc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1c4bcd-76ffbffdcf-q4ttx" Oct 30 18:38:48.999171 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:48.998263 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1e71600-36ee-4faa-872d-da1203e65bdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-1c4bcd-76ffbffdcf-q4ttx\" (UID: \"f1e71600-36ee-4faa-872d-da1203e65bdc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1c4bcd-76ffbffdcf-q4ttx" Oct 30 18:38:49.114257 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:49.114155 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98ldz\" (UniqueName: \"kubernetes.io/projected/f1e71600-36ee-4faa-872d-da1203e65bdc-kube-api-access-98ldz\") pod \"payload-pre-processing-e2e-extmod-a-1c4bcd-76ffbffdcf-q4ttx\" (UID: \"f1e71600-36ee-4faa-872d-da1203e65bdc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1c4bcd-76ffbffdcf-q4ttx" Oct 30 18:38:49.114257 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:49.114233 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1e71600-36ee-4faa-872d-da1203e65bdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-1c4bcd-76ffbffdcf-q4ttx\" (UID: \"f1e71600-36ee-4faa-872d-da1203e65bdc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1c4bcd-76ffbffdcf-q4ttx" Oct 30 18:38:49.115620 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:49.115430 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1e71600-36ee-4faa-872d-da1203e65bdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-1c4bcd-76ffbffdcf-q4ttx\" (UID: \"f1e71600-36ee-4faa-872d-da1203e65bdc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1c4bcd-76ffbffdcf-q4ttx" Oct 30 18:38:49.175773 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:49.175726 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1c4bcd-55d686745d-4hhzp"] Oct 30 18:38:49.178547 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:49.178514 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98ldz\" (UniqueName: \"kubernetes.io/projected/f1e71600-36ee-4faa-872d-da1203e65bdc-kube-api-access-98ldz\") pod \"payload-pre-processing-e2e-extmod-a-1c4bcd-76ffbffdcf-q4ttx\" (UID: \"f1e71600-36ee-4faa-872d-da1203e65bdc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1c4bcd-76ffbffdcf-q4ttx" Oct 30 18:38:49.181689 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:49.181666 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-1c4bcd-55d686745d-4hhzp" Oct 30 18:38:49.184612 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:49.184587 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1c4bcd-55d686745d-4hhzp"] Oct 30 18:38:49.317631 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:49.317584 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74qnd\" (UniqueName: \"kubernetes.io/projected/034e375c-3ae0-40db-94ed-9d328fee5639-kube-api-access-74qnd\") pod \"payload-processing-e2e-extmod-a-1c4bcd-55d686745d-4hhzp\" (UID: \"034e375c-3ae0-40db-94ed-9d328fee5639\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1c4bcd-55d686745d-4hhzp" Oct 30 18:38:49.317841 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:49.317660 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/034e375c-3ae0-40db-94ed-9d328fee5639-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-1c4bcd-55d686745d-4hhzp\" (UID: \"034e375c-3ae0-40db-94ed-9d328fee5639\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1c4bcd-55d686745d-4hhzp" Oct 30 18:38:49.418967 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:49.418927 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74qnd\" (UniqueName: \"kubernetes.io/projected/034e375c-3ae0-40db-94ed-9d328fee5639-kube-api-access-74qnd\") pod \"payload-processing-e2e-extmod-a-1c4bcd-55d686745d-4hhzp\" (UID: \"034e375c-3ae0-40db-94ed-9d328fee5639\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1c4bcd-55d686745d-4hhzp" Oct 30 18:38:49.419214 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:49.418996 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/034e375c-3ae0-40db-94ed-9d328fee5639-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-1c4bcd-55d686745d-4hhzp\" (UID: \"034e375c-3ae0-40db-94ed-9d328fee5639\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1c4bcd-55d686745d-4hhzp" Oct 30 18:38:49.419838 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:49.419819 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/034e375c-3ae0-40db-94ed-9d328fee5639-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-1c4bcd-55d686745d-4hhzp\" (UID: \"034e375c-3ae0-40db-94ed-9d328fee5639\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1c4bcd-55d686745d-4hhzp" Oct 30 18:38:49.455374 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:49.455341 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74qnd\" (UniqueName: \"kubernetes.io/projected/034e375c-3ae0-40db-94ed-9d328fee5639-kube-api-access-74qnd\") pod \"payload-processing-e2e-extmod-a-1c4bcd-55d686745d-4hhzp\" (UID: \"034e375c-3ae0-40db-94ed-9d328fee5639\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1c4bcd-55d686745d-4hhzp" Oct 30 18:38:49.462477 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:49.462441 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1c4bcd-76ffbffdcf-q4ttx" Oct 30 18:38:49.509897 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:49.509616 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-1c4bcd-55d686745d-4hhzp" Oct 30 18:38:49.796983 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:38:49.796954 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1e71600_36ee_4faa_872d_da1203e65bdc.slice/crio-9ece9b5ffc8693361d83b83192b7e0cd346e0677d828271cecc339d912aa55b6 WatchSource:0}: Error finding container 9ece9b5ffc8693361d83b83192b7e0cd346e0677d828271cecc339d912aa55b6: Status 404 returned error can't find the container with id 9ece9b5ffc8693361d83b83192b7e0cd346e0677d828271cecc339d912aa55b6 Oct 30 18:38:49.798126 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:49.798102 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1c4bcd-76ffbffdcf-q4ttx"] Oct 30 18:38:49.879872 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:38:49.879834 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod034e375c_3ae0_40db_94ed_9d328fee5639.slice/crio-da40dbccb7cce3a415514f2bc22392d9173e45b35737679528ab43dc95018598 WatchSource:0}: Error finding container da40dbccb7cce3a415514f2bc22392d9173e45b35737679528ab43dc95018598: Status 404 returned error can't find the container with id da40dbccb7cce3a415514f2bc22392d9173e45b35737679528ab43dc95018598 Oct 30 18:38:49.881443 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:49.881361 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1c4bcd-55d686745d-4hhzp"] Oct 30 18:38:50.183389 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:50.183348 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1c4bcd-76ffbffdcf-q4ttx" event={"ID":"f1e71600-36ee-4faa-872d-da1203e65bdc","Type":"ContainerStarted","Data":"5ed1af2f9609f59c1d498bd70e609d75a638fccd5cf704814b38adda385a986c"} Oct 30 18:38:50.183594 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:50.183403 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1c4bcd-76ffbffdcf-q4ttx" event={"ID":"f1e71600-36ee-4faa-872d-da1203e65bdc","Type":"ContainerStarted","Data":"9ece9b5ffc8693361d83b83192b7e0cd346e0677d828271cecc339d912aa55b6"} Oct 30 18:38:50.183594 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:50.183433 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1c4bcd-76ffbffdcf-q4ttx" Oct 30 18:38:50.185524 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:50.185500 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-1c4bcd-55d686745d-4hhzp" event={"ID":"034e375c-3ae0-40db-94ed-9d328fee5639","Type":"ContainerStarted","Data":"3cc92cf6008580c1750453be5efb7bb823500363f66fa6ef47c52b78a1c51661"} Oct 30 18:38:50.185670 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:50.185565 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-1c4bcd-55d686745d-4hhzp" Oct 30 18:38:50.185670 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:50.185594 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-1c4bcd-55d686745d-4hhzp" event={"ID":"034e375c-3ae0-40db-94ed-9d328fee5639","Type":"ContainerStarted","Data":"da40dbccb7cce3a415514f2bc22392d9173e45b35737679528ab43dc95018598"} Oct 30 18:38:50.247234 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:50.247170 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1c4bcd-76ffbffdcf-q4ttx" podStartSLOduration=2.247149833 podStartE2EDuration="2.247149833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:50.221835588 +0000 UTC m=+2427.311897928" watchObservedRunningTime="2026-10-30 18:38:50.247149833 +0000 UTC m=+2427.337212173" Oct 30 18:38:50.275619 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:50.275563 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-1c4bcd-55d686745d-4hhzp" podStartSLOduration=2.275543789 podStartE2EDuration="2.275543789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:50.274489437 +0000 UTC m=+2427.364551778" watchObservedRunningTime="2026-10-30 18:38:50.275543789 +0000 UTC m=+2427.365606153" Oct 30 18:38:56.172763 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:56.172729 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78"] Oct 30 18:38:56.178783 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:56.178758 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" Oct 30 18:38:56.210456 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:56.210408 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-d1a330-openshift-default-dockercfg-t4z2z\"" Oct 30 18:38:56.211818 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:56.211786 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78"] Oct 30 18:38:56.260158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:56.247076 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd0f5b27-6383-4d55-a840-3938d11d9f6f-istio-podinfo\") pod \"e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78\" (UID: \"dd0f5b27-6383-4d55-a840-3938d11d9f6f\") " pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" Oct 30 18:38:56.260158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:56.247150 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd0f5b27-6383-4d55-a840-3938d11d9f6f-istio-token\") pod \"e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78\" (UID: \"dd0f5b27-6383-4d55-a840-3938d11d9f6f\") " pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" Oct 30 18:38:56.260158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:56.247241 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ggcx\" (UniqueName: \"kubernetes.io/projected/dd0f5b27-6383-4d55-a840-3938d11d9f6f-kube-api-access-4ggcx\") pod \"e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78\" (UID: \"dd0f5b27-6383-4d55-a840-3938d11d9f6f\") " pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" Oct 30 18:38:56.260158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:56.247283 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0f5b27-6383-4d55-a840-3938d11d9f6f-credential-socket\") pod \"e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78\" (UID: \"dd0f5b27-6383-4d55-a840-3938d11d9f6f\") " pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" Oct 30 18:38:56.260158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:56.247336 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd0f5b27-6383-4d55-a840-3938d11d9f6f-istio-data\") pod \"e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78\" (UID: \"dd0f5b27-6383-4d55-a840-3938d11d9f6f\") " pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" Oct 30 18:38:56.260158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:56.247360 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd0f5b27-6383-4d55-a840-3938d11d9f6f-workload-certs\") pod \"e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78\" (UID: \"dd0f5b27-6383-4d55-a840-3938d11d9f6f\") " pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" Oct 30 18:38:56.260158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:56.247410 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0f5b27-6383-4d55-a840-3938d11d9f6f-workload-socket\") pod \"e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78\" (UID: \"dd0f5b27-6383-4d55-a840-3938d11d9f6f\") " pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" Oct 30 18:38:56.260158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:56.247681 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd0f5b27-6383-4d55-a840-3938d11d9f6f-istio-envoy\") pod \"e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78\" (UID: \"dd0f5b27-6383-4d55-a840-3938d11d9f6f\") " pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" Oct 30 18:38:56.260158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:56.247787 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd0f5b27-6383-4d55-a840-3938d11d9f6f-istiod-ca-cert\") pod \"e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78\" (UID: \"dd0f5b27-6383-4d55-a840-3938d11d9f6f\") " pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" Oct 30 18:38:56.358160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:56.355632 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd0f5b27-6383-4d55-a840-3938d11d9f6f-istio-podinfo\") pod \"e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78\" (UID: \"dd0f5b27-6383-4d55-a840-3938d11d9f6f\") " pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" Oct 30 18:38:56.358160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:56.355677 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd0f5b27-6383-4d55-a840-3938d11d9f6f-istio-token\") pod \"e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78\" (UID: \"dd0f5b27-6383-4d55-a840-3938d11d9f6f\") " pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" Oct 30 18:38:56.358160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:56.355760 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ggcx\" (UniqueName: \"kubernetes.io/projected/dd0f5b27-6383-4d55-a840-3938d11d9f6f-kube-api-access-4ggcx\") pod \"e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78\" (UID: \"dd0f5b27-6383-4d55-a840-3938d11d9f6f\") " pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" Oct 30 18:38:56.358160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:56.355795 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0f5b27-6383-4d55-a840-3938d11d9f6f-credential-socket\") pod \"e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78\" (UID: \"dd0f5b27-6383-4d55-a840-3938d11d9f6f\") " pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" Oct 30 18:38:56.358160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:56.355841 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd0f5b27-6383-4d55-a840-3938d11d9f6f-istio-data\") pod \"e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78\" (UID: \"dd0f5b27-6383-4d55-a840-3938d11d9f6f\") " pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" Oct 30 18:38:56.358160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:56.355868 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd0f5b27-6383-4d55-a840-3938d11d9f6f-workload-certs\") pod \"e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78\" (UID: \"dd0f5b27-6383-4d55-a840-3938d11d9f6f\") " pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" Oct 30 18:38:56.358160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:56.355911 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0f5b27-6383-4d55-a840-3938d11d9f6f-workload-socket\") pod \"e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78\" (UID: \"dd0f5b27-6383-4d55-a840-3938d11d9f6f\") " pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" Oct 30 18:38:56.358160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:56.356003 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd0f5b27-6383-4d55-a840-3938d11d9f6f-istio-envoy\") pod \"e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78\" (UID: \"dd0f5b27-6383-4d55-a840-3938d11d9f6f\") " pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" Oct 30 18:38:56.358160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:56.356070 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd0f5b27-6383-4d55-a840-3938d11d9f6f-istiod-ca-cert\") pod \"e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78\" (UID: \"dd0f5b27-6383-4d55-a840-3938d11d9f6f\") " pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" Oct 30 18:38:56.358160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:56.356929 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd0f5b27-6383-4d55-a840-3938d11d9f6f-istiod-ca-cert\") pod \"e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78\" (UID: \"dd0f5b27-6383-4d55-a840-3938d11d9f6f\") " pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" Oct 30 18:38:56.375450 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:56.375414 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd0f5b27-6383-4d55-a840-3938d11d9f6f-istio-data\") pod \"e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78\" (UID: \"dd0f5b27-6383-4d55-a840-3938d11d9f6f\") " pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" Oct 30 18:38:56.376461 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:56.376433 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0f5b27-6383-4d55-a840-3938d11d9f6f-credential-socket\") pod \"e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78\" (UID: \"dd0f5b27-6383-4d55-a840-3938d11d9f6f\") " pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" Oct 30 18:38:56.376910 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:56.376891 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0f5b27-6383-4d55-a840-3938d11d9f6f-workload-socket\") pod \"e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78\" (UID: \"dd0f5b27-6383-4d55-a840-3938d11d9f6f\") " pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" Oct 30 18:38:56.377299 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:56.377279 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd0f5b27-6383-4d55-a840-3938d11d9f6f-workload-certs\") pod \"e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78\" (UID: \"dd0f5b27-6383-4d55-a840-3938d11d9f6f\") " pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" Oct 30 18:38:56.380315 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:56.380269 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd0f5b27-6383-4d55-a840-3938d11d9f6f-istio-envoy\") pod \"e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78\" (UID: \"dd0f5b27-6383-4d55-a840-3938d11d9f6f\") " pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" Oct 30 18:38:56.439765 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:56.437898 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ggcx\" (UniqueName: \"kubernetes.io/projected/dd0f5b27-6383-4d55-a840-3938d11d9f6f-kube-api-access-4ggcx\") pod \"e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78\" (UID: \"dd0f5b27-6383-4d55-a840-3938d11d9f6f\") " pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" Oct 30 18:38:56.439765 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:56.438392 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd0f5b27-6383-4d55-a840-3938d11d9f6f-istio-token\") pod \"e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78\" (UID: \"dd0f5b27-6383-4d55-a840-3938d11d9f6f\") " pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" Oct 30 18:38:56.439765 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:56.439671 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd0f5b27-6383-4d55-a840-3938d11d9f6f-istio-podinfo\") pod \"e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78\" (UID: \"dd0f5b27-6383-4d55-a840-3938d11d9f6f\") " pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" Oct 30 18:38:56.516465 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:56.516421 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" Oct 30 18:38:56.900152 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:56.900095 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78"] Oct 30 18:38:56.915078 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:56.915046 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:38:56.915982 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:56.915197 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:38:56.915982 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:56.915276 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:38:57.298533 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:57.295430 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" event={"ID":"dd0f5b27-6383-4d55-a840-3938d11d9f6f","Type":"ContainerStarted","Data":"12e03a772cb15bc6cd60d5687c642999f0338add7027485fe6b82477e3cfc317"} Oct 30 18:38:57.298533 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:57.295501 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" event={"ID":"dd0f5b27-6383-4d55-a840-3938d11d9f6f","Type":"ContainerStarted","Data":"ab41cf59b31cad5bc16cf88362293c2a951c2cb5144ad5ad1c30c3f6a1c3e08e"} Oct 30 18:38:57.370604 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:57.370546 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" podStartSLOduration=1.370526406 podStartE2EDuration="1.370526406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:57.365303246 +0000 UTC m=+2434.455365586" watchObservedRunningTime="2026-10-30 18:38:57.370526406 +0000 UTC m=+2434.460588745" Oct 30 18:38:57.520179 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:57.518787 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" Oct 30 18:38:58.523760 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:58.523737 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" Oct 30 18:38:59.313612 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:59.313583 4442 generic.go:356] "Generic (PLEG): container finished" podID="c2be17bb-1852-432e-a24d-24d762b27661" containerID="5898fae1e077c4b36892e8caf35c10bcb3ca6c3a9ffe4ff6b724a2c19ca0413b" exitCode=0 Oct 30 18:38:59.313827 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:59.313663 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" event={"ID":"c2be17bb-1852-432e-a24d-24d762b27661","Type":"ContainerDied","Data":"5898fae1e077c4b36892e8caf35c10bcb3ca6c3a9ffe4ff6b724a2c19ca0413b"} Oct 30 18:38:59.314117 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:59.314095 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" Oct 30 18:38:59.314333 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:59.314316 4442 scope.go:122] "RemoveContainer" containerID="5898fae1e077c4b36892e8caf35c10bcb3ca6c3a9ffe4ff6b724a2c19ca0413b" Oct 30 18:38:59.315502 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:59.315354 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" Oct 30 18:38:59.316260 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:59.316227 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:38:59.316385 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:59.316332 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:38:59.316469 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:38:59.316388 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:39:00.333168 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:00.325102 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" event={"ID":"c2be17bb-1852-432e-a24d-24d762b27661","Type":"ContainerStarted","Data":"0aa40967c7cfb14bbce349d6d9cc1d59e670313aedc729a0e64390ea456b8ee8"} Oct 30 18:39:01.196444 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:01.196413 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-1c4bcd-55d686745d-4hhzp" Oct 30 18:39:01.196656 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:01.196491 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1c4bcd-76ffbffdcf-q4ttx" Oct 30 18:39:01.200324 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:01.200300 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" Oct 30 18:39:01.201212 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:01.201187 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Oct 30 18:39:01.201334 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:01.201242 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" podUID="c2be17bb-1852-432e-a24d-24d762b27661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Oct 30 18:39:02.200073 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:02.200033 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Oct 30 18:39:02.200490 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:02.200109 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" podUID="c2be17bb-1852-432e-a24d-24d762b27661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Oct 30 18:39:03.200483 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:03.200449 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Oct 30 18:39:03.200879 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:03.200519 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" podUID="c2be17bb-1852-432e-a24d-24d762b27661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Oct 30 18:39:04.200680 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:04.200643 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Oct 30 18:39:04.201095 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:04.200701 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" podUID="c2be17bb-1852-432e-a24d-24d762b27661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Oct 30 18:39:04.292851 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:04.292815 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" Oct 30 18:39:05.200847 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:05.200811 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Oct 30 18:39:05.201294 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:05.200875 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" podUID="c2be17bb-1852-432e-a24d-24d762b27661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Oct 30 18:39:05.842008 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:05.841967 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh"] Oct 30 18:39:05.844507 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:05.844473 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh" podUID="86ac6565-97e8-466f-9ba8-f82843763246" containerName="istio-proxy" containerID="cri-o://5894eef59ba11524541f2a14694ac3bd7d8c6aaefe60195f8b4d235b645ff724" gracePeriod=30 Oct 30 18:39:06.295518 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:06.295467 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" Oct 30 18:39:06.297081 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:06.297053 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" Oct 30 18:39:11.138301 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:11.138266 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh" Oct 30 18:39:11.272427 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:11.272396 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86ac6565-97e8-466f-9ba8-f82843763246-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ac6565-97e8-466f-9ba8-f82843763246-credential-socket\") pod \"86ac6565-97e8-466f-9ba8-f82843763246\" (UID: \"86ac6565-97e8-466f-9ba8-f82843763246\") " Oct 30 18:39:11.272640 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:11.272482 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86ac6565-97e8-466f-9ba8-f82843763246-istio-token\" (UniqueName: \"kubernetes.io/projected/86ac6565-97e8-466f-9ba8-f82843763246-istio-token\") pod \"86ac6565-97e8-466f-9ba8-f82843763246\" (UID: \"86ac6565-97e8-466f-9ba8-f82843763246\") " Oct 30 18:39:11.272640 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:11.272516 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86ac6565-97e8-466f-9ba8-f82843763246-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86ac6565-97e8-466f-9ba8-f82843763246-workload-certs\") pod \"86ac6565-97e8-466f-9ba8-f82843763246\" (UID: \"86ac6565-97e8-466f-9ba8-f82843763246\") " Oct 30 18:39:11.272640 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:11.272628 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86ac6565-97e8-466f-9ba8-f82843763246-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86ac6565-97e8-466f-9ba8-f82843763246-istio-podinfo\") pod \"86ac6565-97e8-466f-9ba8-f82843763246\" (UID: \"86ac6565-97e8-466f-9ba8-f82843763246\") " Oct 30 18:39:11.272822 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:11.272670 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86ac6565-97e8-466f-9ba8-f82843763246-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ac6565-97e8-466f-9ba8-f82843763246-workload-socket\") pod \"86ac6565-97e8-466f-9ba8-f82843763246\" (UID: \"86ac6565-97e8-466f-9ba8-f82843763246\") " Oct 30 18:39:11.272822 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:11.272693 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ac6565-97e8-466f-9ba8-f82843763246-credential-socket" pod "86ac6565-97e8-466f-9ba8-f82843763246" (UID: "86ac6565-97e8-466f-9ba8-f82843763246"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:39:11.272822 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:11.272711 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86ac6565-97e8-466f-9ba8-f82843763246-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86ac6565-97e8-466f-9ba8-f82843763246-istio-envoy\") pod \"86ac6565-97e8-466f-9ba8-f82843763246\" (UID: \"86ac6565-97e8-466f-9ba8-f82843763246\") " Oct 30 18:39:11.272822 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:11.272771 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86ac6565-97e8-466f-9ba8-f82843763246-kube-api-access-5klxh\" (UniqueName: \"kubernetes.io/projected/86ac6565-97e8-466f-9ba8-f82843763246-kube-api-access-5klxh\") pod \"86ac6565-97e8-466f-9ba8-f82843763246\" (UID: \"86ac6565-97e8-466f-9ba8-f82843763246\") " Oct 30 18:39:11.273048 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:11.272833 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86ac6565-97e8-466f-9ba8-f82843763246-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86ac6565-97e8-466f-9ba8-f82843763246-istio-data\") pod \"86ac6565-97e8-466f-9ba8-f82843763246\" (UID: \"86ac6565-97e8-466f-9ba8-f82843763246\") " Oct 30 18:39:11.273048 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:11.273014 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ac6565-97e8-466f-9ba8-f82843763246-workload-socket" pod "86ac6565-97e8-466f-9ba8-f82843763246" (UID: "86ac6565-97e8-466f-9ba8-f82843763246"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:39:11.273048 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:11.273021 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86ac6565-97e8-466f-9ba8-f82843763246-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86ac6565-97e8-466f-9ba8-f82843763246-istiod-ca-cert\") pod \"86ac6565-97e8-466f-9ba8-f82843763246\" (UID: \"86ac6565-97e8-466f-9ba8-f82843763246\") " Oct 30 18:39:11.273390 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:11.273364 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ac6565-97e8-466f-9ba8-f82843763246-workload-certs" pod "86ac6565-97e8-466f-9ba8-f82843763246" (UID: "86ac6565-97e8-466f-9ba8-f82843763246"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:39:11.273488 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:11.273392 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86ac6565-97e8-466f-9ba8-f82843763246-istiod-ca-cert" pod "86ac6565-97e8-466f-9ba8-f82843763246" (UID: "86ac6565-97e8-466f-9ba8-f82843763246"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:39:11.274060 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:11.274023 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ac6565-97e8-466f-9ba8-f82843763246-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:39:11.274060 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:11.274056 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86ac6565-97e8-466f-9ba8-f82843763246-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:39:11.274315 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:11.274071 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ac6565-97e8-466f-9ba8-f82843763246-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:39:11.274315 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:11.274085 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86ac6565-97e8-466f-9ba8-f82843763246-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:39:11.274315 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:11.274303 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ac6565-97e8-466f-9ba8-f82843763246-istio-data" pod "86ac6565-97e8-466f-9ba8-f82843763246" (UID: "86ac6565-97e8-466f-9ba8-f82843763246"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:39:11.275393 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:11.275368 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ac6565-97e8-466f-9ba8-f82843763246-istio-envoy" pod "86ac6565-97e8-466f-9ba8-f82843763246" (UID: "86ac6565-97e8-466f-9ba8-f82843763246"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:39:11.275499 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:11.275417 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ac6565-97e8-466f-9ba8-f82843763246-istio-token" pod "86ac6565-97e8-466f-9ba8-f82843763246" (UID: "86ac6565-97e8-466f-9ba8-f82843763246"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:39:11.275499 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:11.275480 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ac6565-97e8-466f-9ba8-f82843763246-kube-api-access-5klxh" pod "86ac6565-97e8-466f-9ba8-f82843763246" (UID: "86ac6565-97e8-466f-9ba8-f82843763246"). InnerVolumeSpecName "kube-api-access-5klxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:39:11.276103 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:11.276075 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86ac6565-97e8-466f-9ba8-f82843763246-istio-podinfo" pod "86ac6565-97e8-466f-9ba8-f82843763246" (UID: "86ac6565-97e8-466f-9ba8-f82843763246"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:39:11.375863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:11.375832 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86ac6565-97e8-466f-9ba8-f82843763246-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:39:11.375863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:11.375860 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86ac6565-97e8-466f-9ba8-f82843763246-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:39:11.375863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:11.375869 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86ac6565-97e8-466f-9ba8-f82843763246-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:39:11.376110 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:11.375878 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5klxh\" (UniqueName: \"kubernetes.io/projected/86ac6565-97e8-466f-9ba8-f82843763246-kube-api-access-5klxh\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:39:11.376110 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:11.375887 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86ac6565-97e8-466f-9ba8-f82843763246-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:39:11.502210 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:11.502175 4442 generic.go:356] "Generic (PLEG): container finished" podID="86ac6565-97e8-466f-9ba8-f82843763246" containerID="5894eef59ba11524541f2a14694ac3bd7d8c6aaefe60195f8b4d235b645ff724" exitCode=0 Oct 30 18:39:11.502389 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:11.502228 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh" Oct 30 18:39:11.502389 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:11.502265 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh" event={"ID":"86ac6565-97e8-466f-9ba8-f82843763246","Type":"ContainerDied","Data":"5894eef59ba11524541f2a14694ac3bd7d8c6aaefe60195f8b4d235b645ff724"} Oct 30 18:39:11.502389 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:11.502325 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh" event={"ID":"86ac6565-97e8-466f-9ba8-f82843763246","Type":"ContainerDied","Data":"6c1c66a2985f99cb33a2823174cbf6afb205b80d182b922e6a3e4ade4a5edc89"} Oct 30 18:39:11.502389 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:11.502358 4442 scope.go:122] "RemoveContainer" containerID="5894eef59ba11524541f2a14694ac3bd7d8c6aaefe60195f8b4d235b645ff724" Oct 30 18:39:11.514815 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:11.514796 4442 scope.go:122] "RemoveContainer" containerID="5894eef59ba11524541f2a14694ac3bd7d8c6aaefe60195f8b4d235b645ff724" Oct 30 18:39:11.515118 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:39:11.515090 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5894eef59ba11524541f2a14694ac3bd7d8c6aaefe60195f8b4d235b645ff724\": container with ID starting with 5894eef59ba11524541f2a14694ac3bd7d8c6aaefe60195f8b4d235b645ff724 not found: ID does not exist" containerID="5894eef59ba11524541f2a14694ac3bd7d8c6aaefe60195f8b4d235b645ff724" Oct 30 18:39:11.515208 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:11.515119 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5894eef59ba11524541f2a14694ac3bd7d8c6aaefe60195f8b4d235b645ff724"} err="failed to get container status \"5894eef59ba11524541f2a14694ac3bd7d8c6aaefe60195f8b4d235b645ff724\": rpc error: code = NotFound desc = could not find container \"5894eef59ba11524541f2a14694ac3bd7d8c6aaefe60195f8b4d235b645ff724\": container with ID starting with 5894eef59ba11524541f2a14694ac3bd7d8c6aaefe60195f8b4d235b645ff724 not found: ID does not exist" Oct 30 18:39:11.555880 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:11.555843 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh"] Oct 30 18:39:11.560423 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:11.560398 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-4c75d7-openshift-default-59bbf9cdb5-qrhzh"] Oct 30 18:39:13.070411 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:13.070374 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86ac6565-97e8-466f-9ba8-f82843763246" path="/var/lib/kubelet/pods/86ac6565-97e8-466f-9ba8-f82843763246/volumes" Oct 30 18:39:17.560839 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:39:17.560805 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:39:24.955039 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:24.954996 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm"] Oct 30 18:39:24.956552 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:24.956522 4442 cpu_manager.go:414] "containerMap: removing container" podUID="86ac6565-97e8-466f-9ba8-f82843763246" containerName="istio-proxy" Oct 30 18:39:24.956552 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:24.956555 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86ac6565-97e8-466f-9ba8-f82843763246" containerName="istio-proxy" Oct 30 18:39:24.956798 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:24.956782 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="86ac6565-97e8-466f-9ba8-f82843763246" containerName="istio-proxy" Oct 30 18:39:24.962199 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:24.962164 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" Oct 30 18:39:24.984552 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:24.984519 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-38e0db-openshift-default-dockercfg-pspfq\"" Oct 30 18:39:25.000761 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:25.000726 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm"] Oct 30 18:39:25.081785 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:25.081751 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvpmj\" (UniqueName: \"kubernetes.io/projected/01d882ef-fc94-45cb-af2e-3debf4b418cd-kube-api-access-jvpmj\") pod \"e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm\" (UID: \"01d882ef-fc94-45cb-af2e-3debf4b418cd\") " pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" Oct 30 18:39:25.082003 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:25.081959 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01d882ef-fc94-45cb-af2e-3debf4b418cd-istio-podinfo\") pod \"e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm\" (UID: \"01d882ef-fc94-45cb-af2e-3debf4b418cd\") " pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" Oct 30 18:39:25.082003 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:25.081995 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01d882ef-fc94-45cb-af2e-3debf4b418cd-istiod-ca-cert\") pod \"e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm\" (UID: \"01d882ef-fc94-45cb-af2e-3debf4b418cd\") " pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" Oct 30 18:39:25.082189 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:25.082071 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01d882ef-fc94-45cb-af2e-3debf4b418cd-istio-data\") pod \"e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm\" (UID: \"01d882ef-fc94-45cb-af2e-3debf4b418cd\") " pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" Oct 30 18:39:25.082189 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:25.082123 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01d882ef-fc94-45cb-af2e-3debf4b418cd-workload-socket\") pod \"e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm\" (UID: \"01d882ef-fc94-45cb-af2e-3debf4b418cd\") " pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" Oct 30 18:39:25.082318 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:25.082260 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01d882ef-fc94-45cb-af2e-3debf4b418cd-istio-token\") pod \"e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm\" (UID: \"01d882ef-fc94-45cb-af2e-3debf4b418cd\") " pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" Oct 30 18:39:25.082360 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:25.082331 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01d882ef-fc94-45cb-af2e-3debf4b418cd-credential-socket\") pod \"e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm\" (UID: \"01d882ef-fc94-45cb-af2e-3debf4b418cd\") " pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" Oct 30 18:39:25.082404 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:25.082372 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01d882ef-fc94-45cb-af2e-3debf4b418cd-istio-envoy\") pod \"e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm\" (UID: \"01d882ef-fc94-45cb-af2e-3debf4b418cd\") " pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" Oct 30 18:39:25.082493 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:25.082478 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01d882ef-fc94-45cb-af2e-3debf4b418cd-workload-certs\") pod \"e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm\" (UID: \"01d882ef-fc94-45cb-af2e-3debf4b418cd\") " pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" Oct 30 18:39:25.192192 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:25.188965 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01d882ef-fc94-45cb-af2e-3debf4b418cd-istio-envoy\") pod \"e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm\" (UID: \"01d882ef-fc94-45cb-af2e-3debf4b418cd\") " pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" Oct 30 18:39:25.192192 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:25.189035 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01d882ef-fc94-45cb-af2e-3debf4b418cd-workload-certs\") pod \"e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm\" (UID: \"01d882ef-fc94-45cb-af2e-3debf4b418cd\") " pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" Oct 30 18:39:25.192192 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:25.189111 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvpmj\" (UniqueName: \"kubernetes.io/projected/01d882ef-fc94-45cb-af2e-3debf4b418cd-kube-api-access-jvpmj\") pod \"e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm\" (UID: \"01d882ef-fc94-45cb-af2e-3debf4b418cd\") " pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" Oct 30 18:39:25.192192 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:25.189278 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01d882ef-fc94-45cb-af2e-3debf4b418cd-istio-podinfo\") pod \"e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm\" (UID: \"01d882ef-fc94-45cb-af2e-3debf4b418cd\") " pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" Oct 30 18:39:25.192192 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:25.189316 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01d882ef-fc94-45cb-af2e-3debf4b418cd-istiod-ca-cert\") pod \"e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm\" (UID: \"01d882ef-fc94-45cb-af2e-3debf4b418cd\") " pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" Oct 30 18:39:25.192192 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:25.189369 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01d882ef-fc94-45cb-af2e-3debf4b418cd-istio-data\") pod \"e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm\" (UID: \"01d882ef-fc94-45cb-af2e-3debf4b418cd\") " pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" Oct 30 18:39:25.192192 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:25.189403 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01d882ef-fc94-45cb-af2e-3debf4b418cd-workload-socket\") pod \"e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm\" (UID: \"01d882ef-fc94-45cb-af2e-3debf4b418cd\") " pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" Oct 30 18:39:25.192192 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:25.189450 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01d882ef-fc94-45cb-af2e-3debf4b418cd-istio-token\") pod \"e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm\" (UID: \"01d882ef-fc94-45cb-af2e-3debf4b418cd\") " pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" Oct 30 18:39:25.192192 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:25.189490 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01d882ef-fc94-45cb-af2e-3debf4b418cd-credential-socket\") pod \"e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm\" (UID: \"01d882ef-fc94-45cb-af2e-3debf4b418cd\") " pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" Oct 30 18:39:25.192192 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:25.189966 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01d882ef-fc94-45cb-af2e-3debf4b418cd-credential-socket\") pod \"e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm\" (UID: \"01d882ef-fc94-45cb-af2e-3debf4b418cd\") " pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" Oct 30 18:39:25.192192 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:25.192035 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01d882ef-fc94-45cb-af2e-3debf4b418cd-istiod-ca-cert\") pod \"e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm\" (UID: \"01d882ef-fc94-45cb-af2e-3debf4b418cd\") " pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" Oct 30 18:39:25.192954 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:25.192316 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01d882ef-fc94-45cb-af2e-3debf4b418cd-workload-certs\") pod \"e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm\" (UID: \"01d882ef-fc94-45cb-af2e-3debf4b418cd\") " pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" Oct 30 18:39:25.210215 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:25.201520 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01d882ef-fc94-45cb-af2e-3debf4b418cd-istio-envoy\") pod \"e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm\" (UID: \"01d882ef-fc94-45cb-af2e-3debf4b418cd\") " pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" Oct 30 18:39:25.210215 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:25.201816 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01d882ef-fc94-45cb-af2e-3debf4b418cd-istio-data\") pod \"e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm\" (UID: \"01d882ef-fc94-45cb-af2e-3debf4b418cd\") " pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" Oct 30 18:39:25.210215 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:25.202038 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01d882ef-fc94-45cb-af2e-3debf4b418cd-workload-socket\") pod \"e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm\" (UID: \"01d882ef-fc94-45cb-af2e-3debf4b418cd\") " pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" Oct 30 18:39:25.210215 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:25.205055 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01d882ef-fc94-45cb-af2e-3debf4b418cd-istio-podinfo\") pod \"e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm\" (UID: \"01d882ef-fc94-45cb-af2e-3debf4b418cd\") " pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" Oct 30 18:39:25.221314 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:25.221235 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvpmj\" (UniqueName: \"kubernetes.io/projected/01d882ef-fc94-45cb-af2e-3debf4b418cd-kube-api-access-jvpmj\") pod \"e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm\" (UID: \"01d882ef-fc94-45cb-af2e-3debf4b418cd\") " pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" Oct 30 18:39:25.226288 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:25.226253 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01d882ef-fc94-45cb-af2e-3debf4b418cd-istio-token\") pod \"e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm\" (UID: \"01d882ef-fc94-45cb-af2e-3debf4b418cd\") " pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" Oct 30 18:39:25.316101 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:25.316063 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" Oct 30 18:39:25.661121 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:25.661082 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm"] Oct 30 18:39:25.662335 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:25.662292 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:39:25.662462 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:25.662374 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:39:25.662462 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:25.662417 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:39:25.691769 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:25.691741 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" event={"ID":"01d882ef-fc94-45cb-af2e-3debf4b418cd","Type":"ContainerStarted","Data":"b3514419bf5716b45ef43d17ad06338bac35c07fb4bf25c0b81274c752ceab5a"} Oct 30 18:39:26.705758 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:26.705718 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" event={"ID":"01d882ef-fc94-45cb-af2e-3debf4b418cd","Type":"ContainerStarted","Data":"ae5bbef007da2c1f37049e61f54a0811e527ad4d84d8c4557a21382b25156174"} Oct 30 18:39:26.770325 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:26.770250 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" podStartSLOduration=2.770228582 podStartE2EDuration="2.770228582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:26.763483245 +0000 UTC m=+2463.853545585" watchObservedRunningTime="2026-10-30 18:39:26.770228582 +0000 UTC m=+2463.860290922" Oct 30 18:39:27.317332 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:27.317299 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" Oct 30 18:39:27.323537 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:27.323504 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" Oct 30 18:39:27.715258 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:27.715232 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" Oct 30 18:39:27.716501 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:27.716481 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" Oct 30 18:39:28.725026 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:28.724991 4442 generic.go:356] "Generic (PLEG): container finished" podID="425bc56e-e6f2-400d-a6dc-a277201acb8f" containerID="529b13f2ac933b33601c939e4f23541c0eb541e41c6d2fa2fefc6ff9bcc9924f" exitCode=0 Oct 30 18:39:28.725596 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:28.725074 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" event={"ID":"425bc56e-e6f2-400d-a6dc-a277201acb8f","Type":"ContainerDied","Data":"529b13f2ac933b33601c939e4f23541c0eb541e41c6d2fa2fefc6ff9bcc9924f"} Oct 30 18:39:28.725832 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:28.725814 4442 scope.go:122] "RemoveContainer" containerID="529b13f2ac933b33601c939e4f23541c0eb541e41c6d2fa2fefc6ff9bcc9924f" Oct 30 18:39:28.727364 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:28.727337 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:39:28.727467 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:28.727406 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:39:28.727467 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:28.727448 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:39:29.759971 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:29.759937 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" event={"ID":"425bc56e-e6f2-400d-a6dc-a277201acb8f","Type":"ContainerStarted","Data":"4f65edc89f7838267e2c63af4430fda749a05285dc4b8bfc9c0576835efb71bc"} Oct 30 18:39:30.253313 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:30.253269 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" Oct 30 18:39:30.255321 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:30.255294 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Oct 30 18:39:30.255478 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:30.255362 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" podUID="425bc56e-e6f2-400d-a6dc-a277201acb8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Oct 30 18:39:31.253159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:31.248421 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Oct 30 18:39:31.253159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:31.248480 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" podUID="425bc56e-e6f2-400d-a6dc-a277201acb8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Oct 30 18:39:31.268985 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:31.268952 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-38e0db-675d6bc647-sjkjz"] Oct 30 18:39:31.275156 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:31.273921 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-38e0db-675d6bc647-sjkjz"] Oct 30 18:39:31.275156 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:31.274050 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-38e0db-675d6bc647-sjkjz" Oct 30 18:39:31.333167 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:31.329126 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2e19b96-f72b-4735-a1e8-a0058dc45470-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-38e0db-675d6bc647-sjkjz\" (UID: \"a2e19b96-f72b-4735-a1e8-a0058dc45470\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-38e0db-675d6bc647-sjkjz" Oct 30 18:39:31.333167 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:31.329271 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfhxm\" (UniqueName: \"kubernetes.io/projected/a2e19b96-f72b-4735-a1e8-a0058dc45470-kube-api-access-cfhxm\") pod \"payload-pre-processing-e2e-extmod-b-38e0db-675d6bc647-sjkjz\" (UID: \"a2e19b96-f72b-4735-a1e8-a0058dc45470\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-38e0db-675d6bc647-sjkjz" Oct 30 18:39:31.557220 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:31.548959 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-38e0db\"" Oct 30 18:39:31.571261 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:31.560460 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2e19b96-f72b-4735-a1e8-a0058dc45470-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-38e0db-675d6bc647-sjkjz\" (UID: \"a2e19b96-f72b-4735-a1e8-a0058dc45470\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-38e0db-675d6bc647-sjkjz" Oct 30 18:39:31.571747 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:31.571699 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfhxm\" (UniqueName: \"kubernetes.io/projected/a2e19b96-f72b-4735-a1e8-a0058dc45470-kube-api-access-cfhxm\") pod \"payload-pre-processing-e2e-extmod-b-38e0db-675d6bc647-sjkjz\" (UID: \"a2e19b96-f72b-4735-a1e8-a0058dc45470\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-38e0db-675d6bc647-sjkjz" Oct 30 18:39:31.578661 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:31.578626 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2e19b96-f72b-4735-a1e8-a0058dc45470-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-38e0db-675d6bc647-sjkjz\" (UID: \"a2e19b96-f72b-4735-a1e8-a0058dc45470\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-38e0db-675d6bc647-sjkjz" Oct 30 18:39:31.605259 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:31.600555 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-38e0db-dockercfg-pp9tl\"" Oct 30 18:39:31.605259 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:31.600685 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfhxm\" (UniqueName: \"kubernetes.io/projected/a2e19b96-f72b-4735-a1e8-a0058dc45470-kube-api-access-cfhxm\") pod \"payload-pre-processing-e2e-extmod-b-38e0db-675d6bc647-sjkjz\" (UID: \"a2e19b96-f72b-4735-a1e8-a0058dc45470\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-38e0db-675d6bc647-sjkjz" Oct 30 18:39:31.622988 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:31.622965 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-38e0db-675d6bc647-sjkjz" Oct 30 18:39:31.956898 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:39:31.956861 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e19b96_f72b_4735_a1e8_a0058dc45470.slice/crio-dd10316c5ab967b0785b1cde308f94ec7c03bdb6c1d6efb829ac45ba202d17ad WatchSource:0}: Error finding container dd10316c5ab967b0785b1cde308f94ec7c03bdb6c1d6efb829ac45ba202d17ad: Status 404 returned error can't find the container with id dd10316c5ab967b0785b1cde308f94ec7c03bdb6c1d6efb829ac45ba202d17ad Oct 30 18:39:31.957106 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:31.957085 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-38e0db-675d6bc647-sjkjz"] Oct 30 18:39:32.252954 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:32.252543 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Oct 30 18:39:32.252954 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:32.252605 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" podUID="425bc56e-e6f2-400d-a6dc-a277201acb8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Oct 30 18:39:32.790058 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:32.790019 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-38e0db-675d6bc647-sjkjz" event={"ID":"a2e19b96-f72b-4735-a1e8-a0058dc45470","Type":"ContainerStarted","Data":"18cb2c09ca4d7d17c1662ddb1e62e690b6bf6c8d4c0192b68eab987aa360554f"} Oct 30 18:39:32.790478 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:32.790082 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-38e0db-675d6bc647-sjkjz" event={"ID":"a2e19b96-f72b-4735-a1e8-a0058dc45470","Type":"ContainerStarted","Data":"dd10316c5ab967b0785b1cde308f94ec7c03bdb6c1d6efb829ac45ba202d17ad"} Oct 30 18:39:32.790478 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:32.790160 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-38e0db-675d6bc647-sjkjz" Oct 30 18:39:32.852500 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:32.852438 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-38e0db-675d6bc647-sjkjz" podStartSLOduration=1.8524169499999998 podStartE2EDuration="1.85241695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:32.849450194 +0000 UTC m=+2469.939512537" watchObservedRunningTime="2026-10-30 18:39:32.85241695 +0000 UTC m=+2469.942479290" Oct 30 18:39:33.254535 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:33.254501 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Oct 30 18:39:33.254831 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:33.254802 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" podUID="425bc56e-e6f2-400d-a6dc-a277201acb8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Oct 30 18:39:33.567506 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:33.567414 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" Oct 30 18:39:34.249675 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:34.249627 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Oct 30 18:39:34.250236 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:34.249695 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" podUID="425bc56e-e6f2-400d-a6dc-a277201acb8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Oct 30 18:39:35.253902 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:35.253871 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" Oct 30 18:39:35.255652 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:35.255631 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" Oct 30 18:39:43.806961 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:43.806933 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-38e0db-675d6bc647-sjkjz" Oct 30 18:39:47.581536 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:39:47.581495 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:39:49.563226 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:49.563193 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn"] Oct 30 18:39:49.568008 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:49.567985 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" Oct 30 18:39:49.590626 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:49.590597 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn"] Oct 30 18:39:49.598114 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:49.598079 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-713dc8-openshift-default-dockercfg-qchzm\"" Oct 30 18:39:49.643689 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:49.643636 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-istiod-ca-cert\") pod \"e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn\" (UID: \"721069e6-60e5-4a5f-b79f-6d8b47ec87aa\") " pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" Oct 30 18:39:49.643689 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:49.643684 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-istio-podinfo\") pod \"e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn\" (UID: \"721069e6-60e5-4a5f-b79f-6d8b47ec87aa\") " pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" Oct 30 18:39:49.643976 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:49.643733 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-credential-socket\") pod \"e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn\" (UID: \"721069e6-60e5-4a5f-b79f-6d8b47ec87aa\") " pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" Oct 30 18:39:49.643976 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:49.643798 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-workload-certs\") pod \"e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn\" (UID: \"721069e6-60e5-4a5f-b79f-6d8b47ec87aa\") " pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" Oct 30 18:39:49.643976 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:49.643825 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt79h\" (UniqueName: \"kubernetes.io/projected/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-kube-api-access-tt79h\") pod \"e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn\" (UID: \"721069e6-60e5-4a5f-b79f-6d8b47ec87aa\") " pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" Oct 30 18:39:49.643976 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:49.643885 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-istio-token\") pod \"e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn\" (UID: \"721069e6-60e5-4a5f-b79f-6d8b47ec87aa\") " pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" Oct 30 18:39:49.643976 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:49.643927 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-workload-socket\") pod \"e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn\" (UID: \"721069e6-60e5-4a5f-b79f-6d8b47ec87aa\") " pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" Oct 30 18:39:49.644280 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:49.644014 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-istio-data\") pod \"e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn\" (UID: \"721069e6-60e5-4a5f-b79f-6d8b47ec87aa\") " pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" Oct 30 18:39:49.644280 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:49.644066 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-istio-envoy\") pod \"e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn\" (UID: \"721069e6-60e5-4a5f-b79f-6d8b47ec87aa\") " pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" Oct 30 18:39:49.746951 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:49.745836 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-istiod-ca-cert\") pod \"e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn\" (UID: \"721069e6-60e5-4a5f-b79f-6d8b47ec87aa\") " pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" Oct 30 18:39:49.746951 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:49.745896 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-istio-podinfo\") pod \"e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn\" (UID: \"721069e6-60e5-4a5f-b79f-6d8b47ec87aa\") " pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" Oct 30 18:39:49.746951 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:49.745947 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-credential-socket\") pod \"e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn\" (UID: \"721069e6-60e5-4a5f-b79f-6d8b47ec87aa\") " pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" Oct 30 18:39:49.746951 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:49.745989 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-workload-certs\") pod \"e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn\" (UID: \"721069e6-60e5-4a5f-b79f-6d8b47ec87aa\") " pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" Oct 30 18:39:49.746951 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:49.746019 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt79h\" (UniqueName: \"kubernetes.io/projected/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-kube-api-access-tt79h\") pod \"e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn\" (UID: \"721069e6-60e5-4a5f-b79f-6d8b47ec87aa\") " pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" Oct 30 18:39:49.746951 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:49.746088 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-istio-token\") pod \"e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn\" (UID: \"721069e6-60e5-4a5f-b79f-6d8b47ec87aa\") " pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" Oct 30 18:39:49.746951 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:49.746152 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-workload-socket\") pod \"e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn\" (UID: \"721069e6-60e5-4a5f-b79f-6d8b47ec87aa\") " pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" Oct 30 18:39:49.746951 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:49.746245 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-istio-data\") pod \"e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn\" (UID: \"721069e6-60e5-4a5f-b79f-6d8b47ec87aa\") " pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" Oct 30 18:39:49.746951 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:49.746300 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-istio-envoy\") pod \"e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn\" (UID: \"721069e6-60e5-4a5f-b79f-6d8b47ec87aa\") " pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" Oct 30 18:39:49.748156 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:49.748074 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-workload-certs\") pod \"e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn\" (UID: \"721069e6-60e5-4a5f-b79f-6d8b47ec87aa\") " pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" Oct 30 18:39:49.749159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:49.748912 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-istiod-ca-cert\") pod \"e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn\" (UID: \"721069e6-60e5-4a5f-b79f-6d8b47ec87aa\") " pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" Oct 30 18:39:49.751502 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:49.751433 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-istio-data\") pod \"e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn\" (UID: \"721069e6-60e5-4a5f-b79f-6d8b47ec87aa\") " pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" Oct 30 18:39:49.752005 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:49.751968 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-credential-socket\") pod \"e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn\" (UID: \"721069e6-60e5-4a5f-b79f-6d8b47ec87aa\") " pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" Oct 30 18:39:49.752438 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:49.752415 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-workload-socket\") pod \"e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn\" (UID: \"721069e6-60e5-4a5f-b79f-6d8b47ec87aa\") " pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" Oct 30 18:39:49.785973 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:49.785550 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-istio-envoy\") pod \"e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn\" (UID: \"721069e6-60e5-4a5f-b79f-6d8b47ec87aa\") " pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" Oct 30 18:39:49.802162 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:49.786834 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-istio-podinfo\") pod \"e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn\" (UID: \"721069e6-60e5-4a5f-b79f-6d8b47ec87aa\") " pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" Oct 30 18:39:49.802162 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:49.798967 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt79h\" (UniqueName: \"kubernetes.io/projected/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-kube-api-access-tt79h\") pod \"e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn\" (UID: \"721069e6-60e5-4a5f-b79f-6d8b47ec87aa\") " pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" Oct 30 18:39:49.836308 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:49.826646 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-istio-token\") pod \"e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn\" (UID: \"721069e6-60e5-4a5f-b79f-6d8b47ec87aa\") " pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" Oct 30 18:39:49.891224 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:49.890901 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" Oct 30 18:39:50.276390 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:39:50.276360 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod721069e6_60e5_4a5f_b79f_6d8b47ec87aa.slice/crio-60a0773e535849461bf30cdab8040be5bf73bc449734257a5988d23bb94680fb WatchSource:0}: Error finding container 60a0773e535849461bf30cdab8040be5bf73bc449734257a5988d23bb94680fb: Status 404 returned error can't find the container with id 60a0773e535849461bf30cdab8040be5bf73bc449734257a5988d23bb94680fb Oct 30 18:39:50.277000 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:50.276980 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn"] Oct 30 18:39:50.280062 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:50.280022 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:39:50.280169 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:50.280115 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:39:50.280220 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:50.280180 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:39:50.342983 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:50.342913 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg"] Oct 30 18:39:50.349245 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:50.349219 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" Oct 30 18:39:50.388272 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:50.388243 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg"] Oct 30 18:39:50.391455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:50.391424 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-861bb3-openshift-default-dockercfg-vns5c\"" Oct 30 18:39:50.468087 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:50.468048 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ae44a8a-6af3-482e-9407-33e148efded7-istio-data\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg\" (UID: \"3ae44a8a-6af3-482e-9407-33e148efded7\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" Oct 30 18:39:50.468514 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:50.468490 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ae44a8a-6af3-482e-9407-33e148efded7-istio-podinfo\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg\" (UID: \"3ae44a8a-6af3-482e-9407-33e148efded7\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" Oct 30 18:39:50.468698 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:50.468683 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ae44a8a-6af3-482e-9407-33e148efded7-istio-token\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg\" (UID: \"3ae44a8a-6af3-482e-9407-33e148efded7\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" Oct 30 18:39:50.468914 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:50.468897 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96vsx\" (UniqueName: \"kubernetes.io/projected/3ae44a8a-6af3-482e-9407-33e148efded7-kube-api-access-96vsx\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg\" (UID: \"3ae44a8a-6af3-482e-9407-33e148efded7\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" Oct 30 18:39:50.477404 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:50.477374 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ae44a8a-6af3-482e-9407-33e148efded7-istiod-ca-cert\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg\" (UID: \"3ae44a8a-6af3-482e-9407-33e148efded7\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" Oct 30 18:39:50.477655 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:50.477637 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ae44a8a-6af3-482e-9407-33e148efded7-istio-envoy\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg\" (UID: \"3ae44a8a-6af3-482e-9407-33e148efded7\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" Oct 30 18:39:50.477885 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:50.477869 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ae44a8a-6af3-482e-9407-33e148efded7-workload-certs\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg\" (UID: \"3ae44a8a-6af3-482e-9407-33e148efded7\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" Oct 30 18:39:50.478031 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:50.478017 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae44a8a-6af3-482e-9407-33e148efded7-workload-socket\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg\" (UID: \"3ae44a8a-6af3-482e-9407-33e148efded7\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" Oct 30 18:39:50.495088 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:50.495061 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae44a8a-6af3-482e-9407-33e148efded7-credential-socket\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg\" (UID: \"3ae44a8a-6af3-482e-9407-33e148efded7\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" Oct 30 18:39:50.624211 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:50.602295 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ae44a8a-6af3-482e-9407-33e148efded7-istiod-ca-cert\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg\" (UID: \"3ae44a8a-6af3-482e-9407-33e148efded7\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" Oct 30 18:39:50.624211 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:50.602362 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ae44a8a-6af3-482e-9407-33e148efded7-istio-envoy\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg\" (UID: \"3ae44a8a-6af3-482e-9407-33e148efded7\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" Oct 30 18:39:50.624211 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:50.602479 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ae44a8a-6af3-482e-9407-33e148efded7-workload-certs\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg\" (UID: \"3ae44a8a-6af3-482e-9407-33e148efded7\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" Oct 30 18:39:50.624211 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:50.602533 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae44a8a-6af3-482e-9407-33e148efded7-workload-socket\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg\" (UID: \"3ae44a8a-6af3-482e-9407-33e148efded7\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" Oct 30 18:39:50.624211 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:50.602568 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae44a8a-6af3-482e-9407-33e148efded7-credential-socket\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg\" (UID: \"3ae44a8a-6af3-482e-9407-33e148efded7\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" Oct 30 18:39:50.624211 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:50.602636 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ae44a8a-6af3-482e-9407-33e148efded7-istio-data\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg\" (UID: \"3ae44a8a-6af3-482e-9407-33e148efded7\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" Oct 30 18:39:50.624211 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:50.602679 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ae44a8a-6af3-482e-9407-33e148efded7-istio-podinfo\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg\" (UID: \"3ae44a8a-6af3-482e-9407-33e148efded7\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" Oct 30 18:39:50.624211 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:50.602706 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ae44a8a-6af3-482e-9407-33e148efded7-istio-token\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg\" (UID: \"3ae44a8a-6af3-482e-9407-33e148efded7\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" Oct 30 18:39:50.624211 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:50.602803 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96vsx\" (UniqueName: \"kubernetes.io/projected/3ae44a8a-6af3-482e-9407-33e148efded7-kube-api-access-96vsx\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg\" (UID: \"3ae44a8a-6af3-482e-9407-33e148efded7\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" Oct 30 18:39:50.624211 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:50.603966 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ae44a8a-6af3-482e-9407-33e148efded7-istiod-ca-cert\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg\" (UID: \"3ae44a8a-6af3-482e-9407-33e148efded7\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" Oct 30 18:39:50.624211 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:50.604620 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae44a8a-6af3-482e-9407-33e148efded7-credential-socket\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg\" (UID: \"3ae44a8a-6af3-482e-9407-33e148efded7\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" Oct 30 18:39:50.624211 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:50.604867 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ae44a8a-6af3-482e-9407-33e148efded7-workload-certs\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg\" (UID: \"3ae44a8a-6af3-482e-9407-33e148efded7\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" Oct 30 18:39:50.624211 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:50.605109 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae44a8a-6af3-482e-9407-33e148efded7-workload-socket\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg\" (UID: \"3ae44a8a-6af3-482e-9407-33e148efded7\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" Oct 30 18:39:50.624211 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:50.613018 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ae44a8a-6af3-482e-9407-33e148efded7-istio-data\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg\" (UID: \"3ae44a8a-6af3-482e-9407-33e148efded7\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" Oct 30 18:39:50.624211 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:50.619221 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ae44a8a-6af3-482e-9407-33e148efded7-istio-podinfo\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg\" (UID: \"3ae44a8a-6af3-482e-9407-33e148efded7\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" Oct 30 18:39:50.624211 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:50.619712 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ae44a8a-6af3-482e-9407-33e148efded7-istio-envoy\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg\" (UID: \"3ae44a8a-6af3-482e-9407-33e148efded7\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" Oct 30 18:39:50.632152 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:50.630807 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96vsx\" (UniqueName: \"kubernetes.io/projected/3ae44a8a-6af3-482e-9407-33e148efded7-kube-api-access-96vsx\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg\" (UID: \"3ae44a8a-6af3-482e-9407-33e148efded7\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" Oct 30 18:39:50.669810 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:50.669701 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ae44a8a-6af3-482e-9407-33e148efded7-istio-token\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg\" (UID: \"3ae44a8a-6af3-482e-9407-33e148efded7\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" Oct 30 18:39:50.972813 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:50.972704 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" Oct 30 18:39:51.043494 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:51.043460 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" event={"ID":"721069e6-60e5-4a5f-b79f-6d8b47ec87aa","Type":"ContainerStarted","Data":"3cf7b8b6065b5f84a4d079839b2ba4f271c173797f04751f9a02f4a64b42a53f"} Oct 30 18:39:51.043649 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:51.043556 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" event={"ID":"721069e6-60e5-4a5f-b79f-6d8b47ec87aa","Type":"ContainerStarted","Data":"60a0773e535849461bf30cdab8040be5bf73bc449734257a5988d23bb94680fb"} Oct 30 18:39:51.089685 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:51.089632 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" podStartSLOduration=2.089590989 podStartE2EDuration="2.089590989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:51.086977523 +0000 UTC m=+2488.177039861" watchObservedRunningTime="2026-10-30 18:39:51.089590989 +0000 UTC m=+2488.179653328" Oct 30 18:39:51.307620 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:39:51.307589 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ae44a8a_6af3_482e_9407_33e148efded7.slice/crio-19b67b3d10907e77ed1d5cefc78848a726c9f3bb42794da75e358e05dadf01dd WatchSource:0}: Error finding container 19b67b3d10907e77ed1d5cefc78848a726c9f3bb42794da75e358e05dadf01dd: Status 404 returned error can't find the container with id 19b67b3d10907e77ed1d5cefc78848a726c9f3bb42794da75e358e05dadf01dd Oct 30 18:39:51.308058 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:51.308037 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg"] Oct 30 18:39:51.311506 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:51.311471 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:39:51.311637 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:51.311563 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:39:51.311708 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:51.311635 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:39:51.893933 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:51.893835 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" Oct 30 18:39:51.901759 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:51.901725 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" Oct 30 18:39:52.055739 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:52.055701 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" event={"ID":"3ae44a8a-6af3-482e-9407-33e148efded7","Type":"ContainerStarted","Data":"c8d3693599f4d919d5892252c2fd81271195ac69684137b5d32182fb27471a81"} Oct 30 18:39:52.055949 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:52.055767 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" event={"ID":"3ae44a8a-6af3-482e-9407-33e148efded7","Type":"ContainerStarted","Data":"19b67b3d10907e77ed1d5cefc78848a726c9f3bb42794da75e358e05dadf01dd"} Oct 30 18:39:52.056279 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:52.056250 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" Oct 30 18:39:52.057319 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:52.057294 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" Oct 30 18:39:52.098532 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:52.098476 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" podStartSLOduration=2.098458292 podStartE2EDuration="2.098458292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:52.095073685 +0000 UTC m=+2489.185136070" watchObservedRunningTime="2026-10-30 18:39:52.098458292 +0000 UTC m=+2489.188520631" Oct 30 18:39:52.973954 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:52.973920 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" Oct 30 18:39:52.981766 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:52.981738 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" Oct 30 18:39:53.067018 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:53.066993 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" Oct 30 18:39:53.067220 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:53.067068 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" Oct 30 18:39:55.083570 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:55.083492 4442 generic.go:356] "Generic (PLEG): container finished" podID="b0040e62-2a9b-4af5-82bd-730b0f5aba4f" containerID="da4ca7073b149a7b14639f368ab7d085b7db275859c059436f4cffa352298e1f" exitCode=0 Oct 30 18:39:55.083570 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:55.083528 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" event={"ID":"b0040e62-2a9b-4af5-82bd-730b0f5aba4f","Type":"ContainerDied","Data":"da4ca7073b149a7b14639f368ab7d085b7db275859c059436f4cffa352298e1f"} Oct 30 18:39:55.084269 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:55.084250 4442 scope.go:122] "RemoveContainer" containerID="da4ca7073b149a7b14639f368ab7d085b7db275859c059436f4cffa352298e1f" Oct 30 18:39:55.087174 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:55.086360 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:39:55.087174 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:55.086463 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:39:55.087174 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:55.086514 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:39:56.094930 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:56.094899 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" event={"ID":"b0040e62-2a9b-4af5-82bd-730b0f5aba4f","Type":"ContainerStarted","Data":"e9bb720ca2ef9b43720554fb70468edcd32a224006209bb7433772ae4e552716"} Oct 30 18:39:57.083584 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:57.083554 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" Oct 30 18:39:57.085461 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:57.085428 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Oct 30 18:39:57.085602 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:57.085483 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" podUID="b0040e62-2a9b-4af5-82bd-730b0f5aba4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Oct 30 18:39:58.085531 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:58.085500 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Oct 30 18:39:58.086155 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:58.086093 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" podUID="b0040e62-2a9b-4af5-82bd-730b0f5aba4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Oct 30 18:39:59.083971 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:59.083939 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Oct 30 18:39:59.084185 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:59.083993 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" podUID="b0040e62-2a9b-4af5-82bd-730b0f5aba4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Oct 30 18:39:59.990247 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:39:59.990203 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" Oct 30 18:40:00.084723 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:00.084683 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Oct 30 18:40:00.084905 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:00.084748 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" podUID="b0040e62-2a9b-4af5-82bd-730b0f5aba4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Oct 30 18:40:01.084668 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:01.084624 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Oct 30 18:40:01.085216 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:01.084690 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" podUID="b0040e62-2a9b-4af5-82bd-730b0f5aba4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Oct 30 18:40:02.083779 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:02.083741 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Oct 30 18:40:02.083971 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:02.083797 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" podUID="b0040e62-2a9b-4af5-82bd-730b0f5aba4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Oct 30 18:40:03.089604 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:03.089574 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" Oct 30 18:40:03.090943 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:03.090918 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" Oct 30 18:40:10.962211 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:10.961592 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-8b8e54-77dc6fdfc4-j8v2w"] Oct 30 18:40:10.969800 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:10.969762 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-8b8e54-77dc6fdfc4-j8v2w" Oct 30 18:40:10.970844 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:10.969828 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-8b8e54-77dc6fdfc4-j8v2w"] Oct 30 18:40:10.988345 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:10.987405 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-8b8e54\"" Oct 30 18:40:10.988345 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:10.987548 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-8b8e54-dockercfg-spq4w\"" Oct 30 18:40:11.051659 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:11.051612 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p99lb\" (UniqueName: \"kubernetes.io/projected/58035120-0d0b-48c0-9096-c531d8b10f82-kube-api-access-p99lb\") pod \"payload-processing-e2e-extmod-b-8b8e54-77dc6fdfc4-j8v2w\" (UID: \"58035120-0d0b-48c0-9096-c531d8b10f82\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-8b8e54-77dc6fdfc4-j8v2w" Oct 30 18:40:11.051873 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:11.051852 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58035120-0d0b-48c0-9096-c531d8b10f82-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-8b8e54-77dc6fdfc4-j8v2w\" (UID: \"58035120-0d0b-48c0-9096-c531d8b10f82\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-8b8e54-77dc6fdfc4-j8v2w" Oct 30 18:40:11.153204 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:11.153161 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58035120-0d0b-48c0-9096-c531d8b10f82-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-8b8e54-77dc6fdfc4-j8v2w\" (UID: \"58035120-0d0b-48c0-9096-c531d8b10f82\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-8b8e54-77dc6fdfc4-j8v2w" Oct 30 18:40:11.153426 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:11.153408 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p99lb\" (UniqueName: \"kubernetes.io/projected/58035120-0d0b-48c0-9096-c531d8b10f82-kube-api-access-p99lb\") pod \"payload-processing-e2e-extmod-b-8b8e54-77dc6fdfc4-j8v2w\" (UID: \"58035120-0d0b-48c0-9096-c531d8b10f82\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-8b8e54-77dc6fdfc4-j8v2w" Oct 30 18:40:11.153980 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:11.153957 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58035120-0d0b-48c0-9096-c531d8b10f82-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-8b8e54-77dc6fdfc4-j8v2w\" (UID: \"58035120-0d0b-48c0-9096-c531d8b10f82\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-8b8e54-77dc6fdfc4-j8v2w" Oct 30 18:40:11.173109 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:11.173074 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p99lb\" (UniqueName: \"kubernetes.io/projected/58035120-0d0b-48c0-9096-c531d8b10f82-kube-api-access-p99lb\") pod \"payload-processing-e2e-extmod-b-8b8e54-77dc6fdfc4-j8v2w\" (UID: \"58035120-0d0b-48c0-9096-c531d8b10f82\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-8b8e54-77dc6fdfc4-j8v2w" Oct 30 18:40:11.321963 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:11.321871 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-8b8e54-77dc6fdfc4-j8v2w" Oct 30 18:40:11.630382 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:11.630352 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-8b8e54-77dc6fdfc4-j8v2w"] Oct 30 18:40:12.497273 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:12.497123 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-8b8e54-77dc6fdfc4-j8v2w" event={"ID":"58035120-0d0b-48c0-9096-c531d8b10f82","Type":"ContainerStarted","Data":"a4ad388a3031f8871663c467537bb09ad22349a65b035c4109a04828a9598ea4"} Oct 30 18:40:12.497273 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:12.497237 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-8b8e54-77dc6fdfc4-j8v2w" event={"ID":"58035120-0d0b-48c0-9096-c531d8b10f82","Type":"ContainerStarted","Data":"b4bf0c355707486f1cd688921f426a9afcc81528ee8e64fc0a507454ec469ffb"} Oct 30 18:40:12.497840 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:12.497284 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-8b8e54-77dc6fdfc4-j8v2w" Oct 30 18:40:12.535948 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:12.535887 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-8b8e54-77dc6fdfc4-j8v2w" podStartSLOduration=2.5358708290000003 podStartE2EDuration="2.535870829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:40:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:40:12.534806417 +0000 UTC m=+2509.624868758" watchObservedRunningTime="2026-10-30 18:40:12.535870829 +0000 UTC m=+2509.625933506" Oct 30 18:40:15.091754 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:15.090690 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": read tcp 10.134.0.2:60368->10.134.0.231:15021: read: connection reset by peer" start-of-body= Oct 30 18:40:15.091754 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:15.090751 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" podUID="dd0f5b27-6383-4d55-a840-3938d11d9f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": read tcp 10.134.0.2:60368->10.134.0.231:15021: read: connection reset by peer" Oct 30 18:40:15.636181 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:15.635446 4442 generic.go:356] "Generic (PLEG): container finished" podID="dd0f5b27-6383-4d55-a840-3938d11d9f6f" containerID="12e03a772cb15bc6cd60d5687c642999f0338add7027485fe6b82477e3cfc317" exitCode=0 Oct 30 18:40:15.636181 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:15.635727 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" event={"ID":"dd0f5b27-6383-4d55-a840-3938d11d9f6f","Type":"ContainerDied","Data":"12e03a772cb15bc6cd60d5687c642999f0338add7027485fe6b82477e3cfc317"} Oct 30 18:40:15.636516 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:15.636413 4442 scope.go:122] "RemoveContainer" containerID="12e03a772cb15bc6cd60d5687c642999f0338add7027485fe6b82477e3cfc317" Oct 30 18:40:15.660160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:15.656667 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:40:15.660160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:15.656769 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:40:15.660160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:15.656822 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:40:16.662180 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:16.662105 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" event={"ID":"dd0f5b27-6383-4d55-a840-3938d11d9f6f","Type":"ContainerStarted","Data":"f954150b0c9ed044391ce6a5e6a8a8c56b3fc36d4dc080f184f31b80d8777e35"} Oct 30 18:40:17.517702 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:17.517668 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" Oct 30 18:40:17.517901 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:17.517864 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body= Oct 30 18:40:17.517951 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:17.517918 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" podUID="dd0f5b27-6383-4d55-a840-3938d11d9f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" Oct 30 18:40:17.588170 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:40:17.584536 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:40:18.517919 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:18.517884 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body= Oct 30 18:40:18.518354 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:18.517938 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" podUID="dd0f5b27-6383-4d55-a840-3938d11d9f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" Oct 30 18:40:19.517750 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:19.517717 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body= Oct 30 18:40:19.517952 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:19.517769 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" podUID="dd0f5b27-6383-4d55-a840-3938d11d9f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" Oct 30 18:40:20.517609 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:20.517577 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body= Oct 30 18:40:20.517816 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:20.517630 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" podUID="dd0f5b27-6383-4d55-a840-3938d11d9f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" Oct 30 18:40:21.517670 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:21.517637 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body= Oct 30 18:40:21.518230 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:21.517693 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" podUID="dd0f5b27-6383-4d55-a840-3938d11d9f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" Oct 30 18:40:22.521674 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:22.521641 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" Oct 30 18:40:22.522158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:22.522127 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" Oct 30 18:40:22.522820 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:22.522805 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" Oct 30 18:40:23.510319 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:23.510292 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-8b8e54-77dc6fdfc4-j8v2w" Oct 30 18:40:26.165892 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:26.165855 4442 scope.go:122] "RemoveContainer" containerID="d6adaa530df56a7cc17d81cfc919924e8d1513be9ca58a7877737f9ac9b5b075" Oct 30 18:40:28.678363 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:28.678324 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm"] Oct 30 18:40:28.685451 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:28.685425 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" Oct 30 18:40:28.712257 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:28.712229 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-6babca-openshift-default-dockercfg-vwzl7\"" Oct 30 18:40:28.713358 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:28.713334 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm"] Oct 30 18:40:28.785204 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:28.785173 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68fhp\" (UniqueName: \"kubernetes.io/projected/57bae3b7-f320-45e8-bdff-e41abbb9f20c-kube-api-access-68fhp\") pod \"e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm\" (UID: \"57bae3b7-f320-45e8-bdff-e41abbb9f20c\") " pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" Oct 30 18:40:28.785386 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:28.785215 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57bae3b7-f320-45e8-bdff-e41abbb9f20c-istio-data\") pod \"e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm\" (UID: \"57bae3b7-f320-45e8-bdff-e41abbb9f20c\") " pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" Oct 30 18:40:28.785386 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:28.785335 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bae3b7-f320-45e8-bdff-e41abbb9f20c-workload-socket\") pod \"e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm\" (UID: \"57bae3b7-f320-45e8-bdff-e41abbb9f20c\") " pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" Oct 30 18:40:28.785469 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:28.785399 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57bae3b7-f320-45e8-bdff-e41abbb9f20c-istio-envoy\") pod \"e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm\" (UID: \"57bae3b7-f320-45e8-bdff-e41abbb9f20c\") " pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" Oct 30 18:40:28.785505 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:28.785479 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bae3b7-f320-45e8-bdff-e41abbb9f20c-credential-socket\") pod \"e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm\" (UID: \"57bae3b7-f320-45e8-bdff-e41abbb9f20c\") " pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" Oct 30 18:40:28.785586 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:28.785573 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57bae3b7-f320-45e8-bdff-e41abbb9f20c-istio-token\") pod \"e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm\" (UID: \"57bae3b7-f320-45e8-bdff-e41abbb9f20c\") " pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" Oct 30 18:40:28.785635 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:28.785624 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57bae3b7-f320-45e8-bdff-e41abbb9f20c-istio-podinfo\") pod \"e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm\" (UID: \"57bae3b7-f320-45e8-bdff-e41abbb9f20c\") " pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" Oct 30 18:40:28.785677 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:28.785660 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57bae3b7-f320-45e8-bdff-e41abbb9f20c-workload-certs\") pod \"e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm\" (UID: \"57bae3b7-f320-45e8-bdff-e41abbb9f20c\") " pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" Oct 30 18:40:28.785742 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:28.785727 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57bae3b7-f320-45e8-bdff-e41abbb9f20c-istiod-ca-cert\") pod \"e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm\" (UID: \"57bae3b7-f320-45e8-bdff-e41abbb9f20c\") " pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" Oct 30 18:40:28.914157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:28.912472 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68fhp\" (UniqueName: \"kubernetes.io/projected/57bae3b7-f320-45e8-bdff-e41abbb9f20c-kube-api-access-68fhp\") pod \"e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm\" (UID: \"57bae3b7-f320-45e8-bdff-e41abbb9f20c\") " pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" Oct 30 18:40:28.914157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:28.912539 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57bae3b7-f320-45e8-bdff-e41abbb9f20c-istio-data\") pod \"e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm\" (UID: \"57bae3b7-f320-45e8-bdff-e41abbb9f20c\") " pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" Oct 30 18:40:28.914157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:28.912640 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bae3b7-f320-45e8-bdff-e41abbb9f20c-workload-socket\") pod \"e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm\" (UID: \"57bae3b7-f320-45e8-bdff-e41abbb9f20c\") " pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" Oct 30 18:40:28.914157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:28.912735 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57bae3b7-f320-45e8-bdff-e41abbb9f20c-istio-envoy\") pod \"e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm\" (UID: \"57bae3b7-f320-45e8-bdff-e41abbb9f20c\") " pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" Oct 30 18:40:28.914157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:28.912802 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bae3b7-f320-45e8-bdff-e41abbb9f20c-credential-socket\") pod \"e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm\" (UID: \"57bae3b7-f320-45e8-bdff-e41abbb9f20c\") " pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" Oct 30 18:40:28.914157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:28.912899 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57bae3b7-f320-45e8-bdff-e41abbb9f20c-istio-token\") pod \"e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm\" (UID: \"57bae3b7-f320-45e8-bdff-e41abbb9f20c\") " pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" Oct 30 18:40:28.914157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:28.912973 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57bae3b7-f320-45e8-bdff-e41abbb9f20c-istio-podinfo\") pod \"e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm\" (UID: \"57bae3b7-f320-45e8-bdff-e41abbb9f20c\") " pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" Oct 30 18:40:28.914157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:28.913033 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57bae3b7-f320-45e8-bdff-e41abbb9f20c-workload-certs\") pod \"e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm\" (UID: \"57bae3b7-f320-45e8-bdff-e41abbb9f20c\") " pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" Oct 30 18:40:28.914157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:28.913085 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57bae3b7-f320-45e8-bdff-e41abbb9f20c-istiod-ca-cert\") pod \"e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm\" (UID: \"57bae3b7-f320-45e8-bdff-e41abbb9f20c\") " pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" Oct 30 18:40:28.914157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:28.914064 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57bae3b7-f320-45e8-bdff-e41abbb9f20c-istiod-ca-cert\") pod \"e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm\" (UID: \"57bae3b7-f320-45e8-bdff-e41abbb9f20c\") " pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" Oct 30 18:40:28.914880 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:28.914674 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57bae3b7-f320-45e8-bdff-e41abbb9f20c-istio-data\") pod \"e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm\" (UID: \"57bae3b7-f320-45e8-bdff-e41abbb9f20c\") " pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" Oct 30 18:40:28.914950 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:28.914918 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bae3b7-f320-45e8-bdff-e41abbb9f20c-workload-socket\") pod \"e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm\" (UID: \"57bae3b7-f320-45e8-bdff-e41abbb9f20c\") " pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" Oct 30 18:40:28.938266 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:28.937468 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bae3b7-f320-45e8-bdff-e41abbb9f20c-credential-socket\") pod \"e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm\" (UID: \"57bae3b7-f320-45e8-bdff-e41abbb9f20c\") " pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" Oct 30 18:40:28.938266 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:28.938247 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57bae3b7-f320-45e8-bdff-e41abbb9f20c-workload-certs\") pod \"e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm\" (UID: \"57bae3b7-f320-45e8-bdff-e41abbb9f20c\") " pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" Oct 30 18:40:28.952103 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:28.952035 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68fhp\" (UniqueName: \"kubernetes.io/projected/57bae3b7-f320-45e8-bdff-e41abbb9f20c-kube-api-access-68fhp\") pod \"e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm\" (UID: \"57bae3b7-f320-45e8-bdff-e41abbb9f20c\") " pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" Oct 30 18:40:28.964546 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:28.961738 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57bae3b7-f320-45e8-bdff-e41abbb9f20c-istio-podinfo\") pod \"e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm\" (UID: \"57bae3b7-f320-45e8-bdff-e41abbb9f20c\") " pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" Oct 30 18:40:28.964546 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:28.964463 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57bae3b7-f320-45e8-bdff-e41abbb9f20c-istio-envoy\") pod \"e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm\" (UID: \"57bae3b7-f320-45e8-bdff-e41abbb9f20c\") " pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" Oct 30 18:40:28.993218 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:28.969434 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57bae3b7-f320-45e8-bdff-e41abbb9f20c-istio-token\") pod \"e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm\" (UID: \"57bae3b7-f320-45e8-bdff-e41abbb9f20c\") " pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" Oct 30 18:40:29.005659 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:29.005400 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" Oct 30 18:40:29.537251 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:40:29.537224 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57bae3b7_f320_45e8_bdff_e41abbb9f20c.slice/crio-f51a08592c66532ae445ae23464680004a7c713471644ffcc7cdba335e15e7eb WatchSource:0}: Error finding container f51a08592c66532ae445ae23464680004a7c713471644ffcc7cdba335e15e7eb: Status 404 returned error can't find the container with id f51a08592c66532ae445ae23464680004a7c713471644ffcc7cdba335e15e7eb Oct 30 18:40:29.539407 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:29.539380 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm"] Oct 30 18:40:29.540826 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:29.540793 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:40:29.540961 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:29.540871 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:40:29.540961 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:29.540920 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:40:29.781967 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:29.781929 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" event={"ID":"57bae3b7-f320-45e8-bdff-e41abbb9f20c","Type":"ContainerStarted","Data":"6d0627b0ed1bf6e7a5d4a27d1ff0aa10bae33940ce515c1bd38a78b1a8466a13"} Oct 30 18:40:29.782475 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:29.781995 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" event={"ID":"57bae3b7-f320-45e8-bdff-e41abbb9f20c","Type":"ContainerStarted","Data":"f51a08592c66532ae445ae23464680004a7c713471644ffcc7cdba335e15e7eb"} Oct 30 18:40:29.866724 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:29.866671 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" podStartSLOduration=1.866657164 podStartE2EDuration="1.866657164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:40:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:40:29.864720774 +0000 UTC m=+2526.954783126" watchObservedRunningTime="2026-10-30 18:40:29.866657164 +0000 UTC m=+2526.956719504" Oct 30 18:40:30.017160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:30.011834 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" Oct 30 18:40:30.023167 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:30.017585 4442 patch_prober.go:29] interesting pod/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 30 18:40:30.023167 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:30.017646 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" podUID="57bae3b7-f320-45e8-bdff-e41abbb9f20c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 30 18:40:31.014186 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:31.014157 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" Oct 30 18:40:31.800579 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:31.800507 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" Oct 30 18:40:31.801657 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:31.801632 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" Oct 30 18:40:43.757287 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:43.757246 4442 patch_prober.go:29] interesting pod/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:40:43.759626 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:43.757319 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" podUID="01d882ef-fc94-45cb-af2e-3debf4b418cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:40:44.033977 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:44.033832 4442 generic.go:356] "Generic (PLEG): container finished" podID="01d882ef-fc94-45cb-af2e-3debf4b418cd" containerID="ae5bbef007da2c1f37049e61f54a0811e527ad4d84d8c4557a21382b25156174" exitCode=0 Oct 30 18:40:44.033977 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:44.033907 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" event={"ID":"01d882ef-fc94-45cb-af2e-3debf4b418cd","Type":"ContainerDied","Data":"ae5bbef007da2c1f37049e61f54a0811e527ad4d84d8c4557a21382b25156174"} Oct 30 18:40:44.034531 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:44.034507 4442 scope.go:122] "RemoveContainer" containerID="ae5bbef007da2c1f37049e61f54a0811e527ad4d84d8c4557a21382b25156174" Oct 30 18:40:44.036917 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:44.036881 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:40:44.037165 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:44.036975 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:40:44.037165 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:44.037019 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:40:44.120192 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:44.120160 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6babca-6dc6645cb4-jw8l5"] Oct 30 18:40:44.126377 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:44.126354 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-6babca-6dc6645cb4-jw8l5" Oct 30 18:40:44.149510 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:44.149479 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-6babca\"" Oct 30 18:40:44.168230 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:44.167875 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6babca-6dc6645cb4-jw8l5"] Oct 30 18:40:44.170195 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:44.170121 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9qpt\" (UniqueName: \"kubernetes.io/projected/0c64d72e-b778-40b9-aeb7-6b7cfad39907-kube-api-access-s9qpt\") pod \"payload-processing-e2e-extmod-b-6babca-6dc6645cb4-jw8l5\" (UID: \"0c64d72e-b778-40b9-aeb7-6b7cfad39907\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6babca-6dc6645cb4-jw8l5" Oct 30 18:40:44.170323 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:44.170242 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c64d72e-b778-40b9-aeb7-6b7cfad39907-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-6babca-6dc6645cb4-jw8l5\" (UID: \"0c64d72e-b778-40b9-aeb7-6b7cfad39907\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6babca-6dc6645cb4-jw8l5" Oct 30 18:40:44.322239 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:44.281125 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9qpt\" (UniqueName: \"kubernetes.io/projected/0c64d72e-b778-40b9-aeb7-6b7cfad39907-kube-api-access-s9qpt\") pod \"payload-processing-e2e-extmod-b-6babca-6dc6645cb4-jw8l5\" (UID: \"0c64d72e-b778-40b9-aeb7-6b7cfad39907\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6babca-6dc6645cb4-jw8l5" Oct 30 18:40:44.322239 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:44.281226 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c64d72e-b778-40b9-aeb7-6b7cfad39907-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-6babca-6dc6645cb4-jw8l5\" (UID: \"0c64d72e-b778-40b9-aeb7-6b7cfad39907\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6babca-6dc6645cb4-jw8l5" Oct 30 18:40:44.322239 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:44.282614 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c64d72e-b778-40b9-aeb7-6b7cfad39907-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-6babca-6dc6645cb4-jw8l5\" (UID: \"0c64d72e-b778-40b9-aeb7-6b7cfad39907\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6babca-6dc6645cb4-jw8l5" Oct 30 18:40:44.435159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:44.434647 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9qpt\" (UniqueName: \"kubernetes.io/projected/0c64d72e-b778-40b9-aeb7-6b7cfad39907-kube-api-access-s9qpt\") pod \"payload-processing-e2e-extmod-b-6babca-6dc6645cb4-jw8l5\" (UID: \"0c64d72e-b778-40b9-aeb7-6b7cfad39907\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6babca-6dc6645cb4-jw8l5" Oct 30 18:40:44.517158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:44.516684 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-6babca-6dc6645cb4-jw8l5" Oct 30 18:40:44.907707 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:44.907666 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6babca-6dc6645cb4-jw8l5"] Oct 30 18:40:45.048007 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:45.047971 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-6babca-6dc6645cb4-jw8l5" event={"ID":"0c64d72e-b778-40b9-aeb7-6b7cfad39907","Type":"ContainerStarted","Data":"8c9c3cdd8114253edf129d148611d17da392b74056ec048786ac0ea25e4b49b0"} Oct 30 18:40:45.048226 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:45.048025 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-6babca-6dc6645cb4-jw8l5" event={"ID":"0c64d72e-b778-40b9-aeb7-6b7cfad39907","Type":"ContainerStarted","Data":"410520d2e903439b28151a26e9dd24f27e27514d8e887d8ae4d070b5789ab7a9"} Oct 30 18:40:45.048226 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:45.048176 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-6babca-6dc6645cb4-jw8l5" Oct 30 18:40:45.050889 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:45.050858 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" event={"ID":"01d882ef-fc94-45cb-af2e-3debf4b418cd","Type":"ContainerStarted","Data":"03d19c61cf3557a6f3f873e1089d17461ff4270865d804cdded18e8df6030485"} Oct 30 18:40:45.089097 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:45.089034 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-6babca-6dc6645cb4-jw8l5" podStartSLOduration=1.089015873 podStartE2EDuration="1.089015873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:40:45.086815506 +0000 UTC m=+2542.176877844" watchObservedRunningTime="2026-10-30 18:40:45.089015873 +0000 UTC m=+2542.179078211" Oct 30 18:40:45.317580 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:45.317488 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" Oct 30 18:40:45.317841 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:45.317819 4442 patch_prober.go:29] interesting pod/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 30 18:40:45.317898 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:45.317880 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" podUID="01d882ef-fc94-45cb-af2e-3debf4b418cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 30 18:40:46.316930 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:46.316895 4442 patch_prober.go:29] interesting pod/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 30 18:40:46.317371 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:46.316955 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" podUID="01d882ef-fc94-45cb-af2e-3debf4b418cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 30 18:40:47.317033 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:47.317002 4442 patch_prober.go:29] interesting pod/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 30 18:40:47.317500 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:47.317051 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" podUID="01d882ef-fc94-45cb-af2e-3debf4b418cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 30 18:40:47.584423 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:40:47.584341 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:40:48.316992 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:48.316953 4442 patch_prober.go:29] interesting pod/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 30 18:40:48.317247 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:48.317010 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" podUID="01d882ef-fc94-45cb-af2e-3debf4b418cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 30 18:40:49.317380 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:49.317341 4442 patch_prober.go:29] interesting pod/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 30 18:40:49.317789 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:49.317397 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" podUID="01d882ef-fc94-45cb-af2e-3debf4b418cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 30 18:40:50.318187 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:50.318151 4442 patch_prober.go:29] interesting pod/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 30 18:40:50.318812 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:50.318779 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" podUID="01d882ef-fc94-45cb-af2e-3debf4b418cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 30 18:40:51.317787 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:51.317751 4442 patch_prober.go:29] interesting pod/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 30 18:40:51.317982 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:51.317806 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" podUID="01d882ef-fc94-45cb-af2e-3debf4b418cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 30 18:40:52.321311 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:52.321278 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" Oct 30 18:40:52.321802 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:52.321782 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" Oct 30 18:40:52.322701 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:52.322680 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" Oct 30 18:40:56.095654 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:40:56.094330 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-6babca-6dc6645cb4-jw8l5" Oct 30 18:41:03.279830 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:03.279749 4442 generic.go:356] "Generic (PLEG): container finished" podID="3ae44a8a-6af3-482e-9407-33e148efded7" containerID="c8d3693599f4d919d5892252c2fd81271195ac69684137b5d32182fb27471a81" exitCode=0 Oct 30 18:41:03.280333 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:03.279821 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" event={"ID":"3ae44a8a-6af3-482e-9407-33e148efded7","Type":"ContainerDied","Data":"c8d3693599f4d919d5892252c2fd81271195ac69684137b5d32182fb27471a81"} Oct 30 18:41:03.280333 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:03.280299 4442 scope.go:122] "RemoveContainer" containerID="c8d3693599f4d919d5892252c2fd81271195ac69684137b5d32182fb27471a81" Oct 30 18:41:03.282105 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:03.282082 4442 generic.go:356] "Generic (PLEG): container finished" podID="721069e6-60e5-4a5f-b79f-6d8b47ec87aa" containerID="3cf7b8b6065b5f84a4d079839b2ba4f271c173797f04751f9a02f4a64b42a53f" exitCode=0 Oct 30 18:41:03.282228 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:03.282185 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" event={"ID":"721069e6-60e5-4a5f-b79f-6d8b47ec87aa","Type":"ContainerDied","Data":"3cf7b8b6065b5f84a4d079839b2ba4f271c173797f04751f9a02f4a64b42a53f"} Oct 30 18:41:03.282342 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:03.282314 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:41:03.282415 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:03.282398 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:41:03.282474 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:03.282458 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:41:03.282657 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:03.282642 4442 scope.go:122] "RemoveContainer" containerID="3cf7b8b6065b5f84a4d079839b2ba4f271c173797f04751f9a02f4a64b42a53f" Oct 30 18:41:03.284326 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:03.284301 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:41:03.284417 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:03.284394 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:41:03.284459 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:03.284429 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:41:04.292544 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:04.292500 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" event={"ID":"3ae44a8a-6af3-482e-9407-33e148efded7","Type":"ContainerStarted","Data":"06a1e4039003f2f75279d7af0e79d0f43fe502b537a67af0009539f1b5fb6be6"} Oct 30 18:41:04.294622 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:04.294595 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" event={"ID":"721069e6-60e5-4a5f-b79f-6d8b47ec87aa","Type":"ContainerStarted","Data":"cda316d7beade1b70bd6f0ae79f0b95ad8153f3eb277dfd0f0f456af5b1c1164"} Oct 30 18:41:04.893532 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:04.893478 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" Oct 30 18:41:04.894945 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:04.894913 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 18:41:04.895086 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:04.894974 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" podUID="721069e6-60e5-4a5f-b79f-6d8b47ec87aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 18:41:04.981159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:04.977150 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" Oct 30 18:41:04.981159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:04.977299 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Oct 30 18:41:04.981159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:04.977343 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" podUID="3ae44a8a-6af3-482e-9407-33e148efded7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Oct 30 18:41:05.895204 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:05.895111 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 18:41:05.895748 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:05.895212 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" podUID="721069e6-60e5-4a5f-b79f-6d8b47ec87aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 18:41:05.975159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:05.974615 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Oct 30 18:41:05.975159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:05.974679 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" podUID="3ae44a8a-6af3-482e-9407-33e148efded7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Oct 30 18:41:06.896203 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:06.895550 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 18:41:06.896203 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:06.895608 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" podUID="721069e6-60e5-4a5f-b79f-6d8b47ec87aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 18:41:06.981643 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:06.981595 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Oct 30 18:41:06.981868 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:06.981664 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" podUID="3ae44a8a-6af3-482e-9407-33e148efded7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Oct 30 18:41:07.090198 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:07.089019 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" Oct 30 18:41:07.896924 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:07.896889 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 18:41:07.897535 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:07.897502 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" podUID="721069e6-60e5-4a5f-b79f-6d8b47ec87aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 18:41:07.978194 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:07.976720 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Oct 30 18:41:07.978194 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:07.976794 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" podUID="3ae44a8a-6af3-482e-9407-33e148efded7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Oct 30 18:41:08.080159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:08.077356 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" Oct 30 18:41:08.893948 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:08.893908 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 18:41:08.894143 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:08.893972 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" podUID="721069e6-60e5-4a5f-b79f-6d8b47ec87aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 18:41:08.975814 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:08.975776 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Oct 30 18:41:08.976521 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:08.976487 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" podUID="3ae44a8a-6af3-482e-9407-33e148efded7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Oct 30 18:41:09.894682 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:09.894648 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 18:41:09.894941 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:09.894918 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" podUID="721069e6-60e5-4a5f-b79f-6d8b47ec87aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 18:41:09.977012 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:09.976549 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Oct 30 18:41:09.977012 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:09.976625 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" podUID="3ae44a8a-6af3-482e-9407-33e148efded7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Oct 30 18:41:10.913671 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:10.913641 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" Oct 30 18:41:10.916178 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:10.916114 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" Oct 30 18:41:10.979300 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:10.979271 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" Oct 30 18:41:10.980338 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:10.980318 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" Oct 30 18:41:17.563498 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:41:17.563459 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:41:44.473413 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:44.473375 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6018b4-6868d8bb99-r5frb"] Oct 30 18:41:44.477927 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:44.477904 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6018b4-6868d8bb99-r5frb" Oct 30 18:41:44.495827 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:44.495798 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6018b4-6868d8bb99-r5frb"] Oct 30 18:41:44.501772 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:44.501734 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-6018b4\"" Oct 30 18:41:44.613597 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:44.613515 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-6018b4-dockercfg-r77lp\"" Oct 30 18:41:44.646194 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:44.615123 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6601257-a86e-414c-9b71-9dc6042b6818-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-6018b4-6868d8bb99-r5frb\" (UID: \"d6601257-a86e-414c-9b71-9dc6042b6818\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6018b4-6868d8bb99-r5frb" Oct 30 18:41:44.646913 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:44.646546 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b56jm\" (UniqueName: \"kubernetes.io/projected/d6601257-a86e-414c-9b71-9dc6042b6818-kube-api-access-b56jm\") pod \"payload-pre-processing-e2e-extmod-b-6018b4-6868d8bb99-r5frb\" (UID: \"d6601257-a86e-414c-9b71-9dc6042b6818\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6018b4-6868d8bb99-r5frb" Oct 30 18:41:44.755165 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:44.755063 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b56jm\" (UniqueName: \"kubernetes.io/projected/d6601257-a86e-414c-9b71-9dc6042b6818-kube-api-access-b56jm\") pod \"payload-pre-processing-e2e-extmod-b-6018b4-6868d8bb99-r5frb\" (UID: \"d6601257-a86e-414c-9b71-9dc6042b6818\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6018b4-6868d8bb99-r5frb" Oct 30 18:41:44.755723 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:44.755692 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6601257-a86e-414c-9b71-9dc6042b6818-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-6018b4-6868d8bb99-r5frb\" (UID: \"d6601257-a86e-414c-9b71-9dc6042b6818\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6018b4-6868d8bb99-r5frb" Oct 30 18:41:44.760886 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:44.756880 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6601257-a86e-414c-9b71-9dc6042b6818-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-6018b4-6868d8bb99-r5frb\" (UID: \"d6601257-a86e-414c-9b71-9dc6042b6818\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6018b4-6868d8bb99-r5frb" Oct 30 18:41:44.822208 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:44.822173 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b56jm\" (UniqueName: \"kubernetes.io/projected/d6601257-a86e-414c-9b71-9dc6042b6818-kube-api-access-b56jm\") pod \"payload-pre-processing-e2e-extmod-b-6018b4-6868d8bb99-r5frb\" (UID: \"d6601257-a86e-414c-9b71-9dc6042b6818\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6018b4-6868d8bb99-r5frb" Oct 30 18:41:45.093348 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:45.093256 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6018b4-6868d8bb99-r5frb" Oct 30 18:41:45.464305 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:45.464276 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6018b4-6868d8bb99-r5frb"] Oct 30 18:41:45.868280 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:45.868238 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6018b4-6868d8bb99-r5frb" event={"ID":"d6601257-a86e-414c-9b71-9dc6042b6818","Type":"ContainerStarted","Data":"a2a65a256d273f1db4cbce27ba4f6d5953e44622ebe1cec1c7c05cb6e4374102"} Oct 30 18:41:45.868791 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:45.868304 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6018b4-6868d8bb99-r5frb" event={"ID":"d6601257-a86e-414c-9b71-9dc6042b6818","Type":"ContainerStarted","Data":"11773a5d1610bee4365d9e3b10d1a258d535823d6d34ad1ab0fd88a5ff3333a3"} Oct 30 18:41:45.868791 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:45.868374 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6018b4-6868d8bb99-r5frb" Oct 30 18:41:45.911844 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:45.911793 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6018b4-6868d8bb99-r5frb" podStartSLOduration=1.911778649 podStartE2EDuration="1.911778649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:41:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:41:45.905907319 +0000 UTC m=+2602.995969674" watchObservedRunningTime="2026-10-30 18:41:45.911778649 +0000 UTC m=+2603.001841001" Oct 30 18:41:47.577566 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:41:47.577522 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:41:49.069894 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:49.069856 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ec94a8b-b5h8z"] Oct 30 18:41:49.070340 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:49.069904 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ec94a8b-b5h8z"] Oct 30 18:41:50.069335 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:50.069297 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8c97e49-rndq4"] Oct 30 18:41:50.074103 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:50.074075 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8c97e49-rndq4"] Oct 30 18:41:51.067627 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:51.067597 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8549bb2f-9746-497d-9620-51aa7fa7a218" path="/var/lib/kubelet/pods/8549bb2f-9746-497d-9620-51aa7fa7a218/volumes" Oct 30 18:41:51.068096 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:51.068083 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9579478-0779-42e0-9073-6175f079d629" path="/var/lib/kubelet/pods/a9579478-0779-42e0-9073-6175f079d629/volumes" Oct 30 18:41:54.953716 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:54.953686 4442 generic.go:356] "Generic (PLEG): container finished" podID="57bae3b7-f320-45e8-bdff-e41abbb9f20c" containerID="6d0627b0ed1bf6e7a5d4a27d1ff0aa10bae33940ce515c1bd38a78b1a8466a13" exitCode=0 Oct 30 18:41:54.954204 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:54.953766 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" event={"ID":"57bae3b7-f320-45e8-bdff-e41abbb9f20c","Type":"ContainerDied","Data":"6d0627b0ed1bf6e7a5d4a27d1ff0aa10bae33940ce515c1bd38a78b1a8466a13"} Oct 30 18:41:54.954297 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:54.954281 4442 scope.go:122] "RemoveContainer" containerID="6d0627b0ed1bf6e7a5d4a27d1ff0aa10bae33940ce515c1bd38a78b1a8466a13" Oct 30 18:41:54.956212 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:54.956185 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:41:54.956305 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:54.956253 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:41:54.956305 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:54.956288 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:41:56.116156 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:56.105932 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" event={"ID":"57bae3b7-f320-45e8-bdff-e41abbb9f20c","Type":"ContainerStarted","Data":"ad46a731672326b4c762008c36d30c8e288327b2cb72e22e9d5b070da3842a39"} Oct 30 18:41:56.880790 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:56.880759 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6018b4-6868d8bb99-r5frb" Oct 30 18:41:57.008958 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:57.008933 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" Oct 30 18:41:57.010094 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:57.010062 4442 patch_prober.go:29] interesting pod/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 30 18:41:57.010260 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:57.010106 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" podUID="57bae3b7-f320-45e8-bdff-e41abbb9f20c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 30 18:41:58.008601 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:58.008568 4442 patch_prober.go:29] interesting pod/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 30 18:41:58.009063 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:58.008641 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" podUID="57bae3b7-f320-45e8-bdff-e41abbb9f20c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 30 18:41:59.008613 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:59.008575 4442 patch_prober.go:29] interesting pod/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 30 18:41:59.009050 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:59.008638 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" podUID="57bae3b7-f320-45e8-bdff-e41abbb9f20c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 30 18:41:59.074514 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:59.074462 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db7d40cf-dd9pg"] Oct 30 18:41:59.074514 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:41:59.074510 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db7d40cf-dd9pg"] Oct 30 18:42:00.009069 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:42:00.009038 4442 patch_prober.go:29] interesting pod/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 30 18:42:00.009499 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:42:00.009089 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" podUID="57bae3b7-f320-45e8-bdff-e41abbb9f20c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 30 18:42:01.008891 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:42:01.008855 4442 patch_prober.go:29] interesting pod/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 30 18:42:01.009097 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:42:01.008916 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" podUID="57bae3b7-f320-45e8-bdff-e41abbb9f20c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 30 18:42:01.066946 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:42:01.066911 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc54b885-2ec0-4053-b535-54db5f1ca5a7" path="/var/lib/kubelet/pods/fc54b885-2ec0-4053-b535-54db5f1ca5a7/volumes" Oct 30 18:42:01.801062 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:42:01.801032 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" Oct 30 18:42:02.013760 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:42:02.013728 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" Oct 30 18:42:02.197375 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:42:02.197304 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" Oct 30 18:42:17.604499 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:42:17.604455 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:42:26.288026 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:42:26.287993 4442 scope.go:122] "RemoveContainer" containerID="3f0f5a9610b7599c6cd39fd20431d3e3be0d3099a8efd90ed8f54b5d556ff206" Oct 30 18:42:47.564435 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:42:47.564398 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:42:50.030965 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:42:50.030931 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18308ff8-6wbzx"] Oct 30 18:42:50.044341 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:42:50.044302 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18308ff8-6wbzx"] Oct 30 18:42:51.069575 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:42:51.069541 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24975938-abe1-4fdf-9f9b-c0d37f84c394" path="/var/lib/kubelet/pods/24975938-abe1-4fdf-9f9b-c0d37f84c394/volumes" Oct 30 18:43:00.033038 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:00.032997 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c46f6e24-rv2dx"] Oct 30 18:43:00.041641 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:00.041600 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c46f6e24-rv2dx"] Oct 30 18:43:01.068662 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:01.068628 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b737eec0-6c8e-4bf2-9a62-286bae2d2b57" path="/var/lib/kubelet/pods/b737eec0-6c8e-4bf2-9a62-286bae2d2b57/volumes" Oct 30 18:43:02.220398 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:02.220365 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f7kdhv2"] Oct 30 18:43:02.233772 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:02.233738 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f7kdhv2" Oct 30 18:43:02.255743 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:02.255711 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f7kdhv2"] Oct 30 18:43:02.256174 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:02.256147 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8b8e54\"/\"payload-processing-external-model-plugins-e2e-extmod-b-8b8e54\"" Oct 30 18:43:02.256638 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:02.256456 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8b8e54\"/\"kube-root-ca.crt\"" Oct 30 18:43:02.256638 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:02.256517 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8b8e54\"/\"routing-overlay\"" Oct 30 18:43:02.256638 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:02.256584 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8b8e54\"/\"openshift-service-ca.crt\"" Oct 30 18:43:02.256638 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:02.256598 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8b8e54\"/\"agc-cred-33ba6407b31e\"" Oct 30 18:43:02.260654 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:02.260636 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8b8e54\"/\"agc-cred-695bf65e6f2e\"" Oct 30 18:43:02.260885 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:02.260871 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8b8e54\"/\"payload-processing-external-model-e2e--13a66b56-dockercfg-kjqxg\"" Oct 30 18:43:02.344226 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:02.344185 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrnxk\" (UniqueName: \"kubernetes.io/projected/e81ba3a0-ff70-40ae-bff9-bef9133a9464-kube-api-access-jrnxk\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-78f7kdhv2\" (UID: \"e81ba3a0-ff70-40ae-bff9-bef9133a9464\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f7kdhv2" Oct 30 18:43:02.344569 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:02.344540 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e81ba3a0-ff70-40ae-bff9-bef9133a9464-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-78f7kdhv2\" (UID: \"e81ba3a0-ff70-40ae-bff9-bef9133a9464\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f7kdhv2" Oct 30 18:43:02.344705 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:02.344677 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e81ba3a0-ff70-40ae-bff9-bef9133a9464-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-78f7kdhv2\" (UID: \"e81ba3a0-ff70-40ae-bff9-bef9133a9464\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f7kdhv2" Oct 30 18:43:02.344890 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:02.344872 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e81ba3a0-ff70-40ae-bff9-bef9133a9464-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-78f7kdhv2\" (UID: \"e81ba3a0-ff70-40ae-bff9-bef9133a9464\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f7kdhv2" Oct 30 18:43:02.446105 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:02.446056 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e81ba3a0-ff70-40ae-bff9-bef9133a9464-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-78f7kdhv2\" (UID: \"e81ba3a0-ff70-40ae-bff9-bef9133a9464\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f7kdhv2" Oct 30 18:43:02.446344 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:02.446240 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrnxk\" (UniqueName: \"kubernetes.io/projected/e81ba3a0-ff70-40ae-bff9-bef9133a9464-kube-api-access-jrnxk\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-78f7kdhv2\" (UID: \"e81ba3a0-ff70-40ae-bff9-bef9133a9464\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f7kdhv2" Oct 30 18:43:02.446344 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:02.446305 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e81ba3a0-ff70-40ae-bff9-bef9133a9464-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-78f7kdhv2\" (UID: \"e81ba3a0-ff70-40ae-bff9-bef9133a9464\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f7kdhv2" Oct 30 18:43:02.446496 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:02.446366 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e81ba3a0-ff70-40ae-bff9-bef9133a9464-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-78f7kdhv2\" (UID: \"e81ba3a0-ff70-40ae-bff9-bef9133a9464\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f7kdhv2" Oct 30 18:43:02.446994 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:02.446956 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e81ba3a0-ff70-40ae-bff9-bef9133a9464-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-78f7kdhv2\" (UID: \"e81ba3a0-ff70-40ae-bff9-bef9133a9464\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f7kdhv2" Oct 30 18:43:02.447387 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:02.447365 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e81ba3a0-ff70-40ae-bff9-bef9133a9464-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-78f7kdhv2\" (UID: \"e81ba3a0-ff70-40ae-bff9-bef9133a9464\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f7kdhv2" Oct 30 18:43:02.449665 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:02.449637 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e81ba3a0-ff70-40ae-bff9-bef9133a9464-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-78f7kdhv2\" (UID: \"e81ba3a0-ff70-40ae-bff9-bef9133a9464\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f7kdhv2" Oct 30 18:43:02.466366 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:02.466339 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrnxk\" (UniqueName: \"kubernetes.io/projected/e81ba3a0-ff70-40ae-bff9-bef9133a9464-kube-api-access-jrnxk\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-78f7kdhv2\" (UID: \"e81ba3a0-ff70-40ae-bff9-bef9133a9464\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f7kdhv2" Oct 30 18:43:02.564519 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:02.564435 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f7kdhv2" Oct 30 18:43:02.876371 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:02.876341 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f7kdhv2"] Oct 30 18:43:02.876529 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:43:02.876338 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode81ba3a0_ff70_40ae_bff9_bef9133a9464.slice/crio-307249883a654b53fac99b1c0a34884fe374026e44a192f086d50b09f3d3db55 WatchSource:0}: Error finding container 307249883a654b53fac99b1c0a34884fe374026e44a192f086d50b09f3d3db55: Status 404 returned error can't find the container with id 307249883a654b53fac99b1c0a34884fe374026e44a192f086d50b09f3d3db55 Oct 30 18:43:02.957410 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:02.957372 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f7kdhv2" event={"ID":"e81ba3a0-ff70-40ae-bff9-bef9133a9464","Type":"ContainerStarted","Data":"307249883a654b53fac99b1c0a34884fe374026e44a192f086d50b09f3d3db55"} Oct 30 18:43:03.969850 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:03.969810 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f7kdhv2" event={"ID":"e81ba3a0-ff70-40ae-bff9-bef9133a9464","Type":"ContainerStarted","Data":"a8eeabf4a797a211bbe093415096cf19032cace10288be1e8687d529b2f6aea5"} Oct 30 18:43:03.970400 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:03.969920 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f7kdhv2" Oct 30 18:43:04.013378 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:04.013326 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f7kdhv2" podStartSLOduration=2.013310832 podStartE2EDuration="2.013310832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:04.010201036 +0000 UTC m=+2681.100263375" watchObservedRunningTime="2026-10-30 18:43:04.013310832 +0000 UTC m=+2681.103373174" Oct 30 18:43:04.033756 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:04.033710 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-434dd2db-xz4ll"] Oct 30 18:43:04.038217 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:04.038189 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-434dd2db-xz4ll"] Oct 30 18:43:05.067505 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:05.067471 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d35f52d-1113-4f7a-8ac6-036152da4093" path="/var/lib/kubelet/pods/5d35f52d-1113-4f7a-8ac6-036152da4093/volumes" Oct 30 18:43:14.982995 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:14.982962 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f7kdhv2" Oct 30 18:43:17.564036 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:43:17.563992 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:43:23.688485 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:23.688457 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52rn6_516c6b6c-fc40-49cf-93b5-1c4471b02cf0/ovn-acl-logging/0.log" Oct 30 18:43:23.688963 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:23.688457 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52rn6_516c6b6c-fc40-49cf-93b5-1c4471b02cf0/ovn-acl-logging/0.log" Oct 30 18:43:26.320526 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:26.320497 4442 scope.go:122] "RemoveContainer" containerID="9eae321ca072e1d2ac322f9901fa58096bda69ebef9ed3ece8e41e6c61604fd9" Oct 30 18:43:26.338368 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:26.338344 4442 scope.go:122] "RemoveContainer" containerID="cf5b0b0104958619f2e046184ba1917bdb849b385a01d616626beabd041bf064" Oct 30 18:43:26.352645 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:26.352624 4442 scope.go:122] "RemoveContainer" containerID="d19ad0efef9d5662ef9730586e9353d2773424b163591fd71c7d5ebdacd82b12" Oct 30 18:43:44.981269 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:44.981228 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f7kdhv2"] Oct 30 18:43:47.566312 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:43:47.566260 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:43:49.982551 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:49.982509 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f7kdhv2" podUID="e81ba3a0-ff70-40ae-bff9-bef9133a9464" containerName="payload-processing" containerID="cri-o://a8eeabf4a797a211bbe093415096cf19032cace10288be1e8687d529b2f6aea5" gracePeriod=25 Oct 30 18:43:50.705349 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:50.705321 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f7kdhv2" Oct 30 18:43:50.781444 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:50.781409 4442 generic.go:356] "Generic (PLEG): container finished" podID="e81ba3a0-ff70-40ae-bff9-bef9133a9464" containerID="a8eeabf4a797a211bbe093415096cf19032cace10288be1e8687d529b2f6aea5" exitCode=0 Oct 30 18:43:50.781957 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:50.781928 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f7kdhv2" event={"ID":"e81ba3a0-ff70-40ae-bff9-bef9133a9464","Type":"ContainerDied","Data":"a8eeabf4a797a211bbe093415096cf19032cace10288be1e8687d529b2f6aea5"} Oct 30 18:43:50.782181 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:50.782162 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f7kdhv2" event={"ID":"e81ba3a0-ff70-40ae-bff9-bef9133a9464","Type":"ContainerDied","Data":"307249883a654b53fac99b1c0a34884fe374026e44a192f086d50b09f3d3db55"} Oct 30 18:43:50.782374 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:50.782358 4442 scope.go:122] "RemoveContainer" containerID="a8eeabf4a797a211bbe093415096cf19032cace10288be1e8687d529b2f6aea5" Oct 30 18:43:50.782681 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:50.782661 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f7kdhv2" Oct 30 18:43:50.832965 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:50.832861 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e81ba3a0-ff70-40ae-bff9-bef9133a9464-kube-api-access-jrnxk\" (UniqueName: \"kubernetes.io/projected/e81ba3a0-ff70-40ae-bff9-bef9133a9464-kube-api-access-jrnxk\") pod \"e81ba3a0-ff70-40ae-bff9-bef9133a9464\" (UID: \"e81ba3a0-ff70-40ae-bff9-bef9133a9464\") " Oct 30 18:43:50.833452 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:50.833414 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e81ba3a0-ff70-40ae-bff9-bef9133a9464-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e81ba3a0-ff70-40ae-bff9-bef9133a9464-provider-credentials\") pod \"e81ba3a0-ff70-40ae-bff9-bef9133a9464\" (UID: \"e81ba3a0-ff70-40ae-bff9-bef9133a9464\") " Oct 30 18:43:50.834256 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:50.834234 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e81ba3a0-ff70-40ae-bff9-bef9133a9464-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e81ba3a0-ff70-40ae-bff9-bef9133a9464-routing-overlay-volume\") pod \"e81ba3a0-ff70-40ae-bff9-bef9133a9464\" (UID: \"e81ba3a0-ff70-40ae-bff9-bef9133a9464\") " Oct 30 18:43:50.835661 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:50.835640 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e81ba3a0-ff70-40ae-bff9-bef9133a9464-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e81ba3a0-ff70-40ae-bff9-bef9133a9464-plugins-config-volume\") pod \"e81ba3a0-ff70-40ae-bff9-bef9133a9464\" (UID: \"e81ba3a0-ff70-40ae-bff9-bef9133a9464\") " Oct 30 18:43:50.839479 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:50.835022 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e81ba3a0-ff70-40ae-bff9-bef9133a9464-routing-overlay-volume" pod "e81ba3a0-ff70-40ae-bff9-bef9133a9464" (UID: "e81ba3a0-ff70-40ae-bff9-bef9133a9464"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:43:50.840020 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:50.838376 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e81ba3a0-ff70-40ae-bff9-bef9133a9464-plugins-config-volume" pod "e81ba3a0-ff70-40ae-bff9-bef9133a9464" (UID: "e81ba3a0-ff70-40ae-bff9-bef9133a9464"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:43:50.840447 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:50.840426 4442 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e81ba3a0-ff70-40ae-bff9-bef9133a9464-routing-overlay-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:43:50.840582 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:50.840568 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e81ba3a0-ff70-40ae-bff9-bef9133a9464-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:43:50.845998 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:50.845929 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e81ba3a0-ff70-40ae-bff9-bef9133a9464-provider-credentials" pod "e81ba3a0-ff70-40ae-bff9-bef9133a9464" (UID: "e81ba3a0-ff70-40ae-bff9-bef9133a9464"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:43:50.855759 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:50.855599 4442 scope.go:122] "RemoveContainer" containerID="a8eeabf4a797a211bbe093415096cf19032cace10288be1e8687d529b2f6aea5" Oct 30 18:43:50.856564 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:43:50.856472 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8eeabf4a797a211bbe093415096cf19032cace10288be1e8687d529b2f6aea5\": container with ID starting with a8eeabf4a797a211bbe093415096cf19032cace10288be1e8687d529b2f6aea5 not found: ID does not exist" containerID="a8eeabf4a797a211bbe093415096cf19032cace10288be1e8687d529b2f6aea5" Oct 30 18:43:50.856564 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:50.856515 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8eeabf4a797a211bbe093415096cf19032cace10288be1e8687d529b2f6aea5"} err="failed to get container status \"a8eeabf4a797a211bbe093415096cf19032cace10288be1e8687d529b2f6aea5\": rpc error: code = NotFound desc = could not find container \"a8eeabf4a797a211bbe093415096cf19032cace10288be1e8687d529b2f6aea5\": container with ID starting with a8eeabf4a797a211bbe093415096cf19032cace10288be1e8687d529b2f6aea5 not found: ID does not exist" Oct 30 18:43:50.857052 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:50.857028 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e81ba3a0-ff70-40ae-bff9-bef9133a9464-kube-api-access-jrnxk" pod "e81ba3a0-ff70-40ae-bff9-bef9133a9464" (UID: "e81ba3a0-ff70-40ae-bff9-bef9133a9464"). InnerVolumeSpecName "kube-api-access-jrnxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:43:50.942600 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:50.942569 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrnxk\" (UniqueName: \"kubernetes.io/projected/e81ba3a0-ff70-40ae-bff9-bef9133a9464-kube-api-access-jrnxk\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:43:50.942600 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:50.942597 4442 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e81ba3a0-ff70-40ae-bff9-bef9133a9464-provider-credentials\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:43:51.144319 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:51.144289 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f7kdhv2"] Oct 30 18:43:51.158806 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:51.158762 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f7kdhv2"] Oct 30 18:43:53.068552 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:43:53.068522 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e81ba3a0-ff70-40ae-bff9-bef9133a9464" path="/var/lib/kubelet/pods/e81ba3a0-ff70-40ae-bff9-bef9133a9464/volumes" Oct 30 18:44:17.563015 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:44:17.562930 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:44:20.819270 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:20.819224 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m"] Oct 30 18:44:20.820306 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:20.820283 4442 cpu_manager.go:414] "containerMap: removing container" podUID="e81ba3a0-ff70-40ae-bff9-bef9133a9464" containerName="payload-processing" Oct 30 18:44:20.820306 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:20.820305 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e81ba3a0-ff70-40ae-bff9-bef9133a9464" containerName="payload-processing" Oct 30 18:44:20.820499 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:20.820489 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="e81ba3a0-ff70-40ae-bff9-bef9133a9464" containerName="payload-processing" Oct 30 18:44:20.823292 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:20.823272 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" Oct 30 18:44:20.842298 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:20.842272 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-8b8e54-openshift-default-dockercfg-q4lxn\"" Oct 30 18:44:20.849034 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:20.849004 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aac37dcc-c395-40e0-8d20-ec7733106b01-istio-podinfo\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m\" (UID: \"aac37dcc-c395-40e0-8d20-ec7733106b01\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" Oct 30 18:44:20.849203 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:20.849067 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aac37dcc-c395-40e0-8d20-ec7733106b01-istio-token\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m\" (UID: \"aac37dcc-c395-40e0-8d20-ec7733106b01\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" Oct 30 18:44:20.849287 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:20.849272 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aac37dcc-c395-40e0-8d20-ec7733106b01-istio-data\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m\" (UID: \"aac37dcc-c395-40e0-8d20-ec7733106b01\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" Oct 30 18:44:20.849345 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:20.849295 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aac37dcc-c395-40e0-8d20-ec7733106b01-istio-envoy\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m\" (UID: \"aac37dcc-c395-40e0-8d20-ec7733106b01\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" Oct 30 18:44:20.849345 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:20.849326 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8xk8\" (UniqueName: \"kubernetes.io/projected/aac37dcc-c395-40e0-8d20-ec7733106b01-kube-api-access-w8xk8\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m\" (UID: \"aac37dcc-c395-40e0-8d20-ec7733106b01\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" Oct 30 18:44:20.849465 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:20.849431 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aac37dcc-c395-40e0-8d20-ec7733106b01-workload-certs\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m\" (UID: \"aac37dcc-c395-40e0-8d20-ec7733106b01\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" Oct 30 18:44:20.849526 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:20.849505 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aac37dcc-c395-40e0-8d20-ec7733106b01-istiod-ca-cert\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m\" (UID: \"aac37dcc-c395-40e0-8d20-ec7733106b01\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" Oct 30 18:44:20.849587 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:20.849561 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac37dcc-c395-40e0-8d20-ec7733106b01-credential-socket\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m\" (UID: \"aac37dcc-c395-40e0-8d20-ec7733106b01\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" Oct 30 18:44:20.849643 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:20.849585 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac37dcc-c395-40e0-8d20-ec7733106b01-workload-socket\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m\" (UID: \"aac37dcc-c395-40e0-8d20-ec7733106b01\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" Oct 30 18:44:20.857240 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:20.857214 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m"] Oct 30 18:44:20.951343 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:20.951304 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac37dcc-c395-40e0-8d20-ec7733106b01-credential-socket\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m\" (UID: \"aac37dcc-c395-40e0-8d20-ec7733106b01\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" Oct 30 18:44:20.951343 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:20.951358 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac37dcc-c395-40e0-8d20-ec7733106b01-workload-socket\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m\" (UID: \"aac37dcc-c395-40e0-8d20-ec7733106b01\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" Oct 30 18:44:20.951628 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:20.951498 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aac37dcc-c395-40e0-8d20-ec7733106b01-istio-podinfo\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m\" (UID: \"aac37dcc-c395-40e0-8d20-ec7733106b01\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" Oct 30 18:44:20.951628 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:20.951546 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aac37dcc-c395-40e0-8d20-ec7733106b01-istio-token\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m\" (UID: \"aac37dcc-c395-40e0-8d20-ec7733106b01\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" Oct 30 18:44:20.951628 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:20.951617 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aac37dcc-c395-40e0-8d20-ec7733106b01-istio-data\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m\" (UID: \"aac37dcc-c395-40e0-8d20-ec7733106b01\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" Oct 30 18:44:20.952035 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:20.951641 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aac37dcc-c395-40e0-8d20-ec7733106b01-istio-envoy\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m\" (UID: \"aac37dcc-c395-40e0-8d20-ec7733106b01\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" Oct 30 18:44:20.952035 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:20.951672 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8xk8\" (UniqueName: \"kubernetes.io/projected/aac37dcc-c395-40e0-8d20-ec7733106b01-kube-api-access-w8xk8\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m\" (UID: \"aac37dcc-c395-40e0-8d20-ec7733106b01\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" Oct 30 18:44:20.952035 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:20.951704 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aac37dcc-c395-40e0-8d20-ec7733106b01-workload-certs\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m\" (UID: \"aac37dcc-c395-40e0-8d20-ec7733106b01\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" Oct 30 18:44:20.952035 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:20.951767 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aac37dcc-c395-40e0-8d20-ec7733106b01-istiod-ca-cert\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m\" (UID: \"aac37dcc-c395-40e0-8d20-ec7733106b01\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" Oct 30 18:44:20.952035 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:20.951861 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac37dcc-c395-40e0-8d20-ec7733106b01-workload-socket\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m\" (UID: \"aac37dcc-c395-40e0-8d20-ec7733106b01\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" Oct 30 18:44:20.952397 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:20.952114 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aac37dcc-c395-40e0-8d20-ec7733106b01-istio-data\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m\" (UID: \"aac37dcc-c395-40e0-8d20-ec7733106b01\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" Oct 30 18:44:20.952858 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:20.952629 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac37dcc-c395-40e0-8d20-ec7733106b01-credential-socket\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m\" (UID: \"aac37dcc-c395-40e0-8d20-ec7733106b01\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" Oct 30 18:44:20.953379 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:20.953356 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aac37dcc-c395-40e0-8d20-ec7733106b01-istiod-ca-cert\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m\" (UID: \"aac37dcc-c395-40e0-8d20-ec7733106b01\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" Oct 30 18:44:20.954537 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:20.954511 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aac37dcc-c395-40e0-8d20-ec7733106b01-workload-certs\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m\" (UID: \"aac37dcc-c395-40e0-8d20-ec7733106b01\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" Oct 30 18:44:20.955789 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:20.955760 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aac37dcc-c395-40e0-8d20-ec7733106b01-istio-podinfo\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m\" (UID: \"aac37dcc-c395-40e0-8d20-ec7733106b01\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" Oct 30 18:44:20.955914 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:20.955893 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aac37dcc-c395-40e0-8d20-ec7733106b01-istio-envoy\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m\" (UID: \"aac37dcc-c395-40e0-8d20-ec7733106b01\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" Oct 30 18:44:20.972333 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:20.972305 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aac37dcc-c395-40e0-8d20-ec7733106b01-istio-token\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m\" (UID: \"aac37dcc-c395-40e0-8d20-ec7733106b01\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" Oct 30 18:44:20.972888 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:20.972869 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8xk8\" (UniqueName: \"kubernetes.io/projected/aac37dcc-c395-40e0-8d20-ec7733106b01-kube-api-access-w8xk8\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m\" (UID: \"aac37dcc-c395-40e0-8d20-ec7733106b01\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" Oct 30 18:44:21.146162 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:21.145431 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" Oct 30 18:44:21.732647 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:21.732610 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m"] Oct 30 18:44:21.733151 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:44:21.733112 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaac37dcc_c395_40e0_8d20_ec7733106b01.slice/crio-3103d5cdef52f30815e3ca8dafbcc7140703869aa51c68eba4c9c502ac1f6061 WatchSource:0}: Error finding container 3103d5cdef52f30815e3ca8dafbcc7140703869aa51c68eba4c9c502ac1f6061: Status 404 returned error can't find the container with id 3103d5cdef52f30815e3ca8dafbcc7140703869aa51c68eba4c9c502ac1f6061 Oct 30 18:44:21.736344 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:21.736311 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:44:21.736433 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:21.736387 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:44:21.736433 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:21.736416 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:44:22.379112 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:22.379062 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" event={"ID":"aac37dcc-c395-40e0-8d20-ec7733106b01","Type":"ContainerStarted","Data":"e9ab725a3dd442f3c283138f96df0107b9eaa46818815f533a687c8b8dfb2068"} Oct 30 18:44:22.379623 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:22.379126 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" event={"ID":"aac37dcc-c395-40e0-8d20-ec7733106b01","Type":"ContainerStarted","Data":"3103d5cdef52f30815e3ca8dafbcc7140703869aa51c68eba4c9c502ac1f6061"} Oct 30 18:44:22.436794 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:22.436728 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" podStartSLOduration=2.436710026 podStartE2EDuration="2.436710026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:44:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:22.430618275 +0000 UTC m=+2759.520680616" watchObservedRunningTime="2026-10-30 18:44:22.436710026 +0000 UTC m=+2759.526772365" Oct 30 18:44:23.147584 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:23.147547 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" Oct 30 18:44:23.164406 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:23.164369 4442 patch_prober.go:29] interesting pod/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body= Oct 30 18:44:23.164612 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:23.164443 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" podUID="aac37dcc-c395-40e0-8d20-ec7733106b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" Oct 30 18:44:24.148171 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:24.147573 4442 patch_prober.go:29] interesting pod/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body= Oct 30 18:44:24.148171 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:24.147639 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" podUID="aac37dcc-c395-40e0-8d20-ec7733106b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" Oct 30 18:44:25.155157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:25.154419 4442 patch_prober.go:29] interesting pod/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body= Oct 30 18:44:25.155157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:25.154494 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" podUID="aac37dcc-c395-40e0-8d20-ec7733106b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" Oct 30 18:44:26.155279 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:26.150745 4442 patch_prober.go:29] interesting pod/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body= Oct 30 18:44:26.155279 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:26.150809 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" podUID="aac37dcc-c395-40e0-8d20-ec7733106b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" Oct 30 18:44:26.409628 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:26.409539 4442 scope.go:122] "RemoveContainer" containerID="1eb00d811c0953636c500831ce53df75d45680785a7fe2beba6e08d628bc36c2" Oct 30 18:44:26.422433 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:26.422411 4442 scope.go:122] "RemoveContainer" containerID="d620e9087948f13c8f520b24a7c89ec665c6f08a91745ff3381b9afc2f8a1809" Oct 30 18:44:26.462774 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:26.462751 4442 scope.go:122] "RemoveContainer" containerID="6d63d0f032e217f6f9e3159177cc2e1e205ecad10913b1a284a4b606619aa0ec" Oct 30 18:44:27.147159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:27.146865 4442 patch_prober.go:29] interesting pod/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body= Oct 30 18:44:27.147159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:27.146958 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" podUID="aac37dcc-c395-40e0-8d20-ec7733106b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" Oct 30 18:44:28.148535 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:28.148502 4442 patch_prober.go:29] interesting pod/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body= Oct 30 18:44:28.149092 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:28.148551 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" podUID="aac37dcc-c395-40e0-8d20-ec7733106b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" Oct 30 18:44:29.151384 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:29.151355 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" Oct 30 18:44:29.151876 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:29.151737 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" Oct 30 18:44:29.152509 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:44:29.152489 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" Oct 30 18:44:47.563971 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:44:47.563927 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:45:00.157058 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.157025 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889765-v5zh2"] Oct 30 18:45:00.161829 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.161800 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889765-v5zh2" Oct 30 18:45:00.163762 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.163729 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889765-k88tq"] Oct 30 18:45:00.169374 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.169351 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889765-k88tq" Oct 30 18:45:00.214210 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.197041 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889765-k88tq"] Oct 30 18:45:00.214210 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.204027 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889765-ljqcz"] Oct 30 18:45:00.214210 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.211904 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889765-ljqcz" Oct 30 18:45:00.217544 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.217518 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889765-qf9bg"] Oct 30 18:45:00.227981 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.227957 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889765-qf9bg" Oct 30 18:45:00.231251 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.231225 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889765-v5zh2"] Oct 30 18:45:00.244562 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.244531 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889765-ljqcz"] Oct 30 18:45:00.253420 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.253393 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889765-v4sqc"] Oct 30 18:45:00.260265 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.260238 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889765-qf9bg"] Oct 30 18:45:00.260467 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.260359 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889765-v4sqc" Oct 30 18:45:00.271248 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.268259 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889765-dmf99"] Oct 30 18:45:00.278862 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.278814 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:45:00.279720 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.279585 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889765-rrzvv"] Oct 30 18:45:00.280203 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.279693 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889765-dmf99" Oct 30 18:45:00.297739 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.297290 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkjwh\" (UniqueName: \"kubernetes.io/projected/1e4c2309-c818-46be-abd4-add20576fd6a-kube-api-access-rkjwh\") pod \"maas-api-key-cleanup-e2e-extmod-a-2107d4-29889765-k88tq\" (UID: \"1e4c2309-c818-46be-abd4-add20576fd6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889765-k88tq" Oct 30 18:45:00.297739 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.297355 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e752ff2-638c-4b1f-94a9-5efd1a53a9c6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889765-v5zh2\" (UID: \"5e752ff2-638c-4b1f-94a9-5efd1a53a9c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889765-v5zh2" Oct 30 18:45:00.297739 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.297426 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfnd8\" (UniqueName: \"kubernetes.io/projected/5e752ff2-638c-4b1f-94a9-5efd1a53a9c6-kube-api-access-vfnd8\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889765-v5zh2\" (UID: \"5e752ff2-638c-4b1f-94a9-5efd1a53a9c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889765-v5zh2" Oct 30 18:45:00.297739 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.297523 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e4c2309-c818-46be-abd4-add20576fd6a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2107d4-29889765-k88tq\" (UID: \"1e4c2309-c818-46be-abd4-add20576fd6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889765-k88tq" Oct 30 18:45:00.525188 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.522593 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f486a50-76a3-4cc7-bd5b-3edb80ad3315-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889765-ljqcz\" (UID: \"4f486a50-76a3-4cc7-bd5b-3edb80ad3315\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889765-ljqcz" Oct 30 18:45:00.534476 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.534436 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9hkz\" (UniqueName: \"kubernetes.io/projected/4f486a50-76a3-4cc7-bd5b-3edb80ad3315-kube-api-access-m9hkz\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889765-ljqcz\" (UID: \"4f486a50-76a3-4cc7-bd5b-3edb80ad3315\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889765-ljqcz" Oct 30 18:45:00.534685 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.534530 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq5nd\" (UniqueName: \"kubernetes.io/projected/8ee9297b-cfb8-4154-bb20-d5eb73f78da2-kube-api-access-hq5nd\") pod \"maas-api-key-cleanup-e2e-worker-w0-921450-29889765-v4sqc\" (UID: \"8ee9297b-cfb8-4154-bb20-d5eb73f78da2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889765-v4sqc" Oct 30 18:45:00.534685 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.534613 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83b551f2-22ff-42f3-acc9-3122313400d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-713dc8-29889765-dmf99\" (UID: \"83b551f2-22ff-42f3-acc9-3122313400d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889765-dmf99" Oct 30 18:45:00.534841 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.534779 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a87157ec-dde9-4537-a2dc-d2a0e0c80091-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889765-qf9bg\" (UID: \"a87157ec-dde9-4537-a2dc-d2a0e0c80091\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889765-qf9bg" Oct 30 18:45:00.534841 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.534814 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhmsl\" (UniqueName: \"kubernetes.io/projected/a87157ec-dde9-4537-a2dc-d2a0e0c80091-kube-api-access-xhmsl\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889765-qf9bg\" (UID: \"a87157ec-dde9-4537-a2dc-d2a0e0c80091\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889765-qf9bg" Oct 30 18:45:00.534969 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.534864 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkjwh\" (UniqueName: \"kubernetes.io/projected/1e4c2309-c818-46be-abd4-add20576fd6a-kube-api-access-rkjwh\") pod \"maas-api-key-cleanup-e2e-extmod-a-2107d4-29889765-k88tq\" (UID: \"1e4c2309-c818-46be-abd4-add20576fd6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889765-k88tq" Oct 30 18:45:00.534969 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.534929 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e752ff2-638c-4b1f-94a9-5efd1a53a9c6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889765-v5zh2\" (UID: \"5e752ff2-638c-4b1f-94a9-5efd1a53a9c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889765-v5zh2" Oct 30 18:45:00.535093 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.535021 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfnd8\" (UniqueName: \"kubernetes.io/projected/5e752ff2-638c-4b1f-94a9-5efd1a53a9c6-kube-api-access-vfnd8\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889765-v5zh2\" (UID: \"5e752ff2-638c-4b1f-94a9-5efd1a53a9c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889765-v5zh2" Oct 30 18:45:00.535218 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.535197 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e4c2309-c818-46be-abd4-add20576fd6a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2107d4-29889765-k88tq\" (UID: \"1e4c2309-c818-46be-abd4-add20576fd6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889765-k88tq" Oct 30 18:45:00.535331 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.535313 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq5kc\" (UniqueName: \"kubernetes.io/projected/83b551f2-22ff-42f3-acc9-3122313400d8-kube-api-access-hq5kc\") pod \"maas-api-key-cleanup-e2e-extmod-a-713dc8-29889765-dmf99\" (UID: \"83b551f2-22ff-42f3-acc9-3122313400d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889765-dmf99" Oct 30 18:45:00.535401 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.535360 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ee9297b-cfb8-4154-bb20-d5eb73f78da2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-921450-29889765-v4sqc\" (UID: \"8ee9297b-cfb8-4154-bb20-d5eb73f78da2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889765-v4sqc" Oct 30 18:45:00.540194 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.537648 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e4c2309-c818-46be-abd4-add20576fd6a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2107d4-29889765-k88tq\" (UID: \"1e4c2309-c818-46be-abd4-add20576fd6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889765-k88tq" Oct 30 18:45:00.576593 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.566567 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e752ff2-638c-4b1f-94a9-5efd1a53a9c6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889765-v5zh2\" (UID: \"5e752ff2-638c-4b1f-94a9-5efd1a53a9c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889765-v5zh2" Oct 30 18:45:00.600437 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.582119 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889765-rrzvv" Oct 30 18:45:00.639454 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.639415 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq5kc\" (UniqueName: \"kubernetes.io/projected/83b551f2-22ff-42f3-acc9-3122313400d8-kube-api-access-hq5kc\") pod \"maas-api-key-cleanup-e2e-extmod-a-713dc8-29889765-dmf99\" (UID: \"83b551f2-22ff-42f3-acc9-3122313400d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889765-dmf99" Oct 30 18:45:00.639733 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.639708 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ee9297b-cfb8-4154-bb20-d5eb73f78da2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-921450-29889765-v4sqc\" (UID: \"8ee9297b-cfb8-4154-bb20-d5eb73f78da2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889765-v4sqc" Oct 30 18:45:00.649619 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.639886 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f486a50-76a3-4cc7-bd5b-3edb80ad3315-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889765-ljqcz\" (UID: \"4f486a50-76a3-4cc7-bd5b-3edb80ad3315\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889765-ljqcz" Oct 30 18:45:00.649902 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.649876 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9hkz\" (UniqueName: \"kubernetes.io/projected/4f486a50-76a3-4cc7-bd5b-3edb80ad3315-kube-api-access-m9hkz\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889765-ljqcz\" (UID: \"4f486a50-76a3-4cc7-bd5b-3edb80ad3315\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889765-ljqcz" Oct 30 18:45:00.782218 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.650064 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ee9297b-cfb8-4154-bb20-d5eb73f78da2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-921450-29889765-v4sqc\" (UID: \"8ee9297b-cfb8-4154-bb20-d5eb73f78da2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889765-v4sqc" Oct 30 18:45:00.782218 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.650115 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq5nd\" (UniqueName: \"kubernetes.io/projected/8ee9297b-cfb8-4154-bb20-d5eb73f78da2-kube-api-access-hq5nd\") pod \"maas-api-key-cleanup-e2e-worker-w0-921450-29889765-v4sqc\" (UID: \"8ee9297b-cfb8-4154-bb20-d5eb73f78da2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889765-v4sqc" Oct 30 18:45:00.782218 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.650209 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83b551f2-22ff-42f3-acc9-3122313400d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-713dc8-29889765-dmf99\" (UID: \"83b551f2-22ff-42f3-acc9-3122313400d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889765-dmf99" Oct 30 18:45:00.782218 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.650344 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a87157ec-dde9-4537-a2dc-d2a0e0c80091-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889765-qf9bg\" (UID: \"a87157ec-dde9-4537-a2dc-d2a0e0c80091\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889765-qf9bg" Oct 30 18:45:00.782218 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.650387 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhmsl\" (UniqueName: \"kubernetes.io/projected/a87157ec-dde9-4537-a2dc-d2a0e0c80091-kube-api-access-xhmsl\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889765-qf9bg\" (UID: \"a87157ec-dde9-4537-a2dc-d2a0e0c80091\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889765-qf9bg" Oct 30 18:45:00.782218 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.650813 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f486a50-76a3-4cc7-bd5b-3edb80ad3315-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889765-ljqcz\" (UID: \"4f486a50-76a3-4cc7-bd5b-3edb80ad3315\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889765-ljqcz" Oct 30 18:45:00.782218 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.651651 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a87157ec-dde9-4537-a2dc-d2a0e0c80091-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889765-qf9bg\" (UID: \"a87157ec-dde9-4537-a2dc-d2a0e0c80091\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889765-qf9bg" Oct 30 18:45:00.839848 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.826433 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889765-v4sqc"] Oct 30 18:45:00.839848 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.827564 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83b551f2-22ff-42f3-acc9-3122313400d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-713dc8-29889765-dmf99\" (UID: \"83b551f2-22ff-42f3-acc9-3122313400d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889765-dmf99" Oct 30 18:45:00.839848 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.833797 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkjwh\" (UniqueName: \"kubernetes.io/projected/1e4c2309-c818-46be-abd4-add20576fd6a-kube-api-access-rkjwh\") pod \"maas-api-key-cleanup-e2e-extmod-a-2107d4-29889765-k88tq\" (UID: \"1e4c2309-c818-46be-abd4-add20576fd6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889765-k88tq" Oct 30 18:45:00.849937 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.848287 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889765-k88tq" Oct 30 18:45:00.851258 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.850581 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6ld6\" (UniqueName: \"kubernetes.io/projected/7d52db28-5cd7-4a2a-af8b-b876d41c8453-kube-api-access-n6ld6\") pod \"maas-api-key-cleanup-e2e-extmod-b-6babca-29889765-rrzvv\" (UID: \"7d52db28-5cd7-4a2a-af8b-b876d41c8453\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889765-rrzvv" Oct 30 18:45:00.851258 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.850928 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d52db28-5cd7-4a2a-af8b-b876d41c8453-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-6babca-29889765-rrzvv\" (UID: \"7d52db28-5cd7-4a2a-af8b-b876d41c8453\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889765-rrzvv" Oct 30 18:45:00.854921 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.854890 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq5kc\" (UniqueName: \"kubernetes.io/projected/83b551f2-22ff-42f3-acc9-3122313400d8-kube-api-access-hq5kc\") pod \"maas-api-key-cleanup-e2e-extmod-a-713dc8-29889765-dmf99\" (UID: \"83b551f2-22ff-42f3-acc9-3122313400d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889765-dmf99" Oct 30 18:45:00.855593 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.855567 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfnd8\" (UniqueName: \"kubernetes.io/projected/5e752ff2-638c-4b1f-94a9-5efd1a53a9c6-kube-api-access-vfnd8\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889765-v5zh2\" (UID: \"5e752ff2-638c-4b1f-94a9-5efd1a53a9c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889765-v5zh2" Oct 30 18:45:00.856525 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.856497 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9hkz\" (UniqueName: \"kubernetes.io/projected/4f486a50-76a3-4cc7-bd5b-3edb80ad3315-kube-api-access-m9hkz\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889765-ljqcz\" (UID: \"4f486a50-76a3-4cc7-bd5b-3edb80ad3315\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889765-ljqcz" Oct 30 18:45:00.856954 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.856928 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq5nd\" (UniqueName: \"kubernetes.io/projected/8ee9297b-cfb8-4154-bb20-d5eb73f78da2-kube-api-access-hq5nd\") pod \"maas-api-key-cleanup-e2e-worker-w0-921450-29889765-v4sqc\" (UID: \"8ee9297b-cfb8-4154-bb20-d5eb73f78da2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889765-v4sqc" Oct 30 18:45:00.857049 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.857000 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889765-dmf99"] Oct 30 18:45:00.857339 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.857321 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889765-ljqcz" Oct 30 18:45:00.872084 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.868964 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhmsl\" (UniqueName: \"kubernetes.io/projected/a87157ec-dde9-4537-a2dc-d2a0e0c80091-kube-api-access-xhmsl\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889765-qf9bg\" (UID: \"a87157ec-dde9-4537-a2dc-d2a0e0c80091\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889765-qf9bg" Oct 30 18:45:00.877182 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.874717 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889765-rrzvv"] Oct 30 18:45:00.883365 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.883175 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889765-b2hth"] Oct 30 18:45:00.888573 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.888548 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889765-b2hth" Oct 30 18:45:00.893955 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.890330 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889765-fsnp2"] Oct 30 18:45:00.896569 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.896546 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889765-fsnp2" Oct 30 18:45:00.905881 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.905759 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889765-dmf99" Oct 30 18:45:00.909348 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:00.909315 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889765-4vxnf"] Oct 30 18:45:01.020161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.008964 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d52db28-5cd7-4a2a-af8b-b876d41c8453-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-6babca-29889765-rrzvv\" (UID: \"7d52db28-5cd7-4a2a-af8b-b876d41c8453\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889765-rrzvv" Oct 30 18:45:01.020161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.009158 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36a43766-ec94-4991-bad0-e3d4ed5122c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889765-b2hth\" (UID: \"36a43766-ec94-4991-bad0-e3d4ed5122c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889765-b2hth" Oct 30 18:45:01.020161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.009226 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6ld6\" (UniqueName: \"kubernetes.io/projected/7d52db28-5cd7-4a2a-af8b-b876d41c8453-kube-api-access-n6ld6\") pod \"maas-api-key-cleanup-e2e-extmod-b-6babca-29889765-rrzvv\" (UID: \"7d52db28-5cd7-4a2a-af8b-b876d41c8453\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889765-rrzvv" Oct 30 18:45:01.020161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.009371 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zdcz\" (UniqueName: \"kubernetes.io/projected/f7cdca9f-f772-449c-9f30-067437ef08cc-kube-api-access-2zdcz\") pod \"maas-api-key-cleanup-e2e-worker-w2-45dc34-29889765-fsnp2\" (UID: \"f7cdca9f-f772-449c-9f30-067437ef08cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889765-fsnp2" Oct 30 18:45:01.020161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.009421 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7cdca9f-f772-449c-9f30-067437ef08cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-45dc34-29889765-fsnp2\" (UID: \"f7cdca9f-f772-449c-9f30-067437ef08cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889765-fsnp2" Oct 30 18:45:01.020161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.009488 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lt2h\" (UniqueName: \"kubernetes.io/projected/36a43766-ec94-4991-bad0-e3d4ed5122c8-kube-api-access-5lt2h\") pod \"maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889765-b2hth\" (UID: \"36a43766-ec94-4991-bad0-e3d4ed5122c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889765-b2hth" Oct 30 18:45:01.076220 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.068092 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d52db28-5cd7-4a2a-af8b-b876d41c8453-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-6babca-29889765-rrzvv\" (UID: \"7d52db28-5cd7-4a2a-af8b-b876d41c8453\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889765-rrzvv" Oct 30 18:45:01.083903 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.083660 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889765-v5zh2" Oct 30 18:45:01.100345 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.087218 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889765-4vxnf" Oct 30 18:45:01.102239 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.101059 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6ld6\" (UniqueName: \"kubernetes.io/projected/7d52db28-5cd7-4a2a-af8b-b876d41c8453-kube-api-access-n6ld6\") pod \"maas-api-key-cleanup-e2e-extmod-b-6babca-29889765-rrzvv\" (UID: \"7d52db28-5cd7-4a2a-af8b-b876d41c8453\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889765-rrzvv" Oct 30 18:45:01.110844 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.110811 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zdcz\" (UniqueName: \"kubernetes.io/projected/f7cdca9f-f772-449c-9f30-067437ef08cc-kube-api-access-2zdcz\") pod \"maas-api-key-cleanup-e2e-worker-w2-45dc34-29889765-fsnp2\" (UID: \"f7cdca9f-f772-449c-9f30-067437ef08cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889765-fsnp2" Oct 30 18:45:01.111000 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.110860 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7cdca9f-f772-449c-9f30-067437ef08cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-45dc34-29889765-fsnp2\" (UID: \"f7cdca9f-f772-449c-9f30-067437ef08cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889765-fsnp2" Oct 30 18:45:01.111000 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.110897 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lt2h\" (UniqueName: \"kubernetes.io/projected/36a43766-ec94-4991-bad0-e3d4ed5122c8-kube-api-access-5lt2h\") pod \"maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889765-b2hth\" (UID: \"36a43766-ec94-4991-bad0-e3d4ed5122c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889765-b2hth" Oct 30 18:45:01.111110 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.111013 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36a43766-ec94-4991-bad0-e3d4ed5122c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889765-b2hth\" (UID: \"36a43766-ec94-4991-bad0-e3d4ed5122c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889765-b2hth" Oct 30 18:45:01.111934 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.111906 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36a43766-ec94-4991-bad0-e3d4ed5122c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889765-b2hth\" (UID: \"36a43766-ec94-4991-bad0-e3d4ed5122c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889765-b2hth" Oct 30 18:45:01.112817 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.112791 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7cdca9f-f772-449c-9f30-067437ef08cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-45dc34-29889765-fsnp2\" (UID: \"f7cdca9f-f772-449c-9f30-067437ef08cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889765-fsnp2" Oct 30 18:45:01.132858 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.132825 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889765-rrzvv" Oct 30 18:45:01.134969 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.134833 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889765-v4sqc" Oct 30 18:45:01.164507 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.164475 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889765-qf9bg" Oct 30 18:45:01.176049 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.175982 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lt2h\" (UniqueName: \"kubernetes.io/projected/36a43766-ec94-4991-bad0-e3d4ed5122c8-kube-api-access-5lt2h\") pod \"maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889765-b2hth\" (UID: \"36a43766-ec94-4991-bad0-e3d4ed5122c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889765-b2hth" Oct 30 18:45:01.191994 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.191926 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zdcz\" (UniqueName: \"kubernetes.io/projected/f7cdca9f-f772-449c-9f30-067437ef08cc-kube-api-access-2zdcz\") pod \"maas-api-key-cleanup-e2e-worker-w2-45dc34-29889765-fsnp2\" (UID: \"f7cdca9f-f772-449c-9f30-067437ef08cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889765-fsnp2" Oct 30 18:45:01.199617 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.199591 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889765-b2hth"] Oct 30 18:45:01.200062 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.200045 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889765-tdq6b"] Oct 30 18:45:01.202737 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.200047 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889765-b2hth" Oct 30 18:45:01.209002 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.208593 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889765-tdq6b" Oct 30 18:45:01.209200 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.208507 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889765-fsnp2"] Oct 30 18:45:01.210200 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.209329 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889765-mx8tm"] Oct 30 18:45:01.213599 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.213578 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfq4p\" (UniqueName: \"kubernetes.io/projected/ed9e3fd1-ccad-4d33-bb46-222acf609143-kube-api-access-bfq4p\") pod \"maas-api-key-cleanup-e2e-worker-w1-380c08-29889765-4vxnf\" (UID: \"ed9e3fd1-ccad-4d33-bb46-222acf609143\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889765-4vxnf" Oct 30 18:45:01.215160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.215057 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed9e3fd1-ccad-4d33-bb46-222acf609143-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-380c08-29889765-4vxnf\" (UID: \"ed9e3fd1-ccad-4d33-bb46-222acf609143\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889765-4vxnf" Oct 30 18:45:01.221264 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.221065 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889765-fsnp2" Oct 30 18:45:01.227515 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.226342 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889765-4vxnf"] Oct 30 18:45:01.227515 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.226383 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889765-tdq6b"] Oct 30 18:45:01.227515 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.226410 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889765-mx8tm"] Oct 30 18:45:01.227515 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.226566 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889765-mx8tm" Oct 30 18:45:01.334075 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.333590 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889765-mx8tm\" (UID: \"f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889765-mx8tm" Oct 30 18:45:01.334075 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.333645 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfq4p\" (UniqueName: \"kubernetes.io/projected/ed9e3fd1-ccad-4d33-bb46-222acf609143-kube-api-access-bfq4p\") pod \"maas-api-key-cleanup-e2e-worker-w1-380c08-29889765-4vxnf\" (UID: \"ed9e3fd1-ccad-4d33-bb46-222acf609143\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889765-4vxnf" Oct 30 18:45:01.334075 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.333791 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b700c35-c957-4b7a-882e-bf82dd79f7ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-39c99d-29889765-tdq6b\" (UID: \"9b700c35-c957-4b7a-882e-bf82dd79f7ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889765-tdq6b" Oct 30 18:45:01.334075 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.333916 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rngx\" (UniqueName: \"kubernetes.io/projected/f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04-kube-api-access-7rngx\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889765-mx8tm\" (UID: \"f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889765-mx8tm" Oct 30 18:45:01.334075 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.333994 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh5qb\" (UniqueName: \"kubernetes.io/projected/9b700c35-c957-4b7a-882e-bf82dd79f7ff-kube-api-access-wh5qb\") pod \"maas-api-key-cleanup-e2e-extmod-b-39c99d-29889765-tdq6b\" (UID: \"9b700c35-c957-4b7a-882e-bf82dd79f7ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889765-tdq6b" Oct 30 18:45:01.334075 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.334057 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed9e3fd1-ccad-4d33-bb46-222acf609143-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-380c08-29889765-4vxnf\" (UID: \"ed9e3fd1-ccad-4d33-bb46-222acf609143\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889765-4vxnf" Oct 30 18:45:01.337408 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.337374 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed9e3fd1-ccad-4d33-bb46-222acf609143-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-380c08-29889765-4vxnf\" (UID: \"ed9e3fd1-ccad-4d33-bb46-222acf609143\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889765-4vxnf" Oct 30 18:45:01.356808 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.356731 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889765-k88tq"] Oct 30 18:45:01.357794 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.357742 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfq4p\" (UniqueName: \"kubernetes.io/projected/ed9e3fd1-ccad-4d33-bb46-222acf609143-kube-api-access-bfq4p\") pod \"maas-api-key-cleanup-e2e-worker-w1-380c08-29889765-4vxnf\" (UID: \"ed9e3fd1-ccad-4d33-bb46-222acf609143\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889765-4vxnf" Oct 30 18:45:01.442109 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.436419 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889765-mx8tm\" (UID: \"f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889765-mx8tm" Oct 30 18:45:01.442109 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.436533 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b700c35-c957-4b7a-882e-bf82dd79f7ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-39c99d-29889765-tdq6b\" (UID: \"9b700c35-c957-4b7a-882e-bf82dd79f7ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889765-tdq6b" Oct 30 18:45:01.442109 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.436712 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rngx\" (UniqueName: \"kubernetes.io/projected/f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04-kube-api-access-7rngx\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889765-mx8tm\" (UID: \"f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889765-mx8tm" Oct 30 18:45:01.442109 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.436808 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh5qb\" (UniqueName: \"kubernetes.io/projected/9b700c35-c957-4b7a-882e-bf82dd79f7ff-kube-api-access-wh5qb\") pod \"maas-api-key-cleanup-e2e-extmod-b-39c99d-29889765-tdq6b\" (UID: \"9b700c35-c957-4b7a-882e-bf82dd79f7ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889765-tdq6b" Oct 30 18:45:01.442109 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.438152 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889765-mx8tm\" (UID: \"f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889765-mx8tm" Oct 30 18:45:01.442109 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.438802 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b700c35-c957-4b7a-882e-bf82dd79f7ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-39c99d-29889765-tdq6b\" (UID: \"9b700c35-c957-4b7a-882e-bf82dd79f7ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889765-tdq6b" Oct 30 18:45:01.460113 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.460079 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh5qb\" (UniqueName: \"kubernetes.io/projected/9b700c35-c957-4b7a-882e-bf82dd79f7ff-kube-api-access-wh5qb\") pod \"maas-api-key-cleanup-e2e-extmod-b-39c99d-29889765-tdq6b\" (UID: \"9b700c35-c957-4b7a-882e-bf82dd79f7ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889765-tdq6b" Oct 30 18:45:01.461100 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.461048 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rngx\" (UniqueName: \"kubernetes.io/projected/f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04-kube-api-access-7rngx\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889765-mx8tm\" (UID: \"f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889765-mx8tm" Oct 30 18:45:01.561518 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.561233 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889765-4vxnf" Oct 30 18:45:01.573221 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.573189 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889765-tdq6b" Oct 30 18:45:01.605269 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.605235 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889765-ljqcz"] Oct 30 18:45:01.624198 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:45:01.624160 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f486a50_76a3_4cc7_bd5b_3edb80ad3315.slice/crio-5b10b02d3d39d8920bf7adcefe233ec65b148f0eaf965cd63f6fd1f6bd8d522d WatchSource:0}: Error finding container 5b10b02d3d39d8920bf7adcefe233ec65b148f0eaf965cd63f6fd1f6bd8d522d: Status 404 returned error can't find the container with id 5b10b02d3d39d8920bf7adcefe233ec65b148f0eaf965cd63f6fd1f6bd8d522d Oct 30 18:45:01.637825 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.637796 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889765-mx8tm" Oct 30 18:45:01.720691 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:45:01.720635 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83b551f2_22ff_42f3_acc9_3122313400d8.slice/crio-d8d7e90e9771d4c6b2c413e11ad8fc957b2f00024ba604f04f8af2bbb5d51d0d WatchSource:0}: Error finding container d8d7e90e9771d4c6b2c413e11ad8fc957b2f00024ba604f04f8af2bbb5d51d0d: Status 404 returned error can't find the container with id d8d7e90e9771d4c6b2c413e11ad8fc957b2f00024ba604f04f8af2bbb5d51d0d Oct 30 18:45:01.722368 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.722215 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889765-dmf99"] Oct 30 18:45:01.782204 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.782169 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889765-v4sqc"] Oct 30 18:45:01.825225 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.825196 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889765-rrzvv"] Oct 30 18:45:01.828089 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:45:01.827681 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ee9297b_cfb8_4154_bb20_d5eb73f78da2.slice/crio-0939bff7932657b465de4a683fa026813f711b5bc73b1dafea5cf7ff54d7d917 WatchSource:0}: Error finding container 0939bff7932657b465de4a683fa026813f711b5bc73b1dafea5cf7ff54d7d917: Status 404 returned error can't find the container with id 0939bff7932657b465de4a683fa026813f711b5bc73b1dafea5cf7ff54d7d917 Oct 30 18:45:01.830429 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:45:01.830403 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d52db28_5cd7_4a2a_af8b_b876d41c8453.slice/crio-694a8de4c6e76e62888cb9eb2aae58c48e7167b5409c59a123a062f86f38fa99 WatchSource:0}: Error finding container 694a8de4c6e76e62888cb9eb2aae58c48e7167b5409c59a123a062f86f38fa99: Status 404 returned error can't find the container with id 694a8de4c6e76e62888cb9eb2aae58c48e7167b5409c59a123a062f86f38fa99 Oct 30 18:45:01.867215 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.867120 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889765-qf9bg"] Oct 30 18:45:01.927165 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.926916 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889765-v5zh2"] Oct 30 18:45:01.927618 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.927528 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889765-dmf99" event={"ID":"83b551f2-22ff-42f3-acc9-3122313400d8","Type":"ContainerStarted","Data":"d8d7e90e9771d4c6b2c413e11ad8fc957b2f00024ba604f04f8af2bbb5d51d0d"} Oct 30 18:45:01.942428 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.940795 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889765-k88tq" event={"ID":"1e4c2309-c818-46be-abd4-add20576fd6a","Type":"ContainerStarted","Data":"da0fa2f2ebcb514f755dc68054509d95548012042140d1034eece63165aae2dc"} Oct 30 18:45:01.955385 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.955326 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889765-b2hth"] Oct 30 18:45:01.956558 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.956523 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889765-rrzvv" event={"ID":"7d52db28-5cd7-4a2a-af8b-b876d41c8453","Type":"ContainerStarted","Data":"694a8de4c6e76e62888cb9eb2aae58c48e7167b5409c59a123a062f86f38fa99"} Oct 30 18:45:01.959680 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.959640 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889765-v4sqc" event={"ID":"8ee9297b-cfb8-4154-bb20-d5eb73f78da2","Type":"ContainerStarted","Data":"0939bff7932657b465de4a683fa026813f711b5bc73b1dafea5cf7ff54d7d917"} Oct 30 18:45:01.970989 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.970954 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889765-qf9bg" event={"ID":"a87157ec-dde9-4537-a2dc-d2a0e0c80091","Type":"ContainerStarted","Data":"64b6556e2f19dab5db9865b6db6974e103a555f41eb8c9aceadb86a6c3415361"} Oct 30 18:45:01.972610 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.972585 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889765-4vxnf"] Oct 30 18:45:01.988254 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.978187 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889765-ljqcz" event={"ID":"4f486a50-76a3-4cc7-bd5b-3edb80ad3315","Type":"ContainerStarted","Data":"5b10b02d3d39d8920bf7adcefe233ec65b148f0eaf965cd63f6fd1f6bd8d522d"} Oct 30 18:45:01.988254 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.986093 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889765-tdq6b"] Oct 30 18:45:01.994356 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:01.993387 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889765-fsnp2"] Oct 30 18:45:02.033593 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:02.033567 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889765-mx8tm"] Oct 30 18:45:02.988856 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:02.988814 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889765-tdq6b" event={"ID":"9b700c35-c957-4b7a-882e-bf82dd79f7ff","Type":"ContainerStarted","Data":"c6b6bacce38866c4b4a86dce4d1a1421edfbc50aff37f39f5315b48f470651b5"} Oct 30 18:45:02.991638 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:02.991607 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889765-fsnp2" event={"ID":"f7cdca9f-f772-449c-9f30-067437ef08cc","Type":"ContainerStarted","Data":"16a09ff3b8ced965d48c79366502e04ecf5b230fef0f5426e0e480d1864b68fb"} Oct 30 18:45:02.994468 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:02.994434 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889765-dmf99" event={"ID":"83b551f2-22ff-42f3-acc9-3122313400d8","Type":"ContainerStarted","Data":"e6215330d38d2b90975cfa58fc96a91067280d1fa65da06974693cdc505a6df2"} Oct 30 18:45:02.997862 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:02.997781 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889765-k88tq" event={"ID":"1e4c2309-c818-46be-abd4-add20576fd6a","Type":"ContainerStarted","Data":"86362f1e9b18ee11357a7a3cf4e94646bbd0a1c96124dca68a3be034e4575eb3"} Oct 30 18:45:03.000594 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:03.000539 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889765-rrzvv" event={"ID":"7d52db28-5cd7-4a2a-af8b-b876d41c8453","Type":"ContainerStarted","Data":"39da0d4efbe78a0fc9d767b6ec81556297815fc2ccb6afaf797442fbdb816760"} Oct 30 18:45:03.003050 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:03.003020 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889765-b2hth" event={"ID":"36a43766-ec94-4991-bad0-e3d4ed5122c8","Type":"ContainerStarted","Data":"bdb59a9ea1af6e330257e87e382b2e52b9d2ff1e7baf8fd4fb86b0c7b1242b64"} Oct 30 18:45:03.003203 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:03.003082 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889765-b2hth" event={"ID":"36a43766-ec94-4991-bad0-e3d4ed5122c8","Type":"ContainerStarted","Data":"def354d8bcc227a9c5bcafb973cfafb895c3a04a36e7e2acaa13ec1dd386e328"} Oct 30 18:45:03.005201 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:03.005166 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889765-v4sqc" event={"ID":"8ee9297b-cfb8-4154-bb20-d5eb73f78da2","Type":"ContainerStarted","Data":"736e3da2e61c81d14671dd5a8390bfef9432def38f8d7c26b3cbb7fc691d8aa2"} Oct 30 18:45:03.006780 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:03.006755 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889765-mx8tm" event={"ID":"f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04","Type":"ContainerStarted","Data":"5c3d22b24b22f176346ac64b7d896bb77aedbef573ebedf604f62cdb8d920c9c"} Oct 30 18:45:03.008647 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:03.008618 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889765-4vxnf" event={"ID":"ed9e3fd1-ccad-4d33-bb46-222acf609143","Type":"ContainerStarted","Data":"f0d4dbcbc26b71f38ddbc63c76d2ce73a1fb22db227c15d9d1153837a8b1213b"} Oct 30 18:45:03.008787 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:03.008674 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889765-4vxnf" event={"ID":"ed9e3fd1-ccad-4d33-bb46-222acf609143","Type":"ContainerStarted","Data":"b9dfaa43bd276ef66d2b643705172831f5a69e5c73cab20244ffcf858bb1516b"} Oct 30 18:45:03.011283 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:03.011252 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889765-v5zh2" event={"ID":"5e752ff2-638c-4b1f-94a9-5efd1a53a9c6","Type":"ContainerStarted","Data":"58abaeca95c993db663598d3a1dd5e2e50fb5d081183ed8224fb0739ae13dcf3"} Oct 30 18:45:03.011410 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:03.011308 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889765-v5zh2" event={"ID":"5e752ff2-638c-4b1f-94a9-5efd1a53a9c6","Type":"ContainerStarted","Data":"7b7b0b2ff5e88ed2660835a323fc5ba32e3ea4c77f666ecd53f42d956af00d04"} Oct 30 18:45:03.013743 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:03.013711 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889765-qf9bg" event={"ID":"a87157ec-dde9-4537-a2dc-d2a0e0c80091","Type":"ContainerStarted","Data":"8312e41330ed4fcb47e62f8bc78c90014b9810f2dc3ff1ac3071a58077c05431"} Oct 30 18:45:03.015888 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:03.015831 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889765-ljqcz" event={"ID":"4f486a50-76a3-4cc7-bd5b-3edb80ad3315","Type":"ContainerStarted","Data":"74df7368b4df5a96e1b9603d6645d33cfec7c957bccf8141c0cc9a57f91adfb1"} Oct 30 18:45:03.040050 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:03.039993 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889765-dmf99" podStartSLOduration=3.039977021 podStartE2EDuration="3.039977021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:03.034013037 +0000 UTC m=+2800.124075402" watchObservedRunningTime="2026-10-30 18:45:03.039977021 +0000 UTC m=+2800.130039404" Oct 30 18:45:03.071970 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:03.071918 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889765-v4sqc" podStartSLOduration=3.071904867 podStartE2EDuration="3.071904867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:03.071273848 +0000 UTC m=+2800.161336189" watchObservedRunningTime="2026-10-30 18:45:03.071904867 +0000 UTC m=+2800.161967219" Oct 30 18:45:03.109151 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:03.109073 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889765-qf9bg" podStartSLOduration=3.109059763 podStartE2EDuration="3.109059763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:03.107942949 +0000 UTC m=+2800.198005287" watchObservedRunningTime="2026-10-30 18:45:03.109059763 +0000 UTC m=+2800.199122121" Oct 30 18:45:03.155808 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:03.155763 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889765-rrzvv" podStartSLOduration=3.155746582 podStartE2EDuration="3.155746582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:03.152745225 +0000 UTC m=+2800.242807563" watchObservedRunningTime="2026-10-30 18:45:03.155746582 +0000 UTC m=+2800.245808921" Oct 30 18:45:03.196987 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:03.196899 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889765-4vxnf" podStartSLOduration=3.196882956 podStartE2EDuration="3.196882956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:03.191093116 +0000 UTC m=+2800.281155456" watchObservedRunningTime="2026-10-30 18:45:03.196882956 +0000 UTC m=+2800.286945289" Oct 30 18:45:03.234465 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:03.234416 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889765-ljqcz" podStartSLOduration=3.234389851 podStartE2EDuration="3.234389851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:03.229780842 +0000 UTC m=+2800.319843183" watchObservedRunningTime="2026-10-30 18:45:03.234389851 +0000 UTC m=+2800.324452191" Oct 30 18:45:03.272576 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:03.272522 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889765-v5zh2" podStartSLOduration=3.272508437 podStartE2EDuration="3.272508437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:03.269319414 +0000 UTC m=+2800.359381751" watchObservedRunningTime="2026-10-30 18:45:03.272508437 +0000 UTC m=+2800.362570775" Oct 30 18:45:03.311566 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:03.311324 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889765-b2hth" podStartSLOduration=3.311307927 podStartE2EDuration="3.311307927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:03.308325284 +0000 UTC m=+2800.398387626" watchObservedRunningTime="2026-10-30 18:45:03.311307927 +0000 UTC m=+2800.401370266" Oct 30 18:45:03.360895 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:03.356057 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889765-k88tq" podStartSLOduration=3.3560375479999998 podStartE2EDuration="3.356037548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:03.353510114 +0000 UTC m=+2800.443572461" watchObservedRunningTime="2026-10-30 18:45:03.356037548 +0000 UTC m=+2800.446099894" Oct 30 18:45:04.028070 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:04.027973 4442 generic.go:356] "Generic (PLEG): container finished" podID="83b551f2-22ff-42f3-acc9-3122313400d8" containerID="e6215330d38d2b90975cfa58fc96a91067280d1fa65da06974693cdc505a6df2" exitCode=60 Oct 30 18:45:04.028070 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:04.028052 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889765-dmf99" event={"ID":"83b551f2-22ff-42f3-acc9-3122313400d8","Type":"ContainerDied","Data":"e6215330d38d2b90975cfa58fc96a91067280d1fa65da06974693cdc505a6df2"} Oct 30 18:45:04.028580 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:04.028395 4442 scope.go:122] "RemoveContainer" containerID="e6215330d38d2b90975cfa58fc96a91067280d1fa65da06974693cdc505a6df2" Oct 30 18:45:04.030286 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:04.030261 4442 generic.go:356] "Generic (PLEG): container finished" podID="1e4c2309-c818-46be-abd4-add20576fd6a" containerID="86362f1e9b18ee11357a7a3cf4e94646bbd0a1c96124dca68a3be034e4575eb3" exitCode=60 Oct 30 18:45:04.030407 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:04.030343 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889765-k88tq" event={"ID":"1e4c2309-c818-46be-abd4-add20576fd6a","Type":"ContainerDied","Data":"86362f1e9b18ee11357a7a3cf4e94646bbd0a1c96124dca68a3be034e4575eb3"} Oct 30 18:45:04.030645 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:04.030627 4442 scope.go:122] "RemoveContainer" containerID="86362f1e9b18ee11357a7a3cf4e94646bbd0a1c96124dca68a3be034e4575eb3" Oct 30 18:45:04.032994 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:04.032972 4442 generic.go:356] "Generic (PLEG): container finished" podID="7d52db28-5cd7-4a2a-af8b-b876d41c8453" containerID="39da0d4efbe78a0fc9d767b6ec81556297815fc2ccb6afaf797442fbdb816760" exitCode=60 Oct 30 18:45:04.033089 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:04.033072 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889765-rrzvv" event={"ID":"7d52db28-5cd7-4a2a-af8b-b876d41c8453","Type":"ContainerDied","Data":"39da0d4efbe78a0fc9d767b6ec81556297815fc2ccb6afaf797442fbdb816760"} Oct 30 18:45:04.033304 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:04.033284 4442 scope.go:122] "RemoveContainer" containerID="39da0d4efbe78a0fc9d767b6ec81556297815fc2ccb6afaf797442fbdb816760" Oct 30 18:45:04.036004 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:04.035987 4442 generic.go:356] "Generic (PLEG): container finished" podID="36a43766-ec94-4991-bad0-e3d4ed5122c8" containerID="bdb59a9ea1af6e330257e87e382b2e52b9d2ff1e7baf8fd4fb86b0c7b1242b64" exitCode=60 Oct 30 18:45:04.036109 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:04.036069 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889765-b2hth" event={"ID":"36a43766-ec94-4991-bad0-e3d4ed5122c8","Type":"ContainerDied","Data":"bdb59a9ea1af6e330257e87e382b2e52b9d2ff1e7baf8fd4fb86b0c7b1242b64"} Oct 30 18:45:04.036831 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:04.036813 4442 scope.go:122] "RemoveContainer" containerID="bdb59a9ea1af6e330257e87e382b2e52b9d2ff1e7baf8fd4fb86b0c7b1242b64" Oct 30 18:45:04.038560 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:04.038538 4442 generic.go:356] "Generic (PLEG): container finished" podID="8ee9297b-cfb8-4154-bb20-d5eb73f78da2" containerID="736e3da2e61c81d14671dd5a8390bfef9432def38f8d7c26b3cbb7fc691d8aa2" exitCode=60 Oct 30 18:45:04.038655 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:04.038619 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889765-v4sqc" event={"ID":"8ee9297b-cfb8-4154-bb20-d5eb73f78da2","Type":"ContainerDied","Data":"736e3da2e61c81d14671dd5a8390bfef9432def38f8d7c26b3cbb7fc691d8aa2"} Oct 30 18:45:04.038828 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:04.038814 4442 scope.go:122] "RemoveContainer" containerID="736e3da2e61c81d14671dd5a8390bfef9432def38f8d7c26b3cbb7fc691d8aa2" Oct 30 18:45:04.040903 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:04.040877 4442 generic.go:356] "Generic (PLEG): container finished" podID="f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04" containerID="41abff661a52a5819ccb9e80529c2fb948aaadbaec802fcee07ee910b56fc1f6" exitCode=60 Oct 30 18:45:04.040991 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:04.040955 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889765-mx8tm" event={"ID":"f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04","Type":"ContainerDied","Data":"41abff661a52a5819ccb9e80529c2fb948aaadbaec802fcee07ee910b56fc1f6"} Oct 30 18:45:04.041209 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:04.041189 4442 scope.go:122] "RemoveContainer" containerID="41abff661a52a5819ccb9e80529c2fb948aaadbaec802fcee07ee910b56fc1f6" Oct 30 18:45:04.044771 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:04.044748 4442 generic.go:356] "Generic (PLEG): container finished" podID="ed9e3fd1-ccad-4d33-bb46-222acf609143" containerID="f0d4dbcbc26b71f38ddbc63c76d2ce73a1fb22db227c15d9d1153837a8b1213b" exitCode=60 Oct 30 18:45:04.045083 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:04.045065 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889765-4vxnf" event={"ID":"ed9e3fd1-ccad-4d33-bb46-222acf609143","Type":"ContainerDied","Data":"f0d4dbcbc26b71f38ddbc63c76d2ce73a1fb22db227c15d9d1153837a8b1213b"} Oct 30 18:45:04.045185 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:04.045108 4442 scope.go:122] "RemoveContainer" containerID="f0d4dbcbc26b71f38ddbc63c76d2ce73a1fb22db227c15d9d1153837a8b1213b" Oct 30 18:45:04.047592 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:04.047557 4442 generic.go:356] "Generic (PLEG): container finished" podID="5e752ff2-638c-4b1f-94a9-5efd1a53a9c6" containerID="58abaeca95c993db663598d3a1dd5e2e50fb5d081183ed8224fb0739ae13dcf3" exitCode=60 Oct 30 18:45:04.047691 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:04.047649 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889765-v5zh2" event={"ID":"5e752ff2-638c-4b1f-94a9-5efd1a53a9c6","Type":"ContainerDied","Data":"58abaeca95c993db663598d3a1dd5e2e50fb5d081183ed8224fb0739ae13dcf3"} Oct 30 18:45:04.047842 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:04.047829 4442 scope.go:122] "RemoveContainer" containerID="58abaeca95c993db663598d3a1dd5e2e50fb5d081183ed8224fb0739ae13dcf3" Oct 30 18:45:04.049467 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:04.049450 4442 generic.go:356] "Generic (PLEG): container finished" podID="a87157ec-dde9-4537-a2dc-d2a0e0c80091" containerID="8312e41330ed4fcb47e62f8bc78c90014b9810f2dc3ff1ac3071a58077c05431" exitCode=60 Oct 30 18:45:04.049561 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:04.049533 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889765-qf9bg" event={"ID":"a87157ec-dde9-4537-a2dc-d2a0e0c80091","Type":"ContainerDied","Data":"8312e41330ed4fcb47e62f8bc78c90014b9810f2dc3ff1ac3071a58077c05431"} Oct 30 18:45:04.049730 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:04.049713 4442 scope.go:122] "RemoveContainer" containerID="8312e41330ed4fcb47e62f8bc78c90014b9810f2dc3ff1ac3071a58077c05431" Oct 30 18:45:04.052298 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:04.052280 4442 generic.go:356] "Generic (PLEG): container finished" podID="4f486a50-76a3-4cc7-bd5b-3edb80ad3315" containerID="74df7368b4df5a96e1b9603d6645d33cfec7c957bccf8141c0cc9a57f91adfb1" exitCode=60 Oct 30 18:45:04.052569 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:04.052552 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889765-ljqcz" event={"ID":"4f486a50-76a3-4cc7-bd5b-3edb80ad3315","Type":"ContainerDied","Data":"74df7368b4df5a96e1b9603d6645d33cfec7c957bccf8141c0cc9a57f91adfb1"} Oct 30 18:45:04.053258 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:04.053237 4442 scope.go:122] "RemoveContainer" containerID="74df7368b4df5a96e1b9603d6645d33cfec7c957bccf8141c0cc9a57f91adfb1" Oct 30 18:45:04.076375 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:04.076346 4442 generic.go:356] "Generic (PLEG): container finished" podID="9b700c35-c957-4b7a-882e-bf82dd79f7ff" containerID="a31c6c3e10b000d441bb7eff969c9a6a416d4be8d5b7250bde7096a6e4dc43bc" exitCode=60 Oct 30 18:45:04.076582 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:04.076504 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889765-tdq6b" event={"ID":"9b700c35-c957-4b7a-882e-bf82dd79f7ff","Type":"ContainerDied","Data":"a31c6c3e10b000d441bb7eff969c9a6a416d4be8d5b7250bde7096a6e4dc43bc"} Oct 30 18:45:04.076822 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:04.076802 4442 scope.go:122] "RemoveContainer" containerID="a31c6c3e10b000d441bb7eff969c9a6a416d4be8d5b7250bde7096a6e4dc43bc" Oct 30 18:45:04.090349 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:04.082273 4442 generic.go:356] "Generic (PLEG): container finished" podID="f7cdca9f-f772-449c-9f30-067437ef08cc" containerID="52154690db6cbf1cf9a62d8f5184b6c0d0d329ae961f40ba31c34749287510ec" exitCode=60 Oct 30 18:45:04.090349 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:04.082328 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889765-fsnp2" event={"ID":"f7cdca9f-f772-449c-9f30-067437ef08cc","Type":"ContainerDied","Data":"52154690db6cbf1cf9a62d8f5184b6c0d0d329ae961f40ba31c34749287510ec"} Oct 30 18:45:04.090349 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:04.082841 4442 scope.go:122] "RemoveContainer" containerID="52154690db6cbf1cf9a62d8f5184b6c0d0d329ae961f40ba31c34749287510ec" Oct 30 18:45:05.096091 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:05.096063 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889765-dmf99" event={"ID":"83b551f2-22ff-42f3-acc9-3122313400d8","Type":"ContainerStarted","Data":"98ff3ff69272788743dbb56c1a20bb502a540a9bc551a166c24993835627a09f"} Oct 30 18:45:05.099374 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:05.098728 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889765-k88tq" event={"ID":"1e4c2309-c818-46be-abd4-add20576fd6a","Type":"ContainerStarted","Data":"44943ed079b62965fb591d484fd63b4c53c78c75fdbcd3c2f40affdb2aef09f8"} Oct 30 18:45:05.101494 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:05.101464 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889765-rrzvv" event={"ID":"7d52db28-5cd7-4a2a-af8b-b876d41c8453","Type":"ContainerStarted","Data":"d8a73df452867f89bf3b4b0c311484a6d826c64b0c3cda8765885bb800b8cba1"} Oct 30 18:45:05.103608 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:05.103587 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889765-b2hth" event={"ID":"36a43766-ec94-4991-bad0-e3d4ed5122c8","Type":"ContainerStarted","Data":"cccd15d88f9e4c84a761b496a7f08c1367905e50b5cac64a0d9305cf54961ccd"} Oct 30 18:45:05.105748 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:05.105727 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889765-mx8tm" event={"ID":"f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04","Type":"ContainerStarted","Data":"e08172a3f0aa71f6f2455dce0cb4d5685badc0d03db7bd503bd890e6ddef01e2"} Oct 30 18:45:05.108126 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:05.108104 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889765-4vxnf" event={"ID":"ed9e3fd1-ccad-4d33-bb46-222acf609143","Type":"ContainerStarted","Data":"f91f5bedb43bfaef4cd4b7dddc5efc26b730a435d8fdaa2f2157b7135e7420fa"} Oct 30 18:45:05.110224 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:05.110200 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889765-tdq6b" event={"ID":"9b700c35-c957-4b7a-882e-bf82dd79f7ff","Type":"ContainerStarted","Data":"7509658e429fd182088c97d6645f06696bc9106721716c3d634ee479d7af0295"} Oct 30 18:45:05.255205 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:05.255158 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889765-mx8tm" podStartSLOduration=5.255128286 podStartE2EDuration="5.255128286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:05.252090854 +0000 UTC m=+2802.342153196" watchObservedRunningTime="2026-10-30 18:45:05.255128286 +0000 UTC m=+2802.345190624" Oct 30 18:45:05.381770 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:05.381696 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889765-tdq6b" podStartSLOduration=5.381679635 podStartE2EDuration="5.381679635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:05.38048568 +0000 UTC m=+2802.470548029" watchObservedRunningTime="2026-10-30 18:45:05.381679635 +0000 UTC m=+2802.471741975" Oct 30 18:45:06.134622 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:06.134576 4442 generic.go:356] "Generic (PLEG): container finished" podID="8ee9297b-cfb8-4154-bb20-d5eb73f78da2" containerID="d6e950026f01c4eda1ef5738a6d4a98f26d4b3e2e922e42cfa737b76369b6378" exitCode=60 Oct 30 18:45:06.135206 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:06.134694 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889765-v4sqc" event={"ID":"8ee9297b-cfb8-4154-bb20-d5eb73f78da2","Type":"ContainerDied","Data":"d6e950026f01c4eda1ef5738a6d4a98f26d4b3e2e922e42cfa737b76369b6378"} Oct 30 18:45:06.135206 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:06.134800 4442 scope.go:122] "RemoveContainer" containerID="736e3da2e61c81d14671dd5a8390bfef9432def38f8d7c26b3cbb7fc691d8aa2" Oct 30 18:45:06.135529 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:06.135504 4442 scope.go:122] "RemoveContainer" containerID="d6e950026f01c4eda1ef5738a6d4a98f26d4b3e2e922e42cfa737b76369b6378" Oct 30 18:45:06.136101 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:45:06.136070 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-921450-29889765-v4sqc_odh-ai-gateway-infra(8ee9297b-cfb8-4154-bb20-d5eb73f78da2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889765-v4sqc" podUID="8ee9297b-cfb8-4154-bb20-d5eb73f78da2" Oct 30 18:45:06.152059 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:06.152014 4442 generic.go:356] "Generic (PLEG): container finished" podID="f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04" containerID="e08172a3f0aa71f6f2455dce0cb4d5685badc0d03db7bd503bd890e6ddef01e2" exitCode=60 Oct 30 18:45:06.152265 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:06.152200 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889765-mx8tm" event={"ID":"f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04","Type":"ContainerDied","Data":"e08172a3f0aa71f6f2455dce0cb4d5685badc0d03db7bd503bd890e6ddef01e2"} Oct 30 18:45:06.152572 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:06.152553 4442 scope.go:122] "RemoveContainer" containerID="e08172a3f0aa71f6f2455dce0cb4d5685badc0d03db7bd503bd890e6ddef01e2" Oct 30 18:45:06.152943 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:45:06.152916 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889765-mx8tm_odh-ai-gateway-infra(f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889765-mx8tm" podUID="f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04" Oct 30 18:45:06.154844 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:06.154782 4442 generic.go:356] "Generic (PLEG): container finished" podID="ed9e3fd1-ccad-4d33-bb46-222acf609143" containerID="f91f5bedb43bfaef4cd4b7dddc5efc26b730a435d8fdaa2f2157b7135e7420fa" exitCode=60 Oct 30 18:45:06.154963 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:06.154863 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889765-4vxnf" event={"ID":"ed9e3fd1-ccad-4d33-bb46-222acf609143","Type":"ContainerDied","Data":"f91f5bedb43bfaef4cd4b7dddc5efc26b730a435d8fdaa2f2157b7135e7420fa"} Oct 30 18:45:06.155165 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:06.155123 4442 scope.go:122] "RemoveContainer" containerID="f91f5bedb43bfaef4cd4b7dddc5efc26b730a435d8fdaa2f2157b7135e7420fa" Oct 30 18:45:06.155644 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:45:06.155613 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-380c08-29889765-4vxnf_odh-ai-gateway-infra(ed9e3fd1-ccad-4d33-bb46-222acf609143)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889765-4vxnf" podUID="ed9e3fd1-ccad-4d33-bb46-222acf609143" Oct 30 18:45:06.157339 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:06.157318 4442 generic.go:356] "Generic (PLEG): container finished" podID="5e752ff2-638c-4b1f-94a9-5efd1a53a9c6" containerID="82b5d86cd4e85307d629bff8c3b20322f75b4367700289d0fc9ed986c9e96a39" exitCode=60 Oct 30 18:45:06.157423 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:06.157408 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889765-v5zh2" event={"ID":"5e752ff2-638c-4b1f-94a9-5efd1a53a9c6","Type":"ContainerDied","Data":"82b5d86cd4e85307d629bff8c3b20322f75b4367700289d0fc9ed986c9e96a39"} Oct 30 18:45:06.157640 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:06.157623 4442 scope.go:122] "RemoveContainer" containerID="82b5d86cd4e85307d629bff8c3b20322f75b4367700289d0fc9ed986c9e96a39" Oct 30 18:45:06.157940 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:45:06.157920 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889765-v5zh2_odh-ai-gateway-infra(5e752ff2-638c-4b1f-94a9-5efd1a53a9c6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889765-v5zh2" podUID="5e752ff2-638c-4b1f-94a9-5efd1a53a9c6" Oct 30 18:45:06.159849 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:06.159824 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889765-qf9bg" event={"ID":"a87157ec-dde9-4537-a2dc-d2a0e0c80091","Type":"ContainerStarted","Data":"6cdc88b466c945853f123c462578209b8e019e59a1cbcaedbdf47a7f13ca306a"} Oct 30 18:45:06.161851 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:06.161829 4442 generic.go:356] "Generic (PLEG): container finished" podID="4f486a50-76a3-4cc7-bd5b-3edb80ad3315" containerID="ae36d143affd0c63101f4bdd97eef4452f484693a8e573abad5a8d8f95414732" exitCode=60 Oct 30 18:45:06.161962 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:06.161912 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889765-ljqcz" event={"ID":"4f486a50-76a3-4cc7-bd5b-3edb80ad3315","Type":"ContainerDied","Data":"ae36d143affd0c63101f4bdd97eef4452f484693a8e573abad5a8d8f95414732"} Oct 30 18:45:06.162097 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:06.162082 4442 scope.go:122] "RemoveContainer" containerID="ae36d143affd0c63101f4bdd97eef4452f484693a8e573abad5a8d8f95414732" Oct 30 18:45:06.162315 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:45:06.162298 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889765-ljqcz_odh-ai-gateway-infra(4f486a50-76a3-4cc7-bd5b-3edb80ad3315)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889765-ljqcz" podUID="4f486a50-76a3-4cc7-bd5b-3edb80ad3315" Oct 30 18:45:06.163896 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:06.163874 4442 generic.go:356] "Generic (PLEG): container finished" podID="9b700c35-c957-4b7a-882e-bf82dd79f7ff" containerID="7509658e429fd182088c97d6645f06696bc9106721716c3d634ee479d7af0295" exitCode=60 Oct 30 18:45:06.163995 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:06.163952 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889765-tdq6b" event={"ID":"9b700c35-c957-4b7a-882e-bf82dd79f7ff","Type":"ContainerDied","Data":"7509658e429fd182088c97d6645f06696bc9106721716c3d634ee479d7af0295"} Oct 30 18:45:06.164181 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:06.164149 4442 scope.go:122] "RemoveContainer" containerID="7509658e429fd182088c97d6645f06696bc9106721716c3d634ee479d7af0295" Oct 30 18:45:06.164387 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:45:06.164365 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-39c99d-29889765-tdq6b_odh-ai-gateway-infra(9b700c35-c957-4b7a-882e-bf82dd79f7ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889765-tdq6b" podUID="9b700c35-c957-4b7a-882e-bf82dd79f7ff" Oct 30 18:45:06.165992 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:06.165965 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889765-fsnp2" event={"ID":"f7cdca9f-f772-449c-9f30-067437ef08cc","Type":"ContainerStarted","Data":"2233ac20fc25a831f21873f1e5b7d33d724f15e370247c40d890e9b323ff150d"} Oct 30 18:45:06.167783 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:06.167761 4442 generic.go:356] "Generic (PLEG): container finished" podID="83b551f2-22ff-42f3-acc9-3122313400d8" containerID="98ff3ff69272788743dbb56c1a20bb502a540a9bc551a166c24993835627a09f" exitCode=60 Oct 30 18:45:06.167876 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:06.167830 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889765-dmf99" event={"ID":"83b551f2-22ff-42f3-acc9-3122313400d8","Type":"ContainerDied","Data":"98ff3ff69272788743dbb56c1a20bb502a540a9bc551a166c24993835627a09f"} Oct 30 18:45:06.168042 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:06.168028 4442 scope.go:122] "RemoveContainer" containerID="98ff3ff69272788743dbb56c1a20bb502a540a9bc551a166c24993835627a09f" Oct 30 18:45:06.168378 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:45:06.168338 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-713dc8-29889765-dmf99_odh-ai-gateway-infra(83b551f2-22ff-42f3-acc9-3122313400d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889765-dmf99" podUID="83b551f2-22ff-42f3-acc9-3122313400d8" Oct 30 18:45:06.169786 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:06.169764 4442 generic.go:356] "Generic (PLEG): container finished" podID="1e4c2309-c818-46be-abd4-add20576fd6a" containerID="44943ed079b62965fb591d484fd63b4c53c78c75fdbcd3c2f40affdb2aef09f8" exitCode=60 Oct 30 18:45:06.170032 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:06.170007 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889765-k88tq" event={"ID":"1e4c2309-c818-46be-abd4-add20576fd6a","Type":"ContainerDied","Data":"44943ed079b62965fb591d484fd63b4c53c78c75fdbcd3c2f40affdb2aef09f8"} Oct 30 18:45:06.170165 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:06.170155 4442 scope.go:122] "RemoveContainer" containerID="44943ed079b62965fb591d484fd63b4c53c78c75fdbcd3c2f40affdb2aef09f8" Oct 30 18:45:06.170444 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:45:06.170417 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2107d4-29889765-k88tq_odh-ai-gateway-infra(1e4c2309-c818-46be-abd4-add20576fd6a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889765-k88tq" podUID="1e4c2309-c818-46be-abd4-add20576fd6a" Oct 30 18:45:06.175321 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:06.172812 4442 generic.go:356] "Generic (PLEG): container finished" podID="7d52db28-5cd7-4a2a-af8b-b876d41c8453" containerID="d8a73df452867f89bf3b4b0c311484a6d826c64b0c3cda8765885bb800b8cba1" exitCode=60 Oct 30 18:45:06.175321 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:06.172885 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889765-rrzvv" event={"ID":"7d52db28-5cd7-4a2a-af8b-b876d41c8453","Type":"ContainerDied","Data":"d8a73df452867f89bf3b4b0c311484a6d826c64b0c3cda8765885bb800b8cba1"} Oct 30 18:45:06.175321 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:06.173240 4442 scope.go:122] "RemoveContainer" containerID="d8a73df452867f89bf3b4b0c311484a6d826c64b0c3cda8765885bb800b8cba1" Oct 30 18:45:06.175321 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:45:06.173523 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-6babca-29889765-rrzvv_odh-ai-gateway-infra(7d52db28-5cd7-4a2a-af8b-b876d41c8453)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889765-rrzvv" podUID="7d52db28-5cd7-4a2a-af8b-b876d41c8453" Oct 30 18:45:06.176395 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:06.176372 4442 generic.go:356] "Generic (PLEG): container finished" podID="36a43766-ec94-4991-bad0-e3d4ed5122c8" containerID="cccd15d88f9e4c84a761b496a7f08c1367905e50b5cac64a0d9305cf54961ccd" exitCode=60 Oct 30 18:45:06.176639 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:06.176615 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889765-b2hth" event={"ID":"36a43766-ec94-4991-bad0-e3d4ed5122c8","Type":"ContainerDied","Data":"cccd15d88f9e4c84a761b496a7f08c1367905e50b5cac64a0d9305cf54961ccd"} Oct 30 18:45:06.177090 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:06.177069 4442 scope.go:122] "RemoveContainer" containerID="cccd15d88f9e4c84a761b496a7f08c1367905e50b5cac64a0d9305cf54961ccd" Oct 30 18:45:06.177476 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:45:06.177453 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889765-b2hth_odh-ai-gateway-infra(36a43766-ec94-4991-bad0-e3d4ed5122c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889765-b2hth" podUID="36a43766-ec94-4991-bad0-e3d4ed5122c8" Oct 30 18:45:06.258353 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:06.258331 4442 scope.go:122] "RemoveContainer" containerID="41abff661a52a5819ccb9e80529c2fb948aaadbaec802fcee07ee910b56fc1f6" Oct 30 18:45:06.274071 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:06.274047 4442 scope.go:122] "RemoveContainer" containerID="f0d4dbcbc26b71f38ddbc63c76d2ce73a1fb22db227c15d9d1153837a8b1213b" Oct 30 18:45:06.290639 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:06.290612 4442 scope.go:122] "RemoveContainer" containerID="58abaeca95c993db663598d3a1dd5e2e50fb5d081183ed8224fb0739ae13dcf3" Oct 30 18:45:06.294488 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:06.294446 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889765-fsnp2" podStartSLOduration=6.294431434 podStartE2EDuration="6.294431434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:06.28907249 +0000 UTC m=+2803.379134829" watchObservedRunningTime="2026-10-30 18:45:06.294431434 +0000 UTC m=+2803.384493773" Oct 30 18:45:06.308288 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:06.308268 4442 scope.go:122] "RemoveContainer" containerID="74df7368b4df5a96e1b9603d6645d33cfec7c957bccf8141c0cc9a57f91adfb1" Oct 30 18:45:06.325735 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:06.325609 4442 scope.go:122] "RemoveContainer" containerID="a31c6c3e10b000d441bb7eff969c9a6a416d4be8d5b7250bde7096a6e4dc43bc" Oct 30 18:45:06.489258 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:06.489232 4442 scope.go:122] "RemoveContainer" containerID="e6215330d38d2b90975cfa58fc96a91067280d1fa65da06974693cdc505a6df2" Oct 30 18:45:06.504985 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:06.504958 4442 scope.go:122] "RemoveContainer" containerID="86362f1e9b18ee11357a7a3cf4e94646bbd0a1c96124dca68a3be034e4575eb3" Oct 30 18:45:06.521924 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:06.521900 4442 scope.go:122] "RemoveContainer" containerID="39da0d4efbe78a0fc9d767b6ec81556297815fc2ccb6afaf797442fbdb816760" Oct 30 18:45:06.536877 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:06.536857 4442 scope.go:122] "RemoveContainer" containerID="bdb59a9ea1af6e330257e87e382b2e52b9d2ff1e7baf8fd4fb86b0c7b1242b64" Oct 30 18:45:07.188198 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:07.188170 4442 scope.go:122] "RemoveContainer" containerID="82b5d86cd4e85307d629bff8c3b20322f75b4367700289d0fc9ed986c9e96a39" Oct 30 18:45:07.188657 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:45:07.188626 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889765-v5zh2_odh-ai-gateway-infra(5e752ff2-638c-4b1f-94a9-5efd1a53a9c6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889765-v5zh2" podUID="5e752ff2-638c-4b1f-94a9-5efd1a53a9c6" Oct 30 18:45:07.190052 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:07.190021 4442 generic.go:356] "Generic (PLEG): container finished" podID="a87157ec-dde9-4537-a2dc-d2a0e0c80091" containerID="6cdc88b466c945853f123c462578209b8e019e59a1cbcaedbdf47a7f13ca306a" exitCode=60 Oct 30 18:45:07.190229 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:07.190117 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889765-qf9bg" event={"ID":"a87157ec-dde9-4537-a2dc-d2a0e0c80091","Type":"ContainerDied","Data":"6cdc88b466c945853f123c462578209b8e019e59a1cbcaedbdf47a7f13ca306a"} Oct 30 18:45:07.190229 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:07.190202 4442 scope.go:122] "RemoveContainer" containerID="8312e41330ed4fcb47e62f8bc78c90014b9810f2dc3ff1ac3071a58077c05431" Oct 30 18:45:07.190437 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:07.190421 4442 scope.go:122] "RemoveContainer" containerID="6cdc88b466c945853f123c462578209b8e019e59a1cbcaedbdf47a7f13ca306a" Oct 30 18:45:07.190781 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:45:07.190760 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889765-qf9bg_odh-ai-gateway-infra(a87157ec-dde9-4537-a2dc-d2a0e0c80091)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889765-qf9bg" podUID="a87157ec-dde9-4537-a2dc-d2a0e0c80091" Oct 30 18:45:07.192641 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:07.192621 4442 scope.go:122] "RemoveContainer" containerID="ae36d143affd0c63101f4bdd97eef4452f484693a8e573abad5a8d8f95414732" Oct 30 18:45:07.192954 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:45:07.192931 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889765-ljqcz_odh-ai-gateway-infra(4f486a50-76a3-4cc7-bd5b-3edb80ad3315)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889765-ljqcz" podUID="4f486a50-76a3-4cc7-bd5b-3edb80ad3315" Oct 30 18:45:07.194349 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:07.194332 4442 scope.go:122] "RemoveContainer" containerID="7509658e429fd182088c97d6645f06696bc9106721716c3d634ee479d7af0295" Oct 30 18:45:07.194603 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:45:07.194581 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-39c99d-29889765-tdq6b_odh-ai-gateway-infra(9b700c35-c957-4b7a-882e-bf82dd79f7ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889765-tdq6b" podUID="9b700c35-c957-4b7a-882e-bf82dd79f7ff" Oct 30 18:45:07.195933 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:07.195909 4442 generic.go:356] "Generic (PLEG): container finished" podID="f7cdca9f-f772-449c-9f30-067437ef08cc" containerID="2233ac20fc25a831f21873f1e5b7d33d724f15e370247c40d890e9b323ff150d" exitCode=60 Oct 30 18:45:07.196035 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:07.195990 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889765-fsnp2" event={"ID":"f7cdca9f-f772-449c-9f30-067437ef08cc","Type":"ContainerDied","Data":"2233ac20fc25a831f21873f1e5b7d33d724f15e370247c40d890e9b323ff150d"} Oct 30 18:45:07.196221 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:07.196204 4442 scope.go:122] "RemoveContainer" containerID="2233ac20fc25a831f21873f1e5b7d33d724f15e370247c40d890e9b323ff150d" Oct 30 18:45:07.196483 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:45:07.196463 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-45dc34-29889765-fsnp2_odh-ai-gateway-infra(f7cdca9f-f772-449c-9f30-067437ef08cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889765-fsnp2" podUID="f7cdca9f-f772-449c-9f30-067437ef08cc" Oct 30 18:45:07.197851 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:07.197834 4442 scope.go:122] "RemoveContainer" containerID="98ff3ff69272788743dbb56c1a20bb502a540a9bc551a166c24993835627a09f" Oct 30 18:45:07.198073 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:45:07.198045 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-713dc8-29889765-dmf99_odh-ai-gateway-infra(83b551f2-22ff-42f3-acc9-3122313400d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889765-dmf99" podUID="83b551f2-22ff-42f3-acc9-3122313400d8" Oct 30 18:45:07.199711 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:07.199692 4442 scope.go:122] "RemoveContainer" containerID="44943ed079b62965fb591d484fd63b4c53c78c75fdbcd3c2f40affdb2aef09f8" Oct 30 18:45:07.199943 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:45:07.199925 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2107d4-29889765-k88tq_odh-ai-gateway-infra(1e4c2309-c818-46be-abd4-add20576fd6a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889765-k88tq" podUID="1e4c2309-c818-46be-abd4-add20576fd6a" Oct 30 18:45:07.201697 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:07.201676 4442 scope.go:122] "RemoveContainer" containerID="d8a73df452867f89bf3b4b0c311484a6d826c64b0c3cda8765885bb800b8cba1" Oct 30 18:45:07.201951 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:45:07.201929 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-6babca-29889765-rrzvv_odh-ai-gateway-infra(7d52db28-5cd7-4a2a-af8b-b876d41c8453)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889765-rrzvv" podUID="7d52db28-5cd7-4a2a-af8b-b876d41c8453" Oct 30 18:45:07.203419 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:07.203404 4442 scope.go:122] "RemoveContainer" containerID="cccd15d88f9e4c84a761b496a7f08c1367905e50b5cac64a0d9305cf54961ccd" Oct 30 18:45:07.203671 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:45:07.203648 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889765-b2hth_odh-ai-gateway-infra(36a43766-ec94-4991-bad0-e3d4ed5122c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889765-b2hth" podUID="36a43766-ec94-4991-bad0-e3d4ed5122c8" Oct 30 18:45:07.205075 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:07.205056 4442 scope.go:122] "RemoveContainer" containerID="d6e950026f01c4eda1ef5738a6d4a98f26d4b3e2e922e42cfa737b76369b6378" Oct 30 18:45:07.205325 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:45:07.205299 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-921450-29889765-v4sqc_odh-ai-gateway-infra(8ee9297b-cfb8-4154-bb20-d5eb73f78da2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889765-v4sqc" podUID="8ee9297b-cfb8-4154-bb20-d5eb73f78da2" Oct 30 18:45:07.206799 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:07.206781 4442 scope.go:122] "RemoveContainer" containerID="e08172a3f0aa71f6f2455dce0cb4d5685badc0d03db7bd503bd890e6ddef01e2" Oct 30 18:45:07.207028 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:45:07.207003 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889765-mx8tm_odh-ai-gateway-infra(f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889765-mx8tm" podUID="f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04" Oct 30 18:45:07.207359 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:07.207343 4442 scope.go:122] "RemoveContainer" containerID="52154690db6cbf1cf9a62d8f5184b6c0d0d329ae961f40ba31c34749287510ec" Oct 30 18:45:07.208450 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:07.208435 4442 scope.go:122] "RemoveContainer" containerID="f91f5bedb43bfaef4cd4b7dddc5efc26b730a435d8fdaa2f2157b7135e7420fa" Oct 30 18:45:07.208654 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:45:07.208637 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-380c08-29889765-4vxnf_odh-ai-gateway-infra(ed9e3fd1-ccad-4d33-bb46-222acf609143)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889765-4vxnf" podUID="ed9e3fd1-ccad-4d33-bb46-222acf609143" Oct 30 18:45:08.219822 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:08.219779 4442 scope.go:122] "RemoveContainer" containerID="2233ac20fc25a831f21873f1e5b7d33d724f15e370247c40d890e9b323ff150d" Oct 30 18:45:08.220314 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:45:08.220185 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-45dc34-29889765-fsnp2_odh-ai-gateway-infra(f7cdca9f-f772-449c-9f30-067437ef08cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889765-fsnp2" podUID="f7cdca9f-f772-449c-9f30-067437ef08cc" Oct 30 18:45:08.221606 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:08.221587 4442 scope.go:122] "RemoveContainer" containerID="6cdc88b466c945853f123c462578209b8e019e59a1cbcaedbdf47a7f13ca306a" Oct 30 18:45:08.221954 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:45:08.221929 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889765-qf9bg_odh-ai-gateway-infra(a87157ec-dde9-4537-a2dc-d2a0e0c80091)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889765-qf9bg" podUID="a87157ec-dde9-4537-a2dc-d2a0e0c80091" Oct 30 18:45:16.057415 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:16.057382 4442 scope.go:122] "RemoveContainer" containerID="cccd15d88f9e4c84a761b496a7f08c1367905e50b5cac64a0d9305cf54961ccd" Oct 30 18:45:17.058484 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:17.058423 4442 scope.go:122] "RemoveContainer" containerID="7509658e429fd182088c97d6645f06696bc9106721716c3d634ee479d7af0295" Oct 30 18:45:17.437164 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:17.414632 4442 generic.go:356] "Generic (PLEG): container finished" podID="36a43766-ec94-4991-bad0-e3d4ed5122c8" containerID="3d270d30625127f3219becf9ada1c9173061b2a81604bf153be9bb8a198c2637" exitCode=60 Oct 30 18:45:17.437164 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:17.414901 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889765-b2hth" event={"ID":"36a43766-ec94-4991-bad0-e3d4ed5122c8","Type":"ContainerDied","Data":"3d270d30625127f3219becf9ada1c9173061b2a81604bf153be9bb8a198c2637"} Oct 30 18:45:17.437164 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:17.414965 4442 scope.go:122] "RemoveContainer" containerID="cccd15d88f9e4c84a761b496a7f08c1367905e50b5cac64a0d9305cf54961ccd" Oct 30 18:45:17.437164 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:17.415573 4442 scope.go:122] "RemoveContainer" containerID="3d270d30625127f3219becf9ada1c9173061b2a81604bf153be9bb8a198c2637" Oct 30 18:45:17.437164 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:45:17.416020 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889765-b2hth_odh-ai-gateway-infra(36a43766-ec94-4991-bad0-e3d4ed5122c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889765-b2hth" podUID="36a43766-ec94-4991-bad0-e3d4ed5122c8" Oct 30 18:45:18.047078 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:45:18.045189 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:45:18.059193 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:18.058025 4442 scope.go:122] "RemoveContainer" containerID="98ff3ff69272788743dbb56c1a20bb502a540a9bc551a166c24993835627a09f" Oct 30 18:45:18.059193 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:18.058315 4442 scope.go:122] "RemoveContainer" containerID="44943ed079b62965fb591d484fd63b4c53c78c75fdbcd3c2f40affdb2aef09f8" Oct 30 18:45:18.059193 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:18.058505 4442 scope.go:122] "RemoveContainer" containerID="d8a73df452867f89bf3b4b0c311484a6d826c64b0c3cda8765885bb800b8cba1" Oct 30 18:45:18.430239 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:18.430208 4442 generic.go:356] "Generic (PLEG): container finished" podID="9b700c35-c957-4b7a-882e-bf82dd79f7ff" containerID="9d54b970e55df9cd0d8b468836c77f2c382247144c75d352b25951b7dd0e00e6" exitCode=60 Oct 30 18:45:18.430418 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:18.430263 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889765-tdq6b" event={"ID":"9b700c35-c957-4b7a-882e-bf82dd79f7ff","Type":"ContainerDied","Data":"9d54b970e55df9cd0d8b468836c77f2c382247144c75d352b25951b7dd0e00e6"} Oct 30 18:45:18.430418 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:18.430310 4442 scope.go:122] "RemoveContainer" containerID="7509658e429fd182088c97d6645f06696bc9106721716c3d634ee479d7af0295" Oct 30 18:45:18.430777 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:18.430757 4442 scope.go:122] "RemoveContainer" containerID="9d54b970e55df9cd0d8b468836c77f2c382247144c75d352b25951b7dd0e00e6" Oct 30 18:45:18.431198 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:45:18.431168 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-39c99d-29889765-tdq6b_odh-ai-gateway-infra(9b700c35-c957-4b7a-882e-bf82dd79f7ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889765-tdq6b" podUID="9b700c35-c957-4b7a-882e-bf82dd79f7ff" Oct 30 18:45:18.484106 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:18.484073 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889765-b2hth"] Oct 30 18:45:18.751026 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:18.750999 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889765-b2hth" Oct 30 18:45:18.915178 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:18.915073 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36a43766-ec94-4991-bad0-e3d4ed5122c8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36a43766-ec94-4991-bad0-e3d4ed5122c8-maas-api-service-ca\") pod \"36a43766-ec94-4991-bad0-e3d4ed5122c8\" (UID: \"36a43766-ec94-4991-bad0-e3d4ed5122c8\") " Oct 30 18:45:18.915346 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:18.915186 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36a43766-ec94-4991-bad0-e3d4ed5122c8-kube-api-access-5lt2h\" (UniqueName: \"kubernetes.io/projected/36a43766-ec94-4991-bad0-e3d4ed5122c8-kube-api-access-5lt2h\") pod \"36a43766-ec94-4991-bad0-e3d4ed5122c8\" (UID: \"36a43766-ec94-4991-bad0-e3d4ed5122c8\") " Oct 30 18:45:18.915548 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:18.915516 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36a43766-ec94-4991-bad0-e3d4ed5122c8-maas-api-service-ca" pod "36a43766-ec94-4991-bad0-e3d4ed5122c8" (UID: "36a43766-ec94-4991-bad0-e3d4ed5122c8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:18.915790 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:18.915772 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36a43766-ec94-4991-bad0-e3d4ed5122c8-maas-api-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:45:18.917625 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:18.917597 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36a43766-ec94-4991-bad0-e3d4ed5122c8-kube-api-access-5lt2h" pod "36a43766-ec94-4991-bad0-e3d4ed5122c8" (UID: "36a43766-ec94-4991-bad0-e3d4ed5122c8"). InnerVolumeSpecName "kube-api-access-5lt2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:19.017199 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:19.017165 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lt2h\" (UniqueName: \"kubernetes.io/projected/36a43766-ec94-4991-bad0-e3d4ed5122c8-kube-api-access-5lt2h\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:45:19.057763 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:19.057725 4442 scope.go:122] "RemoveContainer" containerID="ae36d143affd0c63101f4bdd97eef4452f484693a8e573abad5a8d8f95414732" Oct 30 18:45:19.057966 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:19.057877 4442 scope.go:122] "RemoveContainer" containerID="e08172a3f0aa71f6f2455dce0cb4d5685badc0d03db7bd503bd890e6ddef01e2" Oct 30 18:45:19.445013 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:19.444982 4442 generic.go:356] "Generic (PLEG): container finished" podID="83b551f2-22ff-42f3-acc9-3122313400d8" containerID="1d72041fc267fc271ac73a26f100d10bd8e28f6056f92f9ea79681e9baf78dcf" exitCode=60 Oct 30 18:45:19.445501 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:19.445029 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889765-dmf99" event={"ID":"83b551f2-22ff-42f3-acc9-3122313400d8","Type":"ContainerDied","Data":"1d72041fc267fc271ac73a26f100d10bd8e28f6056f92f9ea79681e9baf78dcf"} Oct 30 18:45:19.445501 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:19.445105 4442 scope.go:122] "RemoveContainer" containerID="98ff3ff69272788743dbb56c1a20bb502a540a9bc551a166c24993835627a09f" Oct 30 18:45:19.445610 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:19.445557 4442 scope.go:122] "RemoveContainer" containerID="1d72041fc267fc271ac73a26f100d10bd8e28f6056f92f9ea79681e9baf78dcf" Oct 30 18:45:19.445965 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:45:19.445939 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-713dc8-29889765-dmf99_odh-ai-gateway-infra(83b551f2-22ff-42f3-acc9-3122313400d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889765-dmf99" podUID="83b551f2-22ff-42f3-acc9-3122313400d8" Oct 30 18:45:19.449093 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:19.449065 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889765-k88tq" event={"ID":"1e4c2309-c818-46be-abd4-add20576fd6a","Type":"ContainerStarted","Data":"3149afb2cba38d8527e787d8f78c78294e4c093b6a2ac19ebf149537145996df"} Oct 30 18:45:19.451979 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:19.451953 4442 generic.go:356] "Generic (PLEG): container finished" podID="7d52db28-5cd7-4a2a-af8b-b876d41c8453" containerID="601430709b65c6c662d04d5bf486857e39639fafe5d05b4626fbcdda0498a91d" exitCode=60 Oct 30 18:45:19.452102 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:19.452022 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889765-rrzvv" event={"ID":"7d52db28-5cd7-4a2a-af8b-b876d41c8453","Type":"ContainerDied","Data":"601430709b65c6c662d04d5bf486857e39639fafe5d05b4626fbcdda0498a91d"} Oct 30 18:45:19.452542 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:19.452513 4442 scope.go:122] "RemoveContainer" containerID="601430709b65c6c662d04d5bf486857e39639fafe5d05b4626fbcdda0498a91d" Oct 30 18:45:19.452990 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:45:19.452910 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-6babca-29889765-rrzvv_odh-ai-gateway-infra(7d52db28-5cd7-4a2a-af8b-b876d41c8453)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889765-rrzvv" podUID="7d52db28-5cd7-4a2a-af8b-b876d41c8453" Oct 30 18:45:19.454122 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:19.454106 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889765-b2hth" Oct 30 18:45:19.454658 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:19.454634 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889765-b2hth" event={"ID":"36a43766-ec94-4991-bad0-e3d4ed5122c8","Type":"ContainerDied","Data":"def354d8bcc227a9c5bcafb973cfafb895c3a04a36e7e2acaa13ec1dd386e328"} Oct 30 18:45:19.478797 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:19.478720 4442 scope.go:122] "RemoveContainer" containerID="d8a73df452867f89bf3b4b0c311484a6d826c64b0c3cda8765885bb800b8cba1" Oct 30 18:45:19.486036 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:19.486012 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889765-tdq6b"] Oct 30 18:45:19.498384 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:19.498276 4442 scope.go:122] "RemoveContainer" containerID="3d270d30625127f3219becf9ada1c9173061b2a81604bf153be9bb8a198c2637" Oct 30 18:45:19.617278 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:19.617246 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889765-b2hth"] Oct 30 18:45:19.633961 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:19.633927 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889765-b2hth"] Oct 30 18:45:19.689098 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:19.689071 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889765-tdq6b" Oct 30 18:45:19.726832 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:19.726800 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b700c35-c957-4b7a-882e-bf82dd79f7ff-kube-api-access-wh5qb\" (UniqueName: \"kubernetes.io/projected/9b700c35-c957-4b7a-882e-bf82dd79f7ff-kube-api-access-wh5qb\") pod \"9b700c35-c957-4b7a-882e-bf82dd79f7ff\" (UID: \"9b700c35-c957-4b7a-882e-bf82dd79f7ff\") " Oct 30 18:45:19.727070 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:19.726965 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b700c35-c957-4b7a-882e-bf82dd79f7ff-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b700c35-c957-4b7a-882e-bf82dd79f7ff-maas-api-service-ca\") pod \"9b700c35-c957-4b7a-882e-bf82dd79f7ff\" (UID: \"9b700c35-c957-4b7a-882e-bf82dd79f7ff\") " Oct 30 18:45:19.727678 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:19.727649 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b700c35-c957-4b7a-882e-bf82dd79f7ff-maas-api-service-ca" pod "9b700c35-c957-4b7a-882e-bf82dd79f7ff" (UID: "9b700c35-c957-4b7a-882e-bf82dd79f7ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:19.729947 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:19.729884 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b700c35-c957-4b7a-882e-bf82dd79f7ff-kube-api-access-wh5qb" pod "9b700c35-c957-4b7a-882e-bf82dd79f7ff" (UID: "9b700c35-c957-4b7a-882e-bf82dd79f7ff"). InnerVolumeSpecName "kube-api-access-wh5qb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:19.829342 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:19.829306 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b700c35-c957-4b7a-882e-bf82dd79f7ff-maas-api-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:45:19.829342 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:19.829342 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh5qb\" (UniqueName: \"kubernetes.io/projected/9b700c35-c957-4b7a-882e-bf82dd79f7ff-kube-api-access-wh5qb\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:45:20.058265 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:20.058120 4442 scope.go:122] "RemoveContainer" containerID="6cdc88b466c945853f123c462578209b8e019e59a1cbcaedbdf47a7f13ca306a" Oct 30 18:45:20.058450 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:20.058292 4442 scope.go:122] "RemoveContainer" containerID="2233ac20fc25a831f21873f1e5b7d33d724f15e370247c40d890e9b323ff150d" Oct 30 18:45:20.058514 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:20.058461 4442 scope.go:122] "RemoveContainer" containerID="f91f5bedb43bfaef4cd4b7dddc5efc26b730a435d8fdaa2f2157b7135e7420fa" Oct 30 18:45:20.059508 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:20.059378 4442 scope.go:122] "RemoveContainer" containerID="d6e950026f01c4eda1ef5738a6d4a98f26d4b3e2e922e42cfa737b76369b6378" Oct 30 18:45:20.469556 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:20.469529 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889765-tdq6b" Oct 30 18:45:20.469556 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:20.469546 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889765-tdq6b" event={"ID":"9b700c35-c957-4b7a-882e-bf82dd79f7ff","Type":"ContainerDied","Data":"c6b6bacce38866c4b4a86dce4d1a1421edfbc50aff37f39f5315b48f470651b5"} Oct 30 18:45:20.470085 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:20.469612 4442 scope.go:122] "RemoveContainer" containerID="9d54b970e55df9cd0d8b468836c77f2c382247144c75d352b25951b7dd0e00e6" Oct 30 18:45:20.475435 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:20.474218 4442 generic.go:356] "Generic (PLEG): container finished" podID="1e4c2309-c818-46be-abd4-add20576fd6a" containerID="3149afb2cba38d8527e787d8f78c78294e4c093b6a2ac19ebf149537145996df" exitCode=60 Oct 30 18:45:20.475435 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:20.474311 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889765-k88tq" event={"ID":"1e4c2309-c818-46be-abd4-add20576fd6a","Type":"ContainerDied","Data":"3149afb2cba38d8527e787d8f78c78294e4c093b6a2ac19ebf149537145996df"} Oct 30 18:45:20.475435 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:20.474701 4442 scope.go:122] "RemoveContainer" containerID="3149afb2cba38d8527e787d8f78c78294e4c093b6a2ac19ebf149537145996df" Oct 30 18:45:20.475435 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:45:20.475042 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2107d4-29889765-k88tq_odh-ai-gateway-infra(1e4c2309-c818-46be-abd4-add20576fd6a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889765-k88tq" podUID="1e4c2309-c818-46be-abd4-add20576fd6a" Oct 30 18:45:20.482367 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:20.482207 4442 generic.go:356] "Generic (PLEG): container finished" podID="f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04" containerID="7c8dbafaf5c8bf4e4140861c6a3af662f430b93874e72cffde120d89de36b3c7" exitCode=60 Oct 30 18:45:20.482485 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:20.482436 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889765-mx8tm" event={"ID":"f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04","Type":"ContainerDied","Data":"7c8dbafaf5c8bf4e4140861c6a3af662f430b93874e72cffde120d89de36b3c7"} Oct 30 18:45:20.483938 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:20.482916 4442 scope.go:122] "RemoveContainer" containerID="7c8dbafaf5c8bf4e4140861c6a3af662f430b93874e72cffde120d89de36b3c7" Oct 30 18:45:20.483938 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:45:20.483395 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889765-mx8tm_odh-ai-gateway-infra(f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889765-mx8tm" podUID="f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04" Oct 30 18:45:20.486360 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:20.486338 4442 generic.go:356] "Generic (PLEG): container finished" podID="4f486a50-76a3-4cc7-bd5b-3edb80ad3315" containerID="58fa6f4c64845a56a147109644ec414efe8bb1598b8bd3d05e59b60de052bbaa" exitCode=60 Oct 30 18:45:20.486467 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:20.486416 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889765-ljqcz" event={"ID":"4f486a50-76a3-4cc7-bd5b-3edb80ad3315","Type":"ContainerDied","Data":"58fa6f4c64845a56a147109644ec414efe8bb1598b8bd3d05e59b60de052bbaa"} Oct 30 18:45:20.486685 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:20.486665 4442 scope.go:122] "RemoveContainer" containerID="44943ed079b62965fb591d484fd63b4c53c78c75fdbcd3c2f40affdb2aef09f8" Oct 30 18:45:20.486965 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:20.486944 4442 scope.go:122] "RemoveContainer" containerID="58fa6f4c64845a56a147109644ec414efe8bb1598b8bd3d05e59b60de052bbaa" Oct 30 18:45:20.487357 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:45:20.487331 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889765-ljqcz_odh-ai-gateway-infra(4f486a50-76a3-4cc7-bd5b-3edb80ad3315)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889765-ljqcz" podUID="4f486a50-76a3-4cc7-bd5b-3edb80ad3315" Oct 30 18:45:20.505566 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:20.505545 4442 scope.go:122] "RemoveContainer" containerID="e08172a3f0aa71f6f2455dce0cb4d5685badc0d03db7bd503bd890e6ddef01e2" Oct 30 18:45:20.522536 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:20.522514 4442 scope.go:122] "RemoveContainer" containerID="ae36d143affd0c63101f4bdd97eef4452f484693a8e573abad5a8d8f95414732" Oct 30 18:45:20.523127 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:20.523106 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889765-dmf99"] Oct 30 18:45:20.540582 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:20.540549 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889765-tdq6b"] Oct 30 18:45:20.561650 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:20.561585 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889765-tdq6b"] Oct 30 18:45:20.589840 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:20.589807 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889765-rrzvv"] Oct 30 18:45:20.688511 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:20.688465 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889765-k88tq"] Oct 30 18:45:20.762953 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:20.762926 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889765-dmf99" Oct 30 18:45:20.852962 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:20.852550 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83b551f2-22ff-42f3-acc9-3122313400d8-kube-api-access-hq5kc\" (UniqueName: \"kubernetes.io/projected/83b551f2-22ff-42f3-acc9-3122313400d8-kube-api-access-hq5kc\") pod \"83b551f2-22ff-42f3-acc9-3122313400d8\" (UID: \"83b551f2-22ff-42f3-acc9-3122313400d8\") " Oct 30 18:45:20.853381 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:20.853357 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83b551f2-22ff-42f3-acc9-3122313400d8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83b551f2-22ff-42f3-acc9-3122313400d8-maas-api-service-ca\") pod \"83b551f2-22ff-42f3-acc9-3122313400d8\" (UID: \"83b551f2-22ff-42f3-acc9-3122313400d8\") " Oct 30 18:45:20.854099 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:20.854044 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83b551f2-22ff-42f3-acc9-3122313400d8-maas-api-service-ca" pod "83b551f2-22ff-42f3-acc9-3122313400d8" (UID: "83b551f2-22ff-42f3-acc9-3122313400d8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:20.854614 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:20.854593 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83b551f2-22ff-42f3-acc9-3122313400d8-maas-api-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:45:20.861314 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:20.861288 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83b551f2-22ff-42f3-acc9-3122313400d8-kube-api-access-hq5kc" pod "83b551f2-22ff-42f3-acc9-3122313400d8" (UID: "83b551f2-22ff-42f3-acc9-3122313400d8"). InnerVolumeSpecName "kube-api-access-hq5kc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:20.869535 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:20.869509 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889765-rrzvv" Oct 30 18:45:20.956575 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:20.956529 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d52db28-5cd7-4a2a-af8b-b876d41c8453-kube-api-access-n6ld6\" (UniqueName: \"kubernetes.io/projected/7d52db28-5cd7-4a2a-af8b-b876d41c8453-kube-api-access-n6ld6\") pod \"7d52db28-5cd7-4a2a-af8b-b876d41c8453\" (UID: \"7d52db28-5cd7-4a2a-af8b-b876d41c8453\") " Oct 30 18:45:20.956757 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:20.956607 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d52db28-5cd7-4a2a-af8b-b876d41c8453-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d52db28-5cd7-4a2a-af8b-b876d41c8453-maas-api-service-ca\") pod \"7d52db28-5cd7-4a2a-af8b-b876d41c8453\" (UID: \"7d52db28-5cd7-4a2a-af8b-b876d41c8453\") " Oct 30 18:45:20.965437 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:20.957648 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq5kc\" (UniqueName: \"kubernetes.io/projected/83b551f2-22ff-42f3-acc9-3122313400d8-kube-api-access-hq5kc\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:45:20.965437 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:20.958546 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d52db28-5cd7-4a2a-af8b-b876d41c8453-maas-api-service-ca" pod "7d52db28-5cd7-4a2a-af8b-b876d41c8453" (UID: "7d52db28-5cd7-4a2a-af8b-b876d41c8453"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:20.969908 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:20.969870 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d52db28-5cd7-4a2a-af8b-b876d41c8453-kube-api-access-n6ld6" pod "7d52db28-5cd7-4a2a-af8b-b876d41c8453" (UID: "7d52db28-5cd7-4a2a-af8b-b876d41c8453"). InnerVolumeSpecName "kube-api-access-n6ld6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:21.057613 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:21.057581 4442 scope.go:122] "RemoveContainer" containerID="82b5d86cd4e85307d629bff8c3b20322f75b4367700289d0fc9ed986c9e96a39" Oct 30 18:45:21.060558 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:21.060531 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6ld6\" (UniqueName: \"kubernetes.io/projected/7d52db28-5cd7-4a2a-af8b-b876d41c8453-kube-api-access-n6ld6\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:45:21.060689 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:21.060562 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d52db28-5cd7-4a2a-af8b-b876d41c8453-maas-api-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:45:21.235227 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:21.198593 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36a43766-ec94-4991-bad0-e3d4ed5122c8" path="/var/lib/kubelet/pods/36a43766-ec94-4991-bad0-e3d4ed5122c8/volumes" Oct 30 18:45:21.433871 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:21.235697 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b700c35-c957-4b7a-882e-bf82dd79f7ff" path="/var/lib/kubelet/pods/9b700c35-c957-4b7a-882e-bf82dd79f7ff/volumes" Oct 30 18:45:21.691216 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:21.680308 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889765-rrzvv" event={"ID":"7d52db28-5cd7-4a2a-af8b-b876d41c8453","Type":"ContainerDied","Data":"694a8de4c6e76e62888cb9eb2aae58c48e7167b5409c59a123a062f86f38fa99"} Oct 30 18:45:21.691216 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:21.680381 4442 scope.go:122] "RemoveContainer" containerID="601430709b65c6c662d04d5bf486857e39639fafe5d05b4626fbcdda0498a91d" Oct 30 18:45:21.691216 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:21.680494 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889765-rrzvv" Oct 30 18:45:21.759384 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:21.729687 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889765-ljqcz"] Oct 30 18:45:21.806785 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:21.788738 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889765-v4sqc" event={"ID":"8ee9297b-cfb8-4154-bb20-d5eb73f78da2","Type":"ContainerStarted","Data":"934686e473c3e8ee948864fadf7a2330824bf17feb13bce78d32f5ddbccdc0cd"} Oct 30 18:45:21.806785 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:21.789336 4442 scope.go:122] "RemoveContainer" containerID="934686e473c3e8ee948864fadf7a2330824bf17feb13bce78d32f5ddbccdc0cd" Oct 30 18:45:21.806785 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:45:21.789793 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-921450-29889765-v4sqc_odh-ai-gateway-infra(8ee9297b-cfb8-4154-bb20-d5eb73f78da2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889765-v4sqc" podUID="8ee9297b-cfb8-4154-bb20-d5eb73f78da2" Oct 30 18:45:21.851177 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:21.843575 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889765-4vxnf" event={"ID":"ed9e3fd1-ccad-4d33-bb46-222acf609143","Type":"ContainerStarted","Data":"2ae86d1a7dc34b0f64bb30f58edf4cb52dd56ab7bcc7518dea9974bd64aafbfa"} Oct 30 18:45:22.051161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:22.051053 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889765-qf9bg" event={"ID":"a87157ec-dde9-4537-a2dc-d2a0e0c80091","Type":"ContainerStarted","Data":"691a3eb920a9c90ea40c93458a8d75092f7998c36f946b4b3f45732744bbc0cf"} Oct 30 18:45:22.076673 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:22.076645 4442 scope.go:122] "RemoveContainer" containerID="691a3eb920a9c90ea40c93458a8d75092f7998c36f946b4b3f45732744bbc0cf" Oct 30 18:45:22.077477 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:45:22.077444 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889765-qf9bg_odh-ai-gateway-infra(a87157ec-dde9-4537-a2dc-d2a0e0c80091)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889765-qf9bg" podUID="a87157ec-dde9-4537-a2dc-d2a0e0c80091" Oct 30 18:45:22.141797 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:22.141765 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889765-fsnp2" event={"ID":"f7cdca9f-f772-449c-9f30-067437ef08cc","Type":"ContainerStarted","Data":"69b4a4f024c42f1296ea98bc5c7dff1d8b5d686ec55b7d9b781d2bb378fa92be"} Oct 30 18:45:22.144850 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:22.144824 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889765-dmf99" event={"ID":"83b551f2-22ff-42f3-acc9-3122313400d8","Type":"ContainerDied","Data":"d8d7e90e9771d4c6b2c413e11ad8fc957b2f00024ba604f04f8af2bbb5d51d0d"} Oct 30 18:45:22.145078 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:22.145062 4442 scope.go:122] "RemoveContainer" containerID="1d72041fc267fc271ac73a26f100d10bd8e28f6056f92f9ea79681e9baf78dcf" Oct 30 18:45:22.145961 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:22.145944 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889765-dmf99" Oct 30 18:45:22.150017 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:22.145974 4442 scope.go:122] "RemoveContainer" containerID="69b4a4f024c42f1296ea98bc5c7dff1d8b5d686ec55b7d9b781d2bb378fa92be" Oct 30 18:45:22.159825 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:45:22.159789 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-45dc34-29889765-fsnp2_odh-ai-gateway-infra(f7cdca9f-f772-449c-9f30-067437ef08cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889765-fsnp2" podUID="f7cdca9f-f772-449c-9f30-067437ef08cc" Oct 30 18:45:22.182165 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:22.180703 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889765-mx8tm"] Oct 30 18:45:22.222065 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:22.222004 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889765-rrzvv"] Oct 30 18:45:22.228808 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:22.228774 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889765-rrzvv"] Oct 30 18:45:22.276303 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:22.276267 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889765-dmf99"] Oct 30 18:45:22.284209 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:22.284097 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889765-dmf99"] Oct 30 18:45:22.394730 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:22.394695 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889765-ljqcz" Oct 30 18:45:22.478680 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:22.478656 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889765-k88tq" Oct 30 18:45:22.482318 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:22.482297 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889765-mx8tm" Oct 30 18:45:22.567203 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:22.567084 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e4c2309-c818-46be-abd4-add20576fd6a-kube-api-access-rkjwh\" (UniqueName: \"kubernetes.io/projected/1e4c2309-c818-46be-abd4-add20576fd6a-kube-api-access-rkjwh\") pod \"1e4c2309-c818-46be-abd4-add20576fd6a\" (UID: \"1e4c2309-c818-46be-abd4-add20576fd6a\") " Oct 30 18:45:22.567203 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:22.567174 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04-kube-api-access-7rngx\" (UniqueName: \"kubernetes.io/projected/f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04-kube-api-access-7rngx\") pod \"f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04\" (UID: \"f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04\") " Oct 30 18:45:22.567439 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:22.567245 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f486a50-76a3-4cc7-bd5b-3edb80ad3315-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f486a50-76a3-4cc7-bd5b-3edb80ad3315-maas-api-service-ca\") pod \"4f486a50-76a3-4cc7-bd5b-3edb80ad3315\" (UID: \"4f486a50-76a3-4cc7-bd5b-3edb80ad3315\") " Oct 30 18:45:22.567439 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:22.567322 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e4c2309-c818-46be-abd4-add20576fd6a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e4c2309-c818-46be-abd4-add20576fd6a-maas-api-service-ca\") pod \"1e4c2309-c818-46be-abd4-add20576fd6a\" (UID: \"1e4c2309-c818-46be-abd4-add20576fd6a\") " Oct 30 18:45:22.567556 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:22.567490 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04-maas-api-service-ca\") pod \"f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04\" (UID: \"f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04\") " Oct 30 18:45:22.567677 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:22.567661 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f486a50-76a3-4cc7-bd5b-3edb80ad3315-kube-api-access-m9hkz\" (UniqueName: \"kubernetes.io/projected/4f486a50-76a3-4cc7-bd5b-3edb80ad3315-kube-api-access-m9hkz\") pod \"4f486a50-76a3-4cc7-bd5b-3edb80ad3315\" (UID: \"4f486a50-76a3-4cc7-bd5b-3edb80ad3315\") " Oct 30 18:45:22.569036 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:22.569001 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04-maas-api-service-ca" pod "f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04" (UID: "f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:22.570631 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:22.570599 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f486a50-76a3-4cc7-bd5b-3edb80ad3315-kube-api-access-m9hkz" pod "4f486a50-76a3-4cc7-bd5b-3edb80ad3315" (UID: "4f486a50-76a3-4cc7-bd5b-3edb80ad3315"). InnerVolumeSpecName "kube-api-access-m9hkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:22.570934 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:22.570906 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e4c2309-c818-46be-abd4-add20576fd6a-maas-api-service-ca" pod "1e4c2309-c818-46be-abd4-add20576fd6a" (UID: "1e4c2309-c818-46be-abd4-add20576fd6a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:22.571070 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:22.570933 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04-kube-api-access-7rngx" pod "f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04" (UID: "f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04"). InnerVolumeSpecName "kube-api-access-7rngx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:22.571356 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:22.571334 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f486a50-76a3-4cc7-bd5b-3edb80ad3315-maas-api-service-ca" pod "4f486a50-76a3-4cc7-bd5b-3edb80ad3315" (UID: "4f486a50-76a3-4cc7-bd5b-3edb80ad3315"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:22.571460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:22.571360 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e4c2309-c818-46be-abd4-add20576fd6a-kube-api-access-rkjwh" pod "1e4c2309-c818-46be-abd4-add20576fd6a" (UID: "1e4c2309-c818-46be-abd4-add20576fd6a"). InnerVolumeSpecName "kube-api-access-rkjwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:22.670413 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:22.670378 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04-maas-api-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:45:22.670413 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:22.670414 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9hkz\" (UniqueName: \"kubernetes.io/projected/4f486a50-76a3-4cc7-bd5b-3edb80ad3315-kube-api-access-m9hkz\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:45:22.671065 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:22.670432 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkjwh\" (UniqueName: \"kubernetes.io/projected/1e4c2309-c818-46be-abd4-add20576fd6a-kube-api-access-rkjwh\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:45:22.671065 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:22.670451 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rngx\" (UniqueName: \"kubernetes.io/projected/f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04-kube-api-access-7rngx\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:45:22.671065 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:22.670466 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f486a50-76a3-4cc7-bd5b-3edb80ad3315-maas-api-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:45:22.671065 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:22.670480 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e4c2309-c818-46be-abd4-add20576fd6a-maas-api-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:45:22.927571 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:22.927527 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889765-v4sqc"] Oct 30 18:45:22.984662 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:22.984628 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889765-4vxnf"] Oct 30 18:45:23.070770 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.070735 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d52db28-5cd7-4a2a-af8b-b876d41c8453" path="/var/lib/kubelet/pods/7d52db28-5cd7-4a2a-af8b-b876d41c8453/volumes" Oct 30 18:45:23.071461 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.071443 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83b551f2-22ff-42f3-acc9-3122313400d8" path="/var/lib/kubelet/pods/83b551f2-22ff-42f3-acc9-3122313400d8/volumes" Oct 30 18:45:23.112232 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.112205 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889765-qf9bg"] Oct 30 18:45:23.174285 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.174238 4442 generic.go:356] "Generic (PLEG): container finished" podID="ed9e3fd1-ccad-4d33-bb46-222acf609143" containerID="2ae86d1a7dc34b0f64bb30f58edf4cb52dd56ab7bcc7518dea9974bd64aafbfa" exitCode=60 Oct 30 18:45:23.174466 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.174358 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889765-4vxnf" event={"ID":"ed9e3fd1-ccad-4d33-bb46-222acf609143","Type":"ContainerDied","Data":"2ae86d1a7dc34b0f64bb30f58edf4cb52dd56ab7bcc7518dea9974bd64aafbfa"} Oct 30 18:45:23.174466 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.174415 4442 scope.go:122] "RemoveContainer" containerID="f91f5bedb43bfaef4cd4b7dddc5efc26b730a435d8fdaa2f2157b7135e7420fa" Oct 30 18:45:23.178692 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.178615 4442 generic.go:356] "Generic (PLEG): container finished" podID="5e752ff2-638c-4b1f-94a9-5efd1a53a9c6" containerID="ea68b2e62624284e714a20170e1ee85d94661da3bfd1d4d3837813d5d4ca9766" exitCode=60 Oct 30 18:45:23.178830 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.178689 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889765-v5zh2" event={"ID":"5e752ff2-638c-4b1f-94a9-5efd1a53a9c6","Type":"ContainerDied","Data":"ea68b2e62624284e714a20170e1ee85d94661da3bfd1d4d3837813d5d4ca9766"} Oct 30 18:45:23.179127 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.179109 4442 scope.go:122] "RemoveContainer" containerID="ea68b2e62624284e714a20170e1ee85d94661da3bfd1d4d3837813d5d4ca9766" Oct 30 18:45:23.179490 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:45:23.179471 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889765-v5zh2_odh-ai-gateway-infra(5e752ff2-638c-4b1f-94a9-5efd1a53a9c6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889765-v5zh2" podUID="5e752ff2-638c-4b1f-94a9-5efd1a53a9c6" Oct 30 18:45:23.180878 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.180856 4442 generic.go:356] "Generic (PLEG): container finished" podID="a87157ec-dde9-4537-a2dc-d2a0e0c80091" containerID="691a3eb920a9c90ea40c93458a8d75092f7998c36f946b4b3f45732744bbc0cf" exitCode=60 Oct 30 18:45:23.180998 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.180939 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889765-qf9bg" event={"ID":"a87157ec-dde9-4537-a2dc-d2a0e0c80091","Type":"ContainerDied","Data":"691a3eb920a9c90ea40c93458a8d75092f7998c36f946b4b3f45732744bbc0cf"} Oct 30 18:45:23.182829 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.182814 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889765-ljqcz" Oct 30 18:45:23.183246 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.183223 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889765-ljqcz" event={"ID":"4f486a50-76a3-4cc7-bd5b-3edb80ad3315","Type":"ContainerDied","Data":"5b10b02d3d39d8920bf7adcefe233ec65b148f0eaf965cd63f6fd1f6bd8d522d"} Oct 30 18:45:23.186586 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.186562 4442 generic.go:356] "Generic (PLEG): container finished" podID="f7cdca9f-f772-449c-9f30-067437ef08cc" containerID="69b4a4f024c42f1296ea98bc5c7dff1d8b5d686ec55b7d9b781d2bb378fa92be" exitCode=60 Oct 30 18:45:23.186831 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.186812 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889765-fsnp2" event={"ID":"f7cdca9f-f772-449c-9f30-067437ef08cc","Type":"ContainerDied","Data":"69b4a4f024c42f1296ea98bc5c7dff1d8b5d686ec55b7d9b781d2bb378fa92be"} Oct 30 18:45:23.187231 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.187210 4442 scope.go:122] "RemoveContainer" containerID="69b4a4f024c42f1296ea98bc5c7dff1d8b5d686ec55b7d9b781d2bb378fa92be" Oct 30 18:45:23.187643 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:45:23.187620 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-45dc34-29889765-fsnp2_odh-ai-gateway-infra(f7cdca9f-f772-449c-9f30-067437ef08cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889765-fsnp2" podUID="f7cdca9f-f772-449c-9f30-067437ef08cc" Oct 30 18:45:23.189511 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.189492 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889765-k88tq" Oct 30 18:45:23.189607 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.189489 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889765-k88tq" event={"ID":"1e4c2309-c818-46be-abd4-add20576fd6a","Type":"ContainerDied","Data":"da0fa2f2ebcb514f755dc68054509d95548012042140d1034eece63165aae2dc"} Oct 30 18:45:23.194893 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.194791 4442 scope.go:122] "RemoveContainer" containerID="82b5d86cd4e85307d629bff8c3b20322f75b4367700289d0fc9ed986c9e96a39" Oct 30 18:45:23.195668 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.195642 4442 generic.go:356] "Generic (PLEG): container finished" podID="8ee9297b-cfb8-4154-bb20-d5eb73f78da2" containerID="934686e473c3e8ee948864fadf7a2330824bf17feb13bce78d32f5ddbccdc0cd" exitCode=60 Oct 30 18:45:23.195807 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.195703 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889765-v4sqc" event={"ID":"8ee9297b-cfb8-4154-bb20-d5eb73f78da2","Type":"ContainerDied","Data":"934686e473c3e8ee948864fadf7a2330824bf17feb13bce78d32f5ddbccdc0cd"} Oct 30 18:45:23.198547 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.198248 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889765-mx8tm" event={"ID":"f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04","Type":"ContainerDied","Data":"5c3d22b24b22f176346ac64b7d896bb77aedbef573ebedf604f62cdb8d920c9c"} Oct 30 18:45:23.198547 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.198295 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889765-mx8tm" Oct 30 18:45:23.214870 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.214498 4442 scope.go:122] "RemoveContainer" containerID="6cdc88b466c945853f123c462578209b8e019e59a1cbcaedbdf47a7f13ca306a" Oct 30 18:45:23.242274 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.240947 4442 scope.go:122] "RemoveContainer" containerID="58fa6f4c64845a56a147109644ec414efe8bb1598b8bd3d05e59b60de052bbaa" Oct 30 18:45:23.258452 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.256361 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889765-fsnp2"] Oct 30 18:45:23.275196 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.272639 4442 scope.go:122] "RemoveContainer" containerID="2233ac20fc25a831f21873f1e5b7d33d724f15e370247c40d890e9b323ff150d" Oct 30 18:45:23.307782 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.305794 4442 scope.go:122] "RemoveContainer" containerID="3149afb2cba38d8527e787d8f78c78294e4c093b6a2ac19ebf149537145996df" Oct 30 18:45:23.380328 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.362536 4442 scope.go:122] "RemoveContainer" containerID="d6e950026f01c4eda1ef5738a6d4a98f26d4b3e2e922e42cfa737b76369b6378" Oct 30 18:45:23.391419 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.389752 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889765-ljqcz"] Oct 30 18:45:23.391961 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.391940 4442 scope.go:122] "RemoveContainer" containerID="7c8dbafaf5c8bf4e4140861c6a3af662f430b93874e72cffde120d89de36b3c7" Oct 30 18:45:23.406090 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.399280 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889765-ljqcz"] Oct 30 18:45:23.417377 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.417319 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889765-mx8tm"] Oct 30 18:45:23.425924 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.425864 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889765-mx8tm"] Oct 30 18:45:23.519985 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.519958 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889765-k88tq"] Oct 30 18:45:23.521548 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.521525 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889765-qf9bg" Oct 30 18:45:23.524167 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.524114 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889765-v4sqc" Oct 30 18:45:23.532900 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.532878 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889765-4vxnf" Oct 30 18:45:23.541674 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.541646 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889765-k88tq"] Oct 30 18:45:23.687842 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.687812 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a87157ec-dde9-4537-a2dc-d2a0e0c80091-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a87157ec-dde9-4537-a2dc-d2a0e0c80091-maas-api-service-ca\") pod \"a87157ec-dde9-4537-a2dc-d2a0e0c80091\" (UID: \"a87157ec-dde9-4537-a2dc-d2a0e0c80091\") " Oct 30 18:45:23.688026 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.687921 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ee9297b-cfb8-4154-bb20-d5eb73f78da2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ee9297b-cfb8-4154-bb20-d5eb73f78da2-maas-api-service-ca\") pod \"8ee9297b-cfb8-4154-bb20-d5eb73f78da2\" (UID: \"8ee9297b-cfb8-4154-bb20-d5eb73f78da2\") " Oct 30 18:45:23.688104 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.688029 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed9e3fd1-ccad-4d33-bb46-222acf609143-kube-api-access-bfq4p\" (UniqueName: \"kubernetes.io/projected/ed9e3fd1-ccad-4d33-bb46-222acf609143-kube-api-access-bfq4p\") pod \"ed9e3fd1-ccad-4d33-bb46-222acf609143\" (UID: \"ed9e3fd1-ccad-4d33-bb46-222acf609143\") " Oct 30 18:45:23.688104 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.688097 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a87157ec-dde9-4537-a2dc-d2a0e0c80091-kube-api-access-xhmsl\" (UniqueName: \"kubernetes.io/projected/a87157ec-dde9-4537-a2dc-d2a0e0c80091-kube-api-access-xhmsl\") pod \"a87157ec-dde9-4537-a2dc-d2a0e0c80091\" (UID: \"a87157ec-dde9-4537-a2dc-d2a0e0c80091\") " Oct 30 18:45:23.688316 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.688189 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ee9297b-cfb8-4154-bb20-d5eb73f78da2-kube-api-access-hq5nd\" (UniqueName: \"kubernetes.io/projected/8ee9297b-cfb8-4154-bb20-d5eb73f78da2-kube-api-access-hq5nd\") pod \"8ee9297b-cfb8-4154-bb20-d5eb73f78da2\" (UID: \"8ee9297b-cfb8-4154-bb20-d5eb73f78da2\") " Oct 30 18:45:23.688316 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.688261 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed9e3fd1-ccad-4d33-bb46-222acf609143-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed9e3fd1-ccad-4d33-bb46-222acf609143-maas-api-service-ca\") pod \"ed9e3fd1-ccad-4d33-bb46-222acf609143\" (UID: \"ed9e3fd1-ccad-4d33-bb46-222acf609143\") " Oct 30 18:45:23.688316 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.688278 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a87157ec-dde9-4537-a2dc-d2a0e0c80091-maas-api-service-ca" pod "a87157ec-dde9-4537-a2dc-d2a0e0c80091" (UID: "a87157ec-dde9-4537-a2dc-d2a0e0c80091"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:23.688497 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.688434 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ee9297b-cfb8-4154-bb20-d5eb73f78da2-maas-api-service-ca" pod "8ee9297b-cfb8-4154-bb20-d5eb73f78da2" (UID: "8ee9297b-cfb8-4154-bb20-d5eb73f78da2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:23.692664 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.688889 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a87157ec-dde9-4537-a2dc-d2a0e0c80091-maas-api-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:45:23.692664 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.688914 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ee9297b-cfb8-4154-bb20-d5eb73f78da2-maas-api-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:45:23.692664 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.689122 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed9e3fd1-ccad-4d33-bb46-222acf609143-maas-api-service-ca" pod "ed9e3fd1-ccad-4d33-bb46-222acf609143" (UID: "ed9e3fd1-ccad-4d33-bb46-222acf609143"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:23.692664 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.691343 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ee9297b-cfb8-4154-bb20-d5eb73f78da2-kube-api-access-hq5nd" pod "8ee9297b-cfb8-4154-bb20-d5eb73f78da2" (UID: "8ee9297b-cfb8-4154-bb20-d5eb73f78da2"). InnerVolumeSpecName "kube-api-access-hq5nd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:23.692664 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.691376 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed9e3fd1-ccad-4d33-bb46-222acf609143-kube-api-access-bfq4p" pod "ed9e3fd1-ccad-4d33-bb46-222acf609143" (UID: "ed9e3fd1-ccad-4d33-bb46-222acf609143"). InnerVolumeSpecName "kube-api-access-bfq4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:23.693482 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.693447 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a87157ec-dde9-4537-a2dc-d2a0e0c80091-kube-api-access-xhmsl" pod "a87157ec-dde9-4537-a2dc-d2a0e0c80091" (UID: "a87157ec-dde9-4537-a2dc-d2a0e0c80091"). InnerVolumeSpecName "kube-api-access-xhmsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:23.792593 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.792514 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed9e3fd1-ccad-4d33-bb46-222acf609143-maas-api-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:45:23.792593 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.792553 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfq4p\" (UniqueName: \"kubernetes.io/projected/ed9e3fd1-ccad-4d33-bb46-222acf609143-kube-api-access-bfq4p\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:45:23.792593 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.792571 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhmsl\" (UniqueName: \"kubernetes.io/projected/a87157ec-dde9-4537-a2dc-d2a0e0c80091-kube-api-access-xhmsl\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:45:23.792593 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:23.792586 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq5nd\" (UniqueName: \"kubernetes.io/projected/8ee9297b-cfb8-4154-bb20-d5eb73f78da2-kube-api-access-hq5nd\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:45:24.266512 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:24.266480 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889765-4vxnf" Oct 30 18:45:24.266962 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:24.266938 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889765-4vxnf" event={"ID":"ed9e3fd1-ccad-4d33-bb46-222acf609143","Type":"ContainerDied","Data":"b9dfaa43bd276ef66d2b643705172831f5a69e5c73cab20244ffcf858bb1516b"} Oct 30 18:45:24.267058 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:24.267049 4442 scope.go:122] "RemoveContainer" containerID="2ae86d1a7dc34b0f64bb30f58edf4cb52dd56ab7bcc7518dea9974bd64aafbfa" Oct 30 18:45:24.270493 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:24.270471 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889765-qf9bg" event={"ID":"a87157ec-dde9-4537-a2dc-d2a0e0c80091","Type":"ContainerDied","Data":"64b6556e2f19dab5db9865b6db6974e103a555f41eb8c9aceadb86a6c3415361"} Oct 30 18:45:24.270922 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:24.270486 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889765-qf9bg" Oct 30 18:45:24.278556 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:24.278533 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889765-v4sqc" event={"ID":"8ee9297b-cfb8-4154-bb20-d5eb73f78da2","Type":"ContainerDied","Data":"0939bff7932657b465de4a683fa026813f711b5bc73b1dafea5cf7ff54d7d917"} Oct 30 18:45:24.278793 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:24.278778 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889765-v4sqc" Oct 30 18:45:24.283719 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:24.283696 4442 scope.go:122] "RemoveContainer" containerID="691a3eb920a9c90ea40c93458a8d75092f7998c36f946b4b3f45732744bbc0cf" Oct 30 18:45:24.296581 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:24.296490 4442 scope.go:122] "RemoveContainer" containerID="934686e473c3e8ee948864fadf7a2330824bf17feb13bce78d32f5ddbccdc0cd" Oct 30 18:45:24.362489 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:24.362452 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889765-4vxnf"] Oct 30 18:45:24.376060 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:24.376001 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889765-4vxnf"] Oct 30 18:45:24.680112 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:24.680074 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889765-v5zh2"] Oct 30 18:45:24.720481 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:24.720233 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889765-v4sqc"] Oct 30 18:45:24.728191 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:24.728168 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889765-v4sqc"] Oct 30 18:45:24.731838 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:24.731759 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889765-fsnp2" Oct 30 18:45:24.756926 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:24.756862 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889765-qf9bg"] Oct 30 18:45:24.763109 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:24.763058 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889765-qf9bg"] Oct 30 18:45:24.766469 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:24.766446 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7cdca9f-f772-449c-9f30-067437ef08cc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7cdca9f-f772-449c-9f30-067437ef08cc-maas-api-service-ca\") pod \"f7cdca9f-f772-449c-9f30-067437ef08cc\" (UID: \"f7cdca9f-f772-449c-9f30-067437ef08cc\") " Oct 30 18:45:24.767400 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:24.766787 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7cdca9f-f772-449c-9f30-067437ef08cc-kube-api-access-2zdcz\" (UniqueName: \"kubernetes.io/projected/f7cdca9f-f772-449c-9f30-067437ef08cc-kube-api-access-2zdcz\") pod \"f7cdca9f-f772-449c-9f30-067437ef08cc\" (UID: \"f7cdca9f-f772-449c-9f30-067437ef08cc\") " Oct 30 18:45:24.767400 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:24.766985 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7cdca9f-f772-449c-9f30-067437ef08cc-maas-api-service-ca" pod "f7cdca9f-f772-449c-9f30-067437ef08cc" (UID: "f7cdca9f-f772-449c-9f30-067437ef08cc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:24.767400 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:24.767345 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7cdca9f-f772-449c-9f30-067437ef08cc-maas-api-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:45:24.769400 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:24.769372 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7cdca9f-f772-449c-9f30-067437ef08cc-kube-api-access-2zdcz" pod "f7cdca9f-f772-449c-9f30-067437ef08cc" (UID: "f7cdca9f-f772-449c-9f30-067437ef08cc"). InnerVolumeSpecName "kube-api-access-2zdcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:24.875591 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:24.872236 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zdcz\" (UniqueName: \"kubernetes.io/projected/f7cdca9f-f772-449c-9f30-067437ef08cc-kube-api-access-2zdcz\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:45:24.915226 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:24.915203 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889765-v5zh2" Oct 30 18:45:25.071765 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:25.071675 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e4c2309-c818-46be-abd4-add20576fd6a" path="/var/lib/kubelet/pods/1e4c2309-c818-46be-abd4-add20576fd6a/volumes" Oct 30 18:45:25.072440 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:25.072417 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f486a50-76a3-4cc7-bd5b-3edb80ad3315" path="/var/lib/kubelet/pods/4f486a50-76a3-4cc7-bd5b-3edb80ad3315/volumes" Oct 30 18:45:25.073047 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:25.073029 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ee9297b-cfb8-4154-bb20-d5eb73f78da2" path="/var/lib/kubelet/pods/8ee9297b-cfb8-4154-bb20-d5eb73f78da2/volumes" Oct 30 18:45:25.074100 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:25.074072 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a87157ec-dde9-4537-a2dc-d2a0e0c80091" path="/var/lib/kubelet/pods/a87157ec-dde9-4537-a2dc-d2a0e0c80091/volumes" Oct 30 18:45:25.074702 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:25.074682 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed9e3fd1-ccad-4d33-bb46-222acf609143" path="/var/lib/kubelet/pods/ed9e3fd1-ccad-4d33-bb46-222acf609143/volumes" Oct 30 18:45:25.075197 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:25.075178 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e752ff2-638c-4b1f-94a9-5efd1a53a9c6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e752ff2-638c-4b1f-94a9-5efd1a53a9c6-maas-api-service-ca\") pod \"5e752ff2-638c-4b1f-94a9-5efd1a53a9c6\" (UID: \"5e752ff2-638c-4b1f-94a9-5efd1a53a9c6\") " Oct 30 18:45:25.075277 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:25.075265 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e752ff2-638c-4b1f-94a9-5efd1a53a9c6-kube-api-access-vfnd8\" (UniqueName: \"kubernetes.io/projected/5e752ff2-638c-4b1f-94a9-5efd1a53a9c6-kube-api-access-vfnd8\") pod \"5e752ff2-638c-4b1f-94a9-5efd1a53a9c6\" (UID: \"5e752ff2-638c-4b1f-94a9-5efd1a53a9c6\") " Oct 30 18:45:25.075311 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:25.075276 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04" path="/var/lib/kubelet/pods/f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04/volumes" Oct 30 18:45:25.075524 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:25.075505 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e752ff2-638c-4b1f-94a9-5efd1a53a9c6-maas-api-service-ca" pod "5e752ff2-638c-4b1f-94a9-5efd1a53a9c6" (UID: "5e752ff2-638c-4b1f-94a9-5efd1a53a9c6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:25.075719 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:25.075705 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e752ff2-638c-4b1f-94a9-5efd1a53a9c6-maas-api-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:45:25.078042 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:25.078022 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e752ff2-638c-4b1f-94a9-5efd1a53a9c6-kube-api-access-vfnd8" pod "5e752ff2-638c-4b1f-94a9-5efd1a53a9c6" (UID: "5e752ff2-638c-4b1f-94a9-5efd1a53a9c6"). InnerVolumeSpecName "kube-api-access-vfnd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:25.177579 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:25.177544 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfnd8\" (UniqueName: \"kubernetes.io/projected/5e752ff2-638c-4b1f-94a9-5efd1a53a9c6-kube-api-access-vfnd8\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:45:25.291419 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:25.291383 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889765-fsnp2" Oct 30 18:45:25.291419 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:25.291406 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889765-fsnp2" event={"ID":"f7cdca9f-f772-449c-9f30-067437ef08cc","Type":"ContainerDied","Data":"16a09ff3b8ced965d48c79366502e04ecf5b230fef0f5426e0e480d1864b68fb"} Oct 30 18:45:25.291980 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:25.291459 4442 scope.go:122] "RemoveContainer" containerID="69b4a4f024c42f1296ea98bc5c7dff1d8b5d686ec55b7d9b781d2bb378fa92be" Oct 30 18:45:25.295648 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:25.295624 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889765-v5zh2" Oct 30 18:45:25.295648 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:25.295632 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889765-v5zh2" event={"ID":"5e752ff2-638c-4b1f-94a9-5efd1a53a9c6","Type":"ContainerDied","Data":"7b7b0b2ff5e88ed2660835a323fc5ba32e3ea4c77f666ecd53f42d956af00d04"} Oct 30 18:45:25.313172 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:25.313112 4442 scope.go:122] "RemoveContainer" containerID="ea68b2e62624284e714a20170e1ee85d94661da3bfd1d4d3837813d5d4ca9766" Oct 30 18:45:25.345200 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:25.345119 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889765-fsnp2"] Oct 30 18:45:25.350086 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:25.350058 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889765-fsnp2"] Oct 30 18:45:25.397539 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:25.397490 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889765-v5zh2"] Oct 30 18:45:25.401876 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:25.401851 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889765-v5zh2"] Oct 30 18:45:27.069288 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:27.069255 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e752ff2-638c-4b1f-94a9-5efd1a53a9c6" path="/var/lib/kubelet/pods/5e752ff2-638c-4b1f-94a9-5efd1a53a9c6/volumes" Oct 30 18:45:27.069959 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:45:27.069939 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7cdca9f-f772-449c-9f30-067437ef08cc" path="/var/lib/kubelet/pods/f7cdca9f-f772-449c-9f30-067437ef08cc/volumes" Oct 30 18:45:47.565293 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:45:47.565251 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:46:17.561635 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:46:17.561595 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:46:47.571201 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:46:47.571156 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:47:10.486276 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:10.485275 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-x2rr2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:47:10.486276 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:10.485349 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" podUID="e03523d4-1b6f-4a80-9006-26b0345d809d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:47:17.565349 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:47:17.565306 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:47:38.188978 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.188943 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4mfclf"] Oct 30 18:47:38.190111 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190088 4442 cpu_manager.go:414] "containerMap: removing container" podUID="83b551f2-22ff-42f3-acc9-3122313400d8" containerName="cleanup" Oct 30 18:47:38.190209 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190115 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83b551f2-22ff-42f3-acc9-3122313400d8" containerName="cleanup" Oct 30 18:47:38.190209 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190158 4442 cpu_manager.go:414] "containerMap: removing container" podUID="f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04" containerName="cleanup" Oct 30 18:47:38.190209 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190167 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04" containerName="cleanup" Oct 30 18:47:38.190209 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190181 4442 cpu_manager.go:414] "containerMap: removing container" podUID="f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04" containerName="cleanup" Oct 30 18:47:38.190209 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190189 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04" containerName="cleanup" Oct 30 18:47:38.190209 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190200 4442 cpu_manager.go:414] "containerMap: removing container" podUID="9b700c35-c957-4b7a-882e-bf82dd79f7ff" containerName="cleanup" Oct 30 18:47:38.190209 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190207 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b700c35-c957-4b7a-882e-bf82dd79f7ff" containerName="cleanup" Oct 30 18:47:38.190489 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190221 4442 cpu_manager.go:414] "containerMap: removing container" podUID="8ee9297b-cfb8-4154-bb20-d5eb73f78da2" containerName="cleanup" Oct 30 18:47:38.190489 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190228 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ee9297b-cfb8-4154-bb20-d5eb73f78da2" containerName="cleanup" Oct 30 18:47:38.190489 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190239 4442 cpu_manager.go:414] "containerMap: removing container" podUID="5e752ff2-638c-4b1f-94a9-5efd1a53a9c6" containerName="cleanup" Oct 30 18:47:38.190489 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190247 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e752ff2-638c-4b1f-94a9-5efd1a53a9c6" containerName="cleanup" Oct 30 18:47:38.190489 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190262 4442 cpu_manager.go:414] "containerMap: removing container" podUID="36a43766-ec94-4991-bad0-e3d4ed5122c8" containerName="cleanup" Oct 30 18:47:38.190489 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190269 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36a43766-ec94-4991-bad0-e3d4ed5122c8" containerName="cleanup" Oct 30 18:47:38.190489 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190280 4442 cpu_manager.go:414] "containerMap: removing container" podUID="83b551f2-22ff-42f3-acc9-3122313400d8" containerName="cleanup" Oct 30 18:47:38.190489 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190286 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83b551f2-22ff-42f3-acc9-3122313400d8" containerName="cleanup" Oct 30 18:47:38.190489 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190301 4442 cpu_manager.go:414] "containerMap: removing container" podUID="4f486a50-76a3-4cc7-bd5b-3edb80ad3315" containerName="cleanup" Oct 30 18:47:38.190489 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190308 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f486a50-76a3-4cc7-bd5b-3edb80ad3315" containerName="cleanup" Oct 30 18:47:38.190489 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190320 4442 cpu_manager.go:414] "containerMap: removing container" podUID="7d52db28-5cd7-4a2a-af8b-b876d41c8453" containerName="cleanup" Oct 30 18:47:38.190489 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190338 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d52db28-5cd7-4a2a-af8b-b876d41c8453" containerName="cleanup" Oct 30 18:47:38.190489 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190349 4442 cpu_manager.go:414] "containerMap: removing container" podUID="f7cdca9f-f772-449c-9f30-067437ef08cc" containerName="cleanup" Oct 30 18:47:38.190489 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190356 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7cdca9f-f772-449c-9f30-067437ef08cc" containerName="cleanup" Oct 30 18:47:38.190489 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190368 4442 cpu_manager.go:414] "containerMap: removing container" podUID="f7cdca9f-f772-449c-9f30-067437ef08cc" containerName="cleanup" Oct 30 18:47:38.190489 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190375 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7cdca9f-f772-449c-9f30-067437ef08cc" containerName="cleanup" Oct 30 18:47:38.190489 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190389 4442 cpu_manager.go:414] "containerMap: removing container" podUID="ed9e3fd1-ccad-4d33-bb46-222acf609143" containerName="cleanup" Oct 30 18:47:38.190489 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190396 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed9e3fd1-ccad-4d33-bb46-222acf609143" containerName="cleanup" Oct 30 18:47:38.190489 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190413 4442 cpu_manager.go:414] "containerMap: removing container" podUID="8ee9297b-cfb8-4154-bb20-d5eb73f78da2" containerName="cleanup" Oct 30 18:47:38.190489 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190422 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ee9297b-cfb8-4154-bb20-d5eb73f78da2" containerName="cleanup" Oct 30 18:47:38.190489 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190431 4442 cpu_manager.go:414] "containerMap: removing container" podUID="5e752ff2-638c-4b1f-94a9-5efd1a53a9c6" containerName="cleanup" Oct 30 18:47:38.190489 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190438 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e752ff2-638c-4b1f-94a9-5efd1a53a9c6" containerName="cleanup" Oct 30 18:47:38.190489 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190452 4442 cpu_manager.go:414] "containerMap: removing container" podUID="a87157ec-dde9-4537-a2dc-d2a0e0c80091" containerName="cleanup" Oct 30 18:47:38.190489 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190459 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a87157ec-dde9-4537-a2dc-d2a0e0c80091" containerName="cleanup" Oct 30 18:47:38.190489 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190471 4442 cpu_manager.go:414] "containerMap: removing container" podUID="83b551f2-22ff-42f3-acc9-3122313400d8" containerName="cleanup" Oct 30 18:47:38.190489 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190477 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83b551f2-22ff-42f3-acc9-3122313400d8" containerName="cleanup" Oct 30 18:47:38.190489 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190495 4442 cpu_manager.go:414] "containerMap: removing container" podUID="1e4c2309-c818-46be-abd4-add20576fd6a" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190503 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e4c2309-c818-46be-abd4-add20576fd6a" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190518 4442 cpu_manager.go:414] "containerMap: removing container" podUID="7d52db28-5cd7-4a2a-af8b-b876d41c8453" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190525 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d52db28-5cd7-4a2a-af8b-b876d41c8453" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190538 4442 cpu_manager.go:414] "containerMap: removing container" podUID="8ee9297b-cfb8-4154-bb20-d5eb73f78da2" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190545 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ee9297b-cfb8-4154-bb20-d5eb73f78da2" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190557 4442 cpu_manager.go:414] "containerMap: removing container" podUID="7d52db28-5cd7-4a2a-af8b-b876d41c8453" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190564 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d52db28-5cd7-4a2a-af8b-b876d41c8453" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190576 4442 cpu_manager.go:414] "containerMap: removing container" podUID="a87157ec-dde9-4537-a2dc-d2a0e0c80091" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190583 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a87157ec-dde9-4537-a2dc-d2a0e0c80091" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190602 4442 cpu_manager.go:414] "containerMap: removing container" podUID="1e4c2309-c818-46be-abd4-add20576fd6a" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190609 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e4c2309-c818-46be-abd4-add20576fd6a" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190622 4442 cpu_manager.go:414] "containerMap: removing container" podUID="a87157ec-dde9-4537-a2dc-d2a0e0c80091" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190628 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a87157ec-dde9-4537-a2dc-d2a0e0c80091" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190644 4442 cpu_manager.go:414] "containerMap: removing container" podUID="9b700c35-c957-4b7a-882e-bf82dd79f7ff" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190650 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b700c35-c957-4b7a-882e-bf82dd79f7ff" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190663 4442 cpu_manager.go:414] "containerMap: removing container" podUID="ed9e3fd1-ccad-4d33-bb46-222acf609143" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190669 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed9e3fd1-ccad-4d33-bb46-222acf609143" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190685 4442 cpu_manager.go:414] "containerMap: removing container" podUID="4f486a50-76a3-4cc7-bd5b-3edb80ad3315" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190691 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f486a50-76a3-4cc7-bd5b-3edb80ad3315" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190704 4442 cpu_manager.go:414] "containerMap: removing container" podUID="36a43766-ec94-4991-bad0-e3d4ed5122c8" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190711 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36a43766-ec94-4991-bad0-e3d4ed5122c8" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190923 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="83b551f2-22ff-42f3-acc9-3122313400d8" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190935 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190941 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190947 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b700c35-c957-4b7a-882e-bf82dd79f7ff" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190952 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e752ff2-638c-4b1f-94a9-5efd1a53a9c6" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190959 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ee9297b-cfb8-4154-bb20-d5eb73f78da2" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190965 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="36a43766-ec94-4991-bad0-e3d4ed5122c8" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190970 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f486a50-76a3-4cc7-bd5b-3edb80ad3315" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190975 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d52db28-5cd7-4a2a-af8b-b876d41c8453" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190982 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7cdca9f-f772-449c-9f30-067437ef08cc" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190988 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7cdca9f-f772-449c-9f30-067437ef08cc" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190993 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed9e3fd1-ccad-4d33-bb46-222acf609143" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.190999 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e752ff2-638c-4b1f-94a9-5efd1a53a9c6" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.191004 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="a87157ec-dde9-4537-a2dc-d2a0e0c80091" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.191010 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="83b551f2-22ff-42f3-acc9-3122313400d8" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.191016 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e4c2309-c818-46be-abd4-add20576fd6a" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.191021 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d52db28-5cd7-4a2a-af8b-b876d41c8453" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.191026 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ee9297b-cfb8-4154-bb20-d5eb73f78da2" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.191032 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d52db28-5cd7-4a2a-af8b-b876d41c8453" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.191038 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="a87157ec-dde9-4537-a2dc-d2a0e0c80091" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.191044 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b700c35-c957-4b7a-882e-bf82dd79f7ff" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.191048 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="a87157ec-dde9-4537-a2dc-d2a0e0c80091" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.191053 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed9e3fd1-ccad-4d33-bb46-222acf609143" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.191057 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f486a50-76a3-4cc7-bd5b-3edb80ad3315" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.191063 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed9e3fd1-ccad-4d33-bb46-222acf609143" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.191069 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="36a43766-ec94-4991-bad0-e3d4ed5122c8" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.191074 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7cdca9f-f772-449c-9f30-067437ef08cc" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.191081 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="36a43766-ec94-4991-bad0-e3d4ed5122c8" containerName="cleanup" Oct 30 18:47:38.191455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.191089 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e4c2309-c818-46be-abd4-add20576fd6a" containerName="cleanup" Oct 30 18:47:38.195083 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.195064 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4mfclf" Oct 30 18:47:38.211769 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.211738 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4mfclf"] Oct 30 18:47:38.217361 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.216819 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-713dc8\"/\"payload-processing-external-model-plugins-e2e-extmod-a-713dc8\"" Oct 30 18:47:38.217361 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.216949 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-713dc8\"/\"openshift-service-ca.crt\"" Oct 30 18:47:38.217361 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.217118 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-713dc8\"/\"kube-root-ca.crt\"" Oct 30 18:47:38.217361 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.217215 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-713dc8\"/\"routing-overlay\"" Oct 30 18:47:38.217361 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.217279 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-713dc8\"/\"payload-processing-external-model-e2e--40d660e1-dockercfg-czmv5\"" Oct 30 18:47:38.217361 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.217336 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-713dc8\"/\"agc-cred-12bcdc53060d\"" Oct 30 18:47:38.342148 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.342097 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8046286d-a28b-41d2-916b-027baf316251-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-85c4mfclf\" (UID: \"8046286d-a28b-41d2-916b-027baf316251\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4mfclf" Oct 30 18:47:38.342413 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.342389 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhfdm\" (UniqueName: \"kubernetes.io/projected/8046286d-a28b-41d2-916b-027baf316251-kube-api-access-rhfdm\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-85c4mfclf\" (UID: \"8046286d-a28b-41d2-916b-027baf316251\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4mfclf" Oct 30 18:47:38.342641 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.342615 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8046286d-a28b-41d2-916b-027baf316251-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-85c4mfclf\" (UID: \"8046286d-a28b-41d2-916b-027baf316251\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4mfclf" Oct 30 18:47:38.342773 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.342746 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8046286d-a28b-41d2-916b-027baf316251-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-85c4mfclf\" (UID: \"8046286d-a28b-41d2-916b-027baf316251\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4mfclf" Oct 30 18:47:38.444322 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.444221 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhfdm\" (UniqueName: \"kubernetes.io/projected/8046286d-a28b-41d2-916b-027baf316251-kube-api-access-rhfdm\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-85c4mfclf\" (UID: \"8046286d-a28b-41d2-916b-027baf316251\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4mfclf" Oct 30 18:47:38.445154 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.445111 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8046286d-a28b-41d2-916b-027baf316251-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-85c4mfclf\" (UID: \"8046286d-a28b-41d2-916b-027baf316251\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4mfclf" Oct 30 18:47:38.456734 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.456707 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8046286d-a28b-41d2-916b-027baf316251-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-85c4mfclf\" (UID: \"8046286d-a28b-41d2-916b-027baf316251\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4mfclf" Oct 30 18:47:38.457290 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.457267 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8046286d-a28b-41d2-916b-027baf316251-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-85c4mfclf\" (UID: \"8046286d-a28b-41d2-916b-027baf316251\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4mfclf" Oct 30 18:47:38.458024 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.456400 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8046286d-a28b-41d2-916b-027baf316251-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-85c4mfclf\" (UID: \"8046286d-a28b-41d2-916b-027baf316251\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4mfclf" Oct 30 18:47:38.458680 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.458622 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8046286d-a28b-41d2-916b-027baf316251-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-85c4mfclf\" (UID: \"8046286d-a28b-41d2-916b-027baf316251\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4mfclf" Oct 30 18:47:38.476467 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.476404 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8046286d-a28b-41d2-916b-027baf316251-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-85c4mfclf\" (UID: \"8046286d-a28b-41d2-916b-027baf316251\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4mfclf" Oct 30 18:47:38.518618 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.518549 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhfdm\" (UniqueName: \"kubernetes.io/projected/8046286d-a28b-41d2-916b-027baf316251-kube-api-access-rhfdm\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-85c4mfclf\" (UID: \"8046286d-a28b-41d2-916b-027baf316251\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4mfclf" Oct 30 18:47:38.519080 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.519057 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4mfclf" Oct 30 18:47:38.951512 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:47:38.951482 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8046286d_a28b_41d2_916b_027baf316251.slice/crio-9955d35f74445ef24678ca11cb6277addf1f4dffa6116729c90512d9fbd5b062 WatchSource:0}: Error finding container 9955d35f74445ef24678ca11cb6277addf1f4dffa6116729c90512d9fbd5b062: Status 404 returned error can't find the container with id 9955d35f74445ef24678ca11cb6277addf1f4dffa6116729c90512d9fbd5b062 Oct 30 18:47:38.951718 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:38.951689 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4mfclf"] Oct 30 18:47:39.227031 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:39.226950 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4mfclf" event={"ID":"8046286d-a28b-41d2-916b-027baf316251","Type":"ContainerStarted","Data":"e484c2c9496828733611646c0f76b899e43ef236e49de7ea612e35737bb57e2d"} Oct 30 18:47:39.227031 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:39.227005 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4mfclf" event={"ID":"8046286d-a28b-41d2-916b-027baf316251","Type":"ContainerStarted","Data":"9955d35f74445ef24678ca11cb6277addf1f4dffa6116729c90512d9fbd5b062"} Oct 30 18:47:39.227477 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:39.227071 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4mfclf" Oct 30 18:47:39.266939 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:39.266884 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4mfclf" podStartSLOduration=1.26687138 podStartE2EDuration="1.26687138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:39.264728775 +0000 UTC m=+2956.354791115" watchObservedRunningTime="2026-10-30 18:47:39.26687138 +0000 UTC m=+2956.356933719" Oct 30 18:47:40.567299 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.567258 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9"] Oct 30 18:47:40.568206 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.568187 4442 cpu_manager.go:414] "containerMap: removing container" podUID="5e752ff2-638c-4b1f-94a9-5efd1a53a9c6" containerName="cleanup" Oct 30 18:47:40.568333 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.568208 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e752ff2-638c-4b1f-94a9-5efd1a53a9c6" containerName="cleanup" Oct 30 18:47:40.568333 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.568240 4442 cpu_manager.go:414] "containerMap: removing container" podUID="1e4c2309-c818-46be-abd4-add20576fd6a" containerName="cleanup" Oct 30 18:47:40.568333 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.568246 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e4c2309-c818-46be-abd4-add20576fd6a" containerName="cleanup" Oct 30 18:47:40.568333 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.568263 4442 cpu_manager.go:414] "containerMap: removing container" podUID="ed9e3fd1-ccad-4d33-bb46-222acf609143" containerName="cleanup" Oct 30 18:47:40.568333 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.568267 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed9e3fd1-ccad-4d33-bb46-222acf609143" containerName="cleanup" Oct 30 18:47:40.568333 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.568308 4442 cpu_manager.go:414] "containerMap: removing container" podUID="f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04" containerName="cleanup" Oct 30 18:47:40.568333 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.568315 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04" containerName="cleanup" Oct 30 18:47:40.568333 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.568325 4442 cpu_manager.go:414] "containerMap: removing container" podUID="9b700c35-c957-4b7a-882e-bf82dd79f7ff" containerName="cleanup" Oct 30 18:47:40.568333 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.568330 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b700c35-c957-4b7a-882e-bf82dd79f7ff" containerName="cleanup" Oct 30 18:47:40.568762 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.568340 4442 cpu_manager.go:414] "containerMap: removing container" podUID="36a43766-ec94-4991-bad0-e3d4ed5122c8" containerName="cleanup" Oct 30 18:47:40.568762 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.568344 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36a43766-ec94-4991-bad0-e3d4ed5122c8" containerName="cleanup" Oct 30 18:47:40.568762 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.568353 4442 cpu_manager.go:414] "containerMap: removing container" podUID="4f486a50-76a3-4cc7-bd5b-3edb80ad3315" containerName="cleanup" Oct 30 18:47:40.568762 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.568360 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f486a50-76a3-4cc7-bd5b-3edb80ad3315" containerName="cleanup" Oct 30 18:47:40.568762 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.568386 4442 cpu_manager.go:414] "containerMap: removing container" podUID="f7cdca9f-f772-449c-9f30-067437ef08cc" containerName="cleanup" Oct 30 18:47:40.568762 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.568392 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7cdca9f-f772-449c-9f30-067437ef08cc" containerName="cleanup" Oct 30 18:47:40.568762 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.568544 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ee9297b-cfb8-4154-bb20-d5eb73f78da2" containerName="cleanup" Oct 30 18:47:40.568762 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.568558 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b700c35-c957-4b7a-882e-bf82dd79f7ff" containerName="cleanup" Oct 30 18:47:40.568762 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.568577 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="f730b844-ae3a-4ec4-b4a1-ee2dd4e88e04" containerName="cleanup" Oct 30 18:47:40.568762 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.568585 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e752ff2-638c-4b1f-94a9-5efd1a53a9c6" containerName="cleanup" Oct 30 18:47:40.568762 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.568596 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f486a50-76a3-4cc7-bd5b-3edb80ad3315" containerName="cleanup" Oct 30 18:47:40.568762 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.568608 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="83b551f2-22ff-42f3-acc9-3122313400d8" containerName="cleanup" Oct 30 18:47:40.568762 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.568617 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e4c2309-c818-46be-abd4-add20576fd6a" containerName="cleanup" Oct 30 18:47:40.572562 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.572543 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" Oct 30 18:47:40.604870 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.604835 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9"] Oct 30 18:47:40.698924 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.698887 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1e466ee-ec7a-4640-bbc5-2f16795107cf-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-z2dn9\" (UID: \"e1e466ee-ec7a-4640-bbc5-2f16795107cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" Oct 30 18:47:40.699123 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.699031 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1e466ee-ec7a-4640-bbc5-2f16795107cf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-z2dn9\" (UID: \"e1e466ee-ec7a-4640-bbc5-2f16795107cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" Oct 30 18:47:40.699258 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.699237 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1e466ee-ec7a-4640-bbc5-2f16795107cf-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-z2dn9\" (UID: \"e1e466ee-ec7a-4640-bbc5-2f16795107cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" Oct 30 18:47:40.699307 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.699289 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1e466ee-ec7a-4640-bbc5-2f16795107cf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-z2dn9\" (UID: \"e1e466ee-ec7a-4640-bbc5-2f16795107cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" Oct 30 18:47:40.699390 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.699373 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1e466ee-ec7a-4640-bbc5-2f16795107cf-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-z2dn9\" (UID: \"e1e466ee-ec7a-4640-bbc5-2f16795107cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" Oct 30 18:47:40.699429 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.699403 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs4sd\" (UniqueName: \"kubernetes.io/projected/e1e466ee-ec7a-4640-bbc5-2f16795107cf-kube-api-access-gs4sd\") pod \"maas-default-gateway-openshift-default-76597d6855-z2dn9\" (UID: \"e1e466ee-ec7a-4640-bbc5-2f16795107cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" Oct 30 18:47:40.699562 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.699546 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1e466ee-ec7a-4640-bbc5-2f16795107cf-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-z2dn9\" (UID: \"e1e466ee-ec7a-4640-bbc5-2f16795107cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" Oct 30 18:47:40.699605 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.699572 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1e466ee-ec7a-4640-bbc5-2f16795107cf-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-z2dn9\" (UID: \"e1e466ee-ec7a-4640-bbc5-2f16795107cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" Oct 30 18:47:40.699674 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.699659 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1e466ee-ec7a-4640-bbc5-2f16795107cf-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-z2dn9\" (UID: \"e1e466ee-ec7a-4640-bbc5-2f16795107cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" Oct 30 18:47:40.801083 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.801049 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1e466ee-ec7a-4640-bbc5-2f16795107cf-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-z2dn9\" (UID: \"e1e466ee-ec7a-4640-bbc5-2f16795107cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" Oct 30 18:47:40.801083 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.801087 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1e466ee-ec7a-4640-bbc5-2f16795107cf-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-z2dn9\" (UID: \"e1e466ee-ec7a-4640-bbc5-2f16795107cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" Oct 30 18:47:40.801392 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.801170 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1e466ee-ec7a-4640-bbc5-2f16795107cf-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-z2dn9\" (UID: \"e1e466ee-ec7a-4640-bbc5-2f16795107cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" Oct 30 18:47:40.801392 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.801220 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1e466ee-ec7a-4640-bbc5-2f16795107cf-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-z2dn9\" (UID: \"e1e466ee-ec7a-4640-bbc5-2f16795107cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" Oct 30 18:47:40.801392 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.801295 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1e466ee-ec7a-4640-bbc5-2f16795107cf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-z2dn9\" (UID: \"e1e466ee-ec7a-4640-bbc5-2f16795107cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" Oct 30 18:47:40.801392 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.801369 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1e466ee-ec7a-4640-bbc5-2f16795107cf-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-z2dn9\" (UID: \"e1e466ee-ec7a-4640-bbc5-2f16795107cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" Oct 30 18:47:40.801645 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.801434 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1e466ee-ec7a-4640-bbc5-2f16795107cf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-z2dn9\" (UID: \"e1e466ee-ec7a-4640-bbc5-2f16795107cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" Oct 30 18:47:40.801645 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.801547 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1e466ee-ec7a-4640-bbc5-2f16795107cf-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-z2dn9\" (UID: \"e1e466ee-ec7a-4640-bbc5-2f16795107cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" Oct 30 18:47:40.801645 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.801584 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1e466ee-ec7a-4640-bbc5-2f16795107cf-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-z2dn9\" (UID: \"e1e466ee-ec7a-4640-bbc5-2f16795107cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" Oct 30 18:47:40.801645 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.801605 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1e466ee-ec7a-4640-bbc5-2f16795107cf-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-z2dn9\" (UID: \"e1e466ee-ec7a-4640-bbc5-2f16795107cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" Oct 30 18:47:40.801645 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.801623 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs4sd\" (UniqueName: \"kubernetes.io/projected/e1e466ee-ec7a-4640-bbc5-2f16795107cf-kube-api-access-gs4sd\") pod \"maas-default-gateway-openshift-default-76597d6855-z2dn9\" (UID: \"e1e466ee-ec7a-4640-bbc5-2f16795107cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" Oct 30 18:47:40.801866 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.801658 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1e466ee-ec7a-4640-bbc5-2f16795107cf-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-z2dn9\" (UID: \"e1e466ee-ec7a-4640-bbc5-2f16795107cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" Oct 30 18:47:40.801866 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.801707 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1e466ee-ec7a-4640-bbc5-2f16795107cf-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-z2dn9\" (UID: \"e1e466ee-ec7a-4640-bbc5-2f16795107cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" Oct 30 18:47:40.802431 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.802408 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1e466ee-ec7a-4640-bbc5-2f16795107cf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-z2dn9\" (UID: \"e1e466ee-ec7a-4640-bbc5-2f16795107cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" Oct 30 18:47:40.804015 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.803991 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1e466ee-ec7a-4640-bbc5-2f16795107cf-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-z2dn9\" (UID: \"e1e466ee-ec7a-4640-bbc5-2f16795107cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" Oct 30 18:47:40.804272 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.804246 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1e466ee-ec7a-4640-bbc5-2f16795107cf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-z2dn9\" (UID: \"e1e466ee-ec7a-4640-bbc5-2f16795107cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" Oct 30 18:47:40.820881 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.820809 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1e466ee-ec7a-4640-bbc5-2f16795107cf-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-z2dn9\" (UID: \"e1e466ee-ec7a-4640-bbc5-2f16795107cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" Oct 30 18:47:40.821417 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.821392 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs4sd\" (UniqueName: \"kubernetes.io/projected/e1e466ee-ec7a-4640-bbc5-2f16795107cf-kube-api-access-gs4sd\") pod \"maas-default-gateway-openshift-default-76597d6855-z2dn9\" (UID: \"e1e466ee-ec7a-4640-bbc5-2f16795107cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" Oct 30 18:47:40.889385 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:40.889351 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" Oct 30 18:47:41.208973 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:47:41.208929 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e466ee_ec7a_4640_bbc5_2f16795107cf.slice/crio-9b452ed1ad564eeeb4d719e925cc2471b196a27c48bfb46a27b0cebd4ccf72d9 WatchSource:0}: Error finding container 9b452ed1ad564eeeb4d719e925cc2471b196a27c48bfb46a27b0cebd4ccf72d9: Status 404 returned error can't find the container with id 9b452ed1ad564eeeb4d719e925cc2471b196a27c48bfb46a27b0cebd4ccf72d9 Oct 30 18:47:41.210250 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:41.210224 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9"] Oct 30 18:47:41.211852 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:41.211820 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:47:41.211940 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:41.211886 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:47:41.211940 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:41.211928 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:47:41.246812 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:41.246781 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" event={"ID":"e1e466ee-ec7a-4640-bbc5-2f16795107cf","Type":"ContainerStarted","Data":"9b452ed1ad564eeeb4d719e925cc2471b196a27c48bfb46a27b0cebd4ccf72d9"} Oct 30 18:47:42.265177 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:42.265111 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" event={"ID":"e1e466ee-ec7a-4640-bbc5-2f16795107cf","Type":"ContainerStarted","Data":"57d653004220b68a80462e3bc6e347e90dd37acae6e8a362c5591a364cfe74fa"} Oct 30 18:47:42.327158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:42.325446 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" podStartSLOduration=2.3254251630000002 podStartE2EDuration="2.325425163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:42.318046301 +0000 UTC m=+2959.408108642" watchObservedRunningTime="2026-10-30 18:47:42.325425163 +0000 UTC m=+2959.415487503" Oct 30 18:47:42.890027 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:42.889988 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" Oct 30 18:47:42.892241 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:42.892215 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-z2dn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body= Oct 30 18:47:42.892368 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:42.892271 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" podUID="e1e466ee-ec7a-4640-bbc5-2f16795107cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" Oct 30 18:47:43.889709 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:43.889674 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-z2dn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body= Oct 30 18:47:43.890151 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:43.889733 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" podUID="e1e466ee-ec7a-4640-bbc5-2f16795107cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" Oct 30 18:47:44.890350 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:44.890312 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-z2dn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body= Oct 30 18:47:44.890802 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:44.890396 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" podUID="e1e466ee-ec7a-4640-bbc5-2f16795107cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" Oct 30 18:47:45.889814 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:45.889774 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-z2dn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body= Oct 30 18:47:45.890030 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:45.889861 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" podUID="e1e466ee-ec7a-4640-bbc5-2f16795107cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" Oct 30 18:47:46.890197 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:46.890157 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-z2dn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body= Oct 30 18:47:46.890616 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:46.890231 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" podUID="e1e466ee-ec7a-4640-bbc5-2f16795107cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" Oct 30 18:47:47.636359 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:47:47.636322 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:47:47.890201 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:47.890081 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-z2dn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body= Oct 30 18:47:47.890201 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:47.890176 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" podUID="e1e466ee-ec7a-4640-bbc5-2f16795107cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" Oct 30 18:47:48.889862 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:48.889824 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-z2dn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body= Oct 30 18:47:48.890047 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:48.889914 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" podUID="e1e466ee-ec7a-4640-bbc5-2f16795107cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" Oct 30 18:47:50.428155 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:50.415491 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4mfclf" Oct 30 18:47:50.931348 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:50.931303 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-z2dn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:47:50.931599 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:50.931379 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" podUID="e1e466ee-ec7a-4640-bbc5-2f16795107cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:47:51.181573 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:51.181485 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" Oct 30 18:47:51.183248 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:51.183076 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" Oct 30 18:47:51.186893 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:51.186869 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" Oct 30 18:47:59.157066 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:47:59.157022 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4mfclf"] Oct 30 18:48:04.161157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:04.160448 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4mfclf" podUID="8046286d-a28b-41d2-916b-027baf316251" containerName="payload-processing" containerID="cri-o://e484c2c9496828733611646c0f76b899e43ef236e49de7ea612e35737bb57e2d" gracePeriod=25 Oct 30 18:48:04.721340 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:04.721270 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4mfclf" Oct 30 18:48:04.898243 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:04.898213 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8046286d-a28b-41d2-916b-027baf316251-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8046286d-a28b-41d2-916b-027baf316251-routing-overlay-volume\") pod \"8046286d-a28b-41d2-916b-027baf316251\" (UID: \"8046286d-a28b-41d2-916b-027baf316251\") " Oct 30 18:48:04.898429 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:04.898316 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8046286d-a28b-41d2-916b-027baf316251-kube-api-access-rhfdm\" (UniqueName: \"kubernetes.io/projected/8046286d-a28b-41d2-916b-027baf316251-kube-api-access-rhfdm\") pod \"8046286d-a28b-41d2-916b-027baf316251\" (UID: \"8046286d-a28b-41d2-916b-027baf316251\") " Oct 30 18:48:04.898498 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:04.898481 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8046286d-a28b-41d2-916b-027baf316251-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8046286d-a28b-41d2-916b-027baf316251-provider-credentials\") pod \"8046286d-a28b-41d2-916b-027baf316251\" (UID: \"8046286d-a28b-41d2-916b-027baf316251\") " Oct 30 18:48:04.898786 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:04.898664 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8046286d-a28b-41d2-916b-027baf316251-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8046286d-a28b-41d2-916b-027baf316251-plugins-config-volume\") pod \"8046286d-a28b-41d2-916b-027baf316251\" (UID: \"8046286d-a28b-41d2-916b-027baf316251\") " Oct 30 18:48:04.898786 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:04.898692 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8046286d-a28b-41d2-916b-027baf316251-routing-overlay-volume" pod "8046286d-a28b-41d2-916b-027baf316251" (UID: "8046286d-a28b-41d2-916b-027baf316251"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:48:04.899152 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:04.899113 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8046286d-a28b-41d2-916b-027baf316251-plugins-config-volume" pod "8046286d-a28b-41d2-916b-027baf316251" (UID: "8046286d-a28b-41d2-916b-027baf316251"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:48:04.899397 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:04.899365 4442 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8046286d-a28b-41d2-916b-027baf316251-routing-overlay-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:48:04.899397 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:04.899389 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8046286d-a28b-41d2-916b-027baf316251-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:48:04.901642 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:04.901604 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8046286d-a28b-41d2-916b-027baf316251-kube-api-access-rhfdm" pod "8046286d-a28b-41d2-916b-027baf316251" (UID: "8046286d-a28b-41d2-916b-027baf316251"). InnerVolumeSpecName "kube-api-access-rhfdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:48:04.901806 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:04.901661 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8046286d-a28b-41d2-916b-027baf316251-provider-credentials" pod "8046286d-a28b-41d2-916b-027baf316251" (UID: "8046286d-a28b-41d2-916b-027baf316251"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:48:04.905944 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:04.905919 4442 generic.go:356] "Generic (PLEG): container finished" podID="8046286d-a28b-41d2-916b-027baf316251" containerID="e484c2c9496828733611646c0f76b899e43ef236e49de7ea612e35737bb57e2d" exitCode=0 Oct 30 18:48:04.906048 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:04.905983 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4mfclf" Oct 30 18:48:04.906048 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:04.906001 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4mfclf" event={"ID":"8046286d-a28b-41d2-916b-027baf316251","Type":"ContainerDied","Data":"e484c2c9496828733611646c0f76b899e43ef236e49de7ea612e35737bb57e2d"} Oct 30 18:48:04.906159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:04.906063 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4mfclf" event={"ID":"8046286d-a28b-41d2-916b-027baf316251","Type":"ContainerDied","Data":"9955d35f74445ef24678ca11cb6277addf1f4dffa6116729c90512d9fbd5b062"} Oct 30 18:48:04.906159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:04.906091 4442 scope.go:122] "RemoveContainer" containerID="e484c2c9496828733611646c0f76b899e43ef236e49de7ea612e35737bb57e2d" Oct 30 18:48:04.939540 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:04.939499 4442 scope.go:122] "RemoveContainer" containerID="e484c2c9496828733611646c0f76b899e43ef236e49de7ea612e35737bb57e2d" Oct 30 18:48:04.939967 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:48:04.939937 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e484c2c9496828733611646c0f76b899e43ef236e49de7ea612e35737bb57e2d\": container with ID starting with e484c2c9496828733611646c0f76b899e43ef236e49de7ea612e35737bb57e2d not found: ID does not exist" containerID="e484c2c9496828733611646c0f76b899e43ef236e49de7ea612e35737bb57e2d" Oct 30 18:48:04.940091 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:04.939980 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e484c2c9496828733611646c0f76b899e43ef236e49de7ea612e35737bb57e2d"} err="failed to get container status \"e484c2c9496828733611646c0f76b899e43ef236e49de7ea612e35737bb57e2d\": rpc error: code = NotFound desc = could not find container \"e484c2c9496828733611646c0f76b899e43ef236e49de7ea612e35737bb57e2d\": container with ID starting with e484c2c9496828733611646c0f76b899e43ef236e49de7ea612e35737bb57e2d not found: ID does not exist" Oct 30 18:48:04.965846 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:04.965797 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4mfclf"] Oct 30 18:48:04.974368 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:04.974275 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4mfclf"] Oct 30 18:48:05.001240 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:05.001209 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhfdm\" (UniqueName: \"kubernetes.io/projected/8046286d-a28b-41d2-916b-027baf316251-kube-api-access-rhfdm\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:48:05.001240 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:05.001236 4442 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8046286d-a28b-41d2-916b-027baf316251-provider-credentials\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:48:05.074273 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:05.074230 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8046286d-a28b-41d2-916b-027baf316251" path="/var/lib/kubelet/pods/8046286d-a28b-41d2-916b-027baf316251/volumes" Oct 30 18:48:17.562974 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:48:17.562938 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:48:23.758776 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:23.758705 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52rn6_516c6b6c-fc40-49cf-93b5-1c4471b02cf0/ovn-acl-logging/0.log" Oct 30 18:48:23.759316 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:23.758993 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52rn6_516c6b6c-fc40-49cf-93b5-1c4471b02cf0/ovn-acl-logging/0.log" Oct 30 18:48:25.703440 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:25.703403 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9"] Oct 30 18:48:25.704570 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:25.704546 4442 cpu_manager.go:414] "containerMap: removing container" podUID="8046286d-a28b-41d2-916b-027baf316251" containerName="payload-processing" Oct 30 18:48:25.704570 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:25.704569 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8046286d-a28b-41d2-916b-027baf316251" containerName="payload-processing" Oct 30 18:48:25.704822 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:25.704801 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="8046286d-a28b-41d2-916b-027baf316251" containerName="payload-processing" Oct 30 18:48:25.708603 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:25.708580 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" Oct 30 18:48:25.738035 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:25.737983 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9"] Oct 30 18:48:25.775266 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:25.775226 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/418064ff-dabc-4b46-b2cf-133f7f20e9d7-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-2pbp9\" (UID: \"418064ff-dabc-4b46-b2cf-133f7f20e9d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" Oct 30 18:48:25.775492 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:25.775277 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/418064ff-dabc-4b46-b2cf-133f7f20e9d7-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-2pbp9\" (UID: \"418064ff-dabc-4b46-b2cf-133f7f20e9d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" Oct 30 18:48:25.775492 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:25.775381 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/418064ff-dabc-4b46-b2cf-133f7f20e9d7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-2pbp9\" (UID: \"418064ff-dabc-4b46-b2cf-133f7f20e9d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" Oct 30 18:48:25.775646 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:25.775540 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/418064ff-dabc-4b46-b2cf-133f7f20e9d7-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-2pbp9\" (UID: \"418064ff-dabc-4b46-b2cf-133f7f20e9d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" Oct 30 18:48:25.775646 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:25.775618 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/418064ff-dabc-4b46-b2cf-133f7f20e9d7-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-2pbp9\" (UID: \"418064ff-dabc-4b46-b2cf-133f7f20e9d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" Oct 30 18:48:25.775791 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:25.775771 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/418064ff-dabc-4b46-b2cf-133f7f20e9d7-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-2pbp9\" (UID: \"418064ff-dabc-4b46-b2cf-133f7f20e9d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" Oct 30 18:48:25.775857 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:25.775826 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/418064ff-dabc-4b46-b2cf-133f7f20e9d7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-2pbp9\" (UID: \"418064ff-dabc-4b46-b2cf-133f7f20e9d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" Oct 30 18:48:25.775910 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:25.775856 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/418064ff-dabc-4b46-b2cf-133f7f20e9d7-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-2pbp9\" (UID: \"418064ff-dabc-4b46-b2cf-133f7f20e9d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" Oct 30 18:48:25.775967 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:25.775923 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k4sb\" (UniqueName: \"kubernetes.io/projected/418064ff-dabc-4b46-b2cf-133f7f20e9d7-kube-api-access-9k4sb\") pod \"maas-default-gateway-openshift-default-76597d6855-2pbp9\" (UID: \"418064ff-dabc-4b46-b2cf-133f7f20e9d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" Oct 30 18:48:25.877773 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:25.877741 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/418064ff-dabc-4b46-b2cf-133f7f20e9d7-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-2pbp9\" (UID: \"418064ff-dabc-4b46-b2cf-133f7f20e9d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" Oct 30 18:48:25.878000 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:25.877788 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/418064ff-dabc-4b46-b2cf-133f7f20e9d7-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-2pbp9\" (UID: \"418064ff-dabc-4b46-b2cf-133f7f20e9d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" Oct 30 18:48:25.878000 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:25.877828 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/418064ff-dabc-4b46-b2cf-133f7f20e9d7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-2pbp9\" (UID: \"418064ff-dabc-4b46-b2cf-133f7f20e9d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" Oct 30 18:48:25.878183 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:25.877987 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/418064ff-dabc-4b46-b2cf-133f7f20e9d7-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-2pbp9\" (UID: \"418064ff-dabc-4b46-b2cf-133f7f20e9d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" Oct 30 18:48:25.878183 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:25.878053 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/418064ff-dabc-4b46-b2cf-133f7f20e9d7-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-2pbp9\" (UID: \"418064ff-dabc-4b46-b2cf-133f7f20e9d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" Oct 30 18:48:25.878326 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:25.878215 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/418064ff-dabc-4b46-b2cf-133f7f20e9d7-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-2pbp9\" (UID: \"418064ff-dabc-4b46-b2cf-133f7f20e9d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" Oct 30 18:48:25.878326 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:25.878266 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/418064ff-dabc-4b46-b2cf-133f7f20e9d7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-2pbp9\" (UID: \"418064ff-dabc-4b46-b2cf-133f7f20e9d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" Oct 30 18:48:25.878326 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:25.878295 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/418064ff-dabc-4b46-b2cf-133f7f20e9d7-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-2pbp9\" (UID: \"418064ff-dabc-4b46-b2cf-133f7f20e9d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" Oct 30 18:48:25.878326 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:25.878311 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/418064ff-dabc-4b46-b2cf-133f7f20e9d7-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-2pbp9\" (UID: \"418064ff-dabc-4b46-b2cf-133f7f20e9d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" Oct 30 18:48:25.878492 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:25.878388 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k4sb\" (UniqueName: \"kubernetes.io/projected/418064ff-dabc-4b46-b2cf-133f7f20e9d7-kube-api-access-9k4sb\") pod \"maas-default-gateway-openshift-default-76597d6855-2pbp9\" (UID: \"418064ff-dabc-4b46-b2cf-133f7f20e9d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" Oct 30 18:48:25.878974 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:25.878947 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/418064ff-dabc-4b46-b2cf-133f7f20e9d7-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-2pbp9\" (UID: \"418064ff-dabc-4b46-b2cf-133f7f20e9d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" Oct 30 18:48:25.878974 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:25.878970 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/418064ff-dabc-4b46-b2cf-133f7f20e9d7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-2pbp9\" (UID: \"418064ff-dabc-4b46-b2cf-133f7f20e9d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" Oct 30 18:48:25.879830 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:25.879782 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/418064ff-dabc-4b46-b2cf-133f7f20e9d7-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-2pbp9\" (UID: \"418064ff-dabc-4b46-b2cf-133f7f20e9d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" Oct 30 18:48:25.880080 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:25.880058 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/418064ff-dabc-4b46-b2cf-133f7f20e9d7-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-2pbp9\" (UID: \"418064ff-dabc-4b46-b2cf-133f7f20e9d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" Oct 30 18:48:25.882150 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:25.882111 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/418064ff-dabc-4b46-b2cf-133f7f20e9d7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-2pbp9\" (UID: \"418064ff-dabc-4b46-b2cf-133f7f20e9d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" Oct 30 18:48:25.882637 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:25.882611 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/418064ff-dabc-4b46-b2cf-133f7f20e9d7-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-2pbp9\" (UID: \"418064ff-dabc-4b46-b2cf-133f7f20e9d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" Oct 30 18:48:25.898444 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:25.898422 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/418064ff-dabc-4b46-b2cf-133f7f20e9d7-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-2pbp9\" (UID: \"418064ff-dabc-4b46-b2cf-133f7f20e9d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" Oct 30 18:48:25.898580 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:25.898561 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k4sb\" (UniqueName: \"kubernetes.io/projected/418064ff-dabc-4b46-b2cf-133f7f20e9d7-kube-api-access-9k4sb\") pod \"maas-default-gateway-openshift-default-76597d6855-2pbp9\" (UID: \"418064ff-dabc-4b46-b2cf-133f7f20e9d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" Oct 30 18:48:26.041796 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:26.041700 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" Oct 30 18:48:26.364072 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:48:26.364043 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod418064ff_dabc_4b46_b2cf_133f7f20e9d7.slice/crio-6e85e0085165d640b722690d32767a610e4371852a5ee415c3127d3cf7d3bb0d WatchSource:0}: Error finding container 6e85e0085165d640b722690d32767a610e4371852a5ee415c3127d3cf7d3bb0d: Status 404 returned error can't find the container with id 6e85e0085165d640b722690d32767a610e4371852a5ee415c3127d3cf7d3bb0d Oct 30 18:48:26.365805 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:26.365775 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9"] Oct 30 18:48:26.367789 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:26.367753 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:48:26.367871 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:26.367836 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:48:26.367915 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:26.367885 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:48:27.186436 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:27.186396 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" event={"ID":"418064ff-dabc-4b46-b2cf-133f7f20e9d7","Type":"ContainerStarted","Data":"befaf9497ac79c29b6005be15ed5ac671308122d37941c11afcee31e71d54214"} Oct 30 18:48:27.186846 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:27.186458 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" event={"ID":"418064ff-dabc-4b46-b2cf-133f7f20e9d7","Type":"ContainerStarted","Data":"6e85e0085165d640b722690d32767a610e4371852a5ee415c3127d3cf7d3bb0d"} Oct 30 18:48:27.232013 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:27.231947 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" podStartSLOduration=2.231927055 podStartE2EDuration="2.231927055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:27.225905116 +0000 UTC m=+3004.315967455" watchObservedRunningTime="2026-10-30 18:48:27.231927055 +0000 UTC m=+3004.321989396" Oct 30 18:48:28.047159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:28.046512 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" Oct 30 18:48:28.053309 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:28.053268 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-2pbp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Oct 30 18:48:28.053619 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:28.053590 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" podUID="418064ff-dabc-4b46-b2cf-133f7f20e9d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Oct 30 18:48:29.048157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:29.044798 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-2pbp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Oct 30 18:48:29.048157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:29.044867 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" podUID="418064ff-dabc-4b46-b2cf-133f7f20e9d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Oct 30 18:48:29.089314 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:29.087834 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr"] Oct 30 18:48:29.094034 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:29.094006 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" Oct 30 18:48:29.120102 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:29.119743 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr"] Oct 30 18:48:29.174947 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:29.174895 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-0cc6b1-openshift-default-dockercfg-mkbph\"" Oct 30 18:48:29.183619 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:29.183112 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba1a472-f362-4900-a939-652f2354d12a-workload-socket\") pod \"e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr\" (UID: \"fba1a472-f362-4900-a939-652f2354d12a\") " pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" Oct 30 18:48:29.184112 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:29.184047 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsn8g\" (UniqueName: \"kubernetes.io/projected/fba1a472-f362-4900-a939-652f2354d12a-kube-api-access-gsn8g\") pod \"e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr\" (UID: \"fba1a472-f362-4900-a939-652f2354d12a\") " pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" Oct 30 18:48:29.184611 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:29.184593 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fba1a472-f362-4900-a939-652f2354d12a-istiod-ca-cert\") pod \"e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr\" (UID: \"fba1a472-f362-4900-a939-652f2354d12a\") " pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" Oct 30 18:48:29.184960 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:29.184932 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fba1a472-f362-4900-a939-652f2354d12a-istio-podinfo\") pod \"e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr\" (UID: \"fba1a472-f362-4900-a939-652f2354d12a\") " pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" Oct 30 18:48:29.185120 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:29.185107 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba1a472-f362-4900-a939-652f2354d12a-credential-socket\") pod \"e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr\" (UID: \"fba1a472-f362-4900-a939-652f2354d12a\") " pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" Oct 30 18:48:29.185311 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:29.185282 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fba1a472-f362-4900-a939-652f2354d12a-istio-data\") pod \"e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr\" (UID: \"fba1a472-f362-4900-a939-652f2354d12a\") " pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" Oct 30 18:48:29.185579 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:29.185561 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fba1a472-f362-4900-a939-652f2354d12a-istio-envoy\") pod \"e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr\" (UID: \"fba1a472-f362-4900-a939-652f2354d12a\") " pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" Oct 30 18:48:29.185801 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:29.185787 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fba1a472-f362-4900-a939-652f2354d12a-istio-token\") pod \"e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr\" (UID: \"fba1a472-f362-4900-a939-652f2354d12a\") " pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" Oct 30 18:48:29.186016 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:29.185999 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fba1a472-f362-4900-a939-652f2354d12a-workload-certs\") pod \"e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr\" (UID: \"fba1a472-f362-4900-a939-652f2354d12a\") " pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" Oct 30 18:48:29.291158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:29.290175 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba1a472-f362-4900-a939-652f2354d12a-workload-socket\") pod \"e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr\" (UID: \"fba1a472-f362-4900-a939-652f2354d12a\") " pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" Oct 30 18:48:29.291158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:29.290234 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsn8g\" (UniqueName: \"kubernetes.io/projected/fba1a472-f362-4900-a939-652f2354d12a-kube-api-access-gsn8g\") pod \"e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr\" (UID: \"fba1a472-f362-4900-a939-652f2354d12a\") " pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" Oct 30 18:48:29.291158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:29.290304 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fba1a472-f362-4900-a939-652f2354d12a-istiod-ca-cert\") pod \"e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr\" (UID: \"fba1a472-f362-4900-a939-652f2354d12a\") " pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" Oct 30 18:48:29.291158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:29.290414 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fba1a472-f362-4900-a939-652f2354d12a-istio-podinfo\") pod \"e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr\" (UID: \"fba1a472-f362-4900-a939-652f2354d12a\") " pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" Oct 30 18:48:29.291158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:29.290456 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba1a472-f362-4900-a939-652f2354d12a-credential-socket\") pod \"e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr\" (UID: \"fba1a472-f362-4900-a939-652f2354d12a\") " pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" Oct 30 18:48:29.291158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:29.290521 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fba1a472-f362-4900-a939-652f2354d12a-istio-data\") pod \"e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr\" (UID: \"fba1a472-f362-4900-a939-652f2354d12a\") " pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" Oct 30 18:48:29.291158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:29.290653 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fba1a472-f362-4900-a939-652f2354d12a-istio-envoy\") pod \"e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr\" (UID: \"fba1a472-f362-4900-a939-652f2354d12a\") " pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" Oct 30 18:48:29.291158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:29.290749 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fba1a472-f362-4900-a939-652f2354d12a-istio-token\") pod \"e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr\" (UID: \"fba1a472-f362-4900-a939-652f2354d12a\") " pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" Oct 30 18:48:29.291158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:29.290789 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fba1a472-f362-4900-a939-652f2354d12a-workload-certs\") pod \"e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr\" (UID: \"fba1a472-f362-4900-a939-652f2354d12a\") " pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" Oct 30 18:48:29.345863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:29.321091 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fba1a472-f362-4900-a939-652f2354d12a-istio-data\") pod \"e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr\" (UID: \"fba1a472-f362-4900-a939-652f2354d12a\") " pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" Oct 30 18:48:29.345863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:29.321403 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba1a472-f362-4900-a939-652f2354d12a-credential-socket\") pod \"e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr\" (UID: \"fba1a472-f362-4900-a939-652f2354d12a\") " pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" Oct 30 18:48:29.345863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:29.322004 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba1a472-f362-4900-a939-652f2354d12a-workload-socket\") pod \"e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr\" (UID: \"fba1a472-f362-4900-a939-652f2354d12a\") " pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" Oct 30 18:48:29.345863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:29.323181 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fba1a472-f362-4900-a939-652f2354d12a-istiod-ca-cert\") pod \"e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr\" (UID: \"fba1a472-f362-4900-a939-652f2354d12a\") " pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" Oct 30 18:48:29.345863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:29.328320 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fba1a472-f362-4900-a939-652f2354d12a-workload-certs\") pod \"e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr\" (UID: \"fba1a472-f362-4900-a939-652f2354d12a\") " pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" Oct 30 18:48:29.376522 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:29.375710 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fba1a472-f362-4900-a939-652f2354d12a-istio-podinfo\") pod \"e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr\" (UID: \"fba1a472-f362-4900-a939-652f2354d12a\") " pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" Oct 30 18:48:29.417175 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:29.416064 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fba1a472-f362-4900-a939-652f2354d12a-istio-envoy\") pod \"e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr\" (UID: \"fba1a472-f362-4900-a939-652f2354d12a\") " pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" Oct 30 18:48:29.421164 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:29.420653 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fba1a472-f362-4900-a939-652f2354d12a-istio-token\") pod \"e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr\" (UID: \"fba1a472-f362-4900-a939-652f2354d12a\") " pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" Oct 30 18:48:29.457321 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:29.456833 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsn8g\" (UniqueName: \"kubernetes.io/projected/fba1a472-f362-4900-a939-652f2354d12a-kube-api-access-gsn8g\") pod \"e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr\" (UID: \"fba1a472-f362-4900-a939-652f2354d12a\") " pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" Oct 30 18:48:29.712064 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:29.712030 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" Oct 30 18:48:30.060195 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:30.060104 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-2pbp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Oct 30 18:48:30.060195 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:30.060177 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" podUID="418064ff-dabc-4b46-b2cf-133f7f20e9d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Oct 30 18:48:30.097781 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:30.097743 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr"] Oct 30 18:48:30.100920 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:48:30.100888 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfba1a472_f362_4900_a939_652f2354d12a.slice/crio-960e406ce70843202f5ac909ec270faba3d1b27cbd05be023c283639a67d76d5 WatchSource:0}: Error finding container 960e406ce70843202f5ac909ec270faba3d1b27cbd05be023c283639a67d76d5: Status 404 returned error can't find the container with id 960e406ce70843202f5ac909ec270faba3d1b27cbd05be023c283639a67d76d5 Oct 30 18:48:30.104988 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:30.104946 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:48:30.105094 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:30.105041 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:48:30.105094 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:30.105083 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:48:30.479018 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:30.478905 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" event={"ID":"fba1a472-f362-4900-a939-652f2354d12a","Type":"ContainerStarted","Data":"d1f0b4b942e6e0660a461c0a45ed03435ac5e54ff0bf6f952963231d7b0f5f17"} Oct 30 18:48:30.479018 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:30.478970 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" event={"ID":"fba1a472-f362-4900-a939-652f2354d12a","Type":"ContainerStarted","Data":"960e406ce70843202f5ac909ec270faba3d1b27cbd05be023c283639a67d76d5"} Oct 30 18:48:30.534161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:30.524064 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" podStartSLOduration=1.524042354 podStartE2EDuration="1.524042354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:30.5189659 +0000 UTC m=+3007.609028238" watchObservedRunningTime="2026-10-30 18:48:30.524042354 +0000 UTC m=+3007.614104694" Oct 30 18:48:30.712321 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:30.712235 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" Oct 30 18:48:31.050820 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:31.050780 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-2pbp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Oct 30 18:48:31.051122 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:31.051090 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" podUID="418064ff-dabc-4b46-b2cf-133f7f20e9d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Oct 30 18:48:31.720145 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:31.720102 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" Oct 30 18:48:32.048217 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:32.047248 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-2pbp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Oct 30 18:48:32.048217 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:32.047322 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" podUID="418064ff-dabc-4b46-b2cf-133f7f20e9d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Oct 30 18:48:32.495593 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:32.495562 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" Oct 30 18:48:32.496807 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:32.496785 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" Oct 30 18:48:33.042792 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:33.042752 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-2pbp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Oct 30 18:48:33.043340 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:33.042819 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" podUID="418064ff-dabc-4b46-b2cf-133f7f20e9d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Oct 30 18:48:34.042581 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:34.042549 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-2pbp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Oct 30 18:48:34.042804 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:34.042608 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" podUID="418064ff-dabc-4b46-b2cf-133f7f20e9d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Oct 30 18:48:35.565887 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:35.563375 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" Oct 30 18:48:35.565887 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:35.565105 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" Oct 30 18:48:35.568765 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:35.568725 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" Oct 30 18:48:47.231790 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:48:47.231746 4442 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-176.ec2.internal" podUID="9e81bd78300430e51b01d30caa1123e7" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:48:47.564674 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:48:47.564584 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:49:17.562847 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:49:17.562812 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:49:47.570317 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:49:47.570271 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:49:48.733264 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:49:48.733228 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1aa59a-765b7b94b7-s9pmv"] Oct 30 18:49:48.738066 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:49:48.738046 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1aa59a-765b7b94b7-s9pmv" Oct 30 18:49:48.756524 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:49:48.756494 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1aa59a-765b7b94b7-s9pmv"] Oct 30 18:49:48.762178 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:49:48.762154 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1aa59a\"" Oct 30 18:49:48.774315 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:49:48.774281 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1aa59a-dockercfg-gbmbw\"" Oct 30 18:49:48.814167 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:49:48.807472 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45hqs\" (UniqueName: \"kubernetes.io/projected/8b79c781-9ddc-473d-8285-120dffb8e016-kube-api-access-45hqs\") pod \"payload-pre-processing-e2e-inf-a-1aa59a-765b7b94b7-s9pmv\" (UID: \"8b79c781-9ddc-473d-8285-120dffb8e016\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1aa59a-765b7b94b7-s9pmv" Oct 30 18:49:49.076222 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:49:49.063727 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b79c781-9ddc-473d-8285-120dffb8e016-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1aa59a-765b7b94b7-s9pmv\" (UID: \"8b79c781-9ddc-473d-8285-120dffb8e016\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1aa59a-765b7b94b7-s9pmv" Oct 30 18:49:49.165246 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:49:49.165208 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45hqs\" (UniqueName: \"kubernetes.io/projected/8b79c781-9ddc-473d-8285-120dffb8e016-kube-api-access-45hqs\") pod \"payload-pre-processing-e2e-inf-a-1aa59a-765b7b94b7-s9pmv\" (UID: \"8b79c781-9ddc-473d-8285-120dffb8e016\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1aa59a-765b7b94b7-s9pmv" Oct 30 18:49:49.165435 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:49:49.165363 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b79c781-9ddc-473d-8285-120dffb8e016-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1aa59a-765b7b94b7-s9pmv\" (UID: \"8b79c781-9ddc-473d-8285-120dffb8e016\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1aa59a-765b7b94b7-s9pmv" Oct 30 18:49:49.166441 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:49:49.166416 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b79c781-9ddc-473d-8285-120dffb8e016-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1aa59a-765b7b94b7-s9pmv\" (UID: \"8b79c781-9ddc-473d-8285-120dffb8e016\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1aa59a-765b7b94b7-s9pmv" Oct 30 18:49:49.184934 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:49:49.184906 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45hqs\" (UniqueName: \"kubernetes.io/projected/8b79c781-9ddc-473d-8285-120dffb8e016-kube-api-access-45hqs\") pod \"payload-pre-processing-e2e-inf-a-1aa59a-765b7b94b7-s9pmv\" (UID: \"8b79c781-9ddc-473d-8285-120dffb8e016\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1aa59a-765b7b94b7-s9pmv" Oct 30 18:49:49.358505 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:49:49.358418 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1aa59a-765b7b94b7-s9pmv" Oct 30 18:49:49.668008 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:49:49.667980 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b79c781_9ddc_473d_8285_120dffb8e016.slice/crio-1589357258b607f0d59a32572cc2b99f712981f29ea4d7a948d4edee3611025c WatchSource:0}: Error finding container 1589357258b607f0d59a32572cc2b99f712981f29ea4d7a948d4edee3611025c: Status 404 returned error can't find the container with id 1589357258b607f0d59a32572cc2b99f712981f29ea4d7a948d4edee3611025c Oct 30 18:49:49.668860 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:49:49.668831 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1aa59a-765b7b94b7-s9pmv"] Oct 30 18:49:50.669143 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:49:50.669086 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1aa59a-765b7b94b7-s9pmv" event={"ID":"8b79c781-9ddc-473d-8285-120dffb8e016","Type":"ContainerStarted","Data":"9195959c568f767238f3d4791ee1266e18d693a2225e13cfc308e4e4a0e84c07"} Oct 30 18:49:50.669601 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:49:50.669185 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1aa59a-765b7b94b7-s9pmv" event={"ID":"8b79c781-9ddc-473d-8285-120dffb8e016","Type":"ContainerStarted","Data":"1589357258b607f0d59a32572cc2b99f712981f29ea4d7a948d4edee3611025c"} Oct 30 18:49:50.669601 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:49:50.669263 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1aa59a-765b7b94b7-s9pmv" Oct 30 18:49:50.709232 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:49:50.709177 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1aa59a-765b7b94b7-s9pmv" podStartSLOduration=2.709163495 podStartE2EDuration="2.709163495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:50.707493444 +0000 UTC m=+3087.797555782" watchObservedRunningTime="2026-10-30 18:49:50.709163495 +0000 UTC m=+3087.799225828" Oct 30 18:50:01.682575 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:01.682543 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1aa59a-765b7b94b7-s9pmv" Oct 30 18:50:17.591825 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:50:17.591735 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:50:27.253519 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:27.253480 4442 generic.go:356] "Generic (PLEG): container finished" podID="fba1a472-f362-4900-a939-652f2354d12a" containerID="d1f0b4b942e6e0660a461c0a45ed03435ac5e54ff0bf6f952963231d7b0f5f17" exitCode=0 Oct 30 18:50:27.254107 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:27.253554 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" event={"ID":"fba1a472-f362-4900-a939-652f2354d12a","Type":"ContainerDied","Data":"d1f0b4b942e6e0660a461c0a45ed03435ac5e54ff0bf6f952963231d7b0f5f17"} Oct 30 18:50:27.254107 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:27.254066 4442 scope.go:122] "RemoveContainer" containerID="d1f0b4b942e6e0660a461c0a45ed03435ac5e54ff0bf6f952963231d7b0f5f17" Oct 30 18:50:27.256024 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:27.255991 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:50:27.256176 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:27.256068 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:50:27.256176 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:27.256098 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:50:28.265868 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:28.265829 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" event={"ID":"fba1a472-f362-4900-a939-652f2354d12a","Type":"ContainerStarted","Data":"2d185dfc549dc350b24d608c838089c6cd0a11b95bbefb0adf0b101f7172c56d"} Oct 30 18:50:28.712633 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:28.712598 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" Oct 30 18:50:28.714453 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:28.714426 4442 patch_prober.go:29] interesting pod/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Oct 30 18:50:28.714607 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:28.714481 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" podUID="fba1a472-f362-4900-a939-652f2354d12a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Oct 30 18:50:28.769848 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:28.769631 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w"] Oct 30 18:50:28.775034 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:28.774482 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" Oct 30 18:50:28.806819 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:28.806771 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w"] Oct 30 18:50:28.842731 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:28.842681 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c52ec7-openshift-default-dockercfg-4xgfv\"" Oct 30 18:50:28.870404 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:28.870369 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-istio-data\") pod \"e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w\" (UID: \"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2\") " pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" Oct 30 18:50:28.871068 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:28.871036 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v5ht\" (UniqueName: \"kubernetes.io/projected/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-kube-api-access-5v5ht\") pod \"e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w\" (UID: \"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2\") " pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" Oct 30 18:50:28.871338 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:28.871317 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-workload-certs\") pod \"e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w\" (UID: \"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2\") " pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" Oct 30 18:50:28.871490 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:28.871475 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-istio-envoy\") pod \"e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w\" (UID: \"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2\") " pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" Oct 30 18:50:28.871603 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:28.871590 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-istio-token\") pod \"e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w\" (UID: \"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2\") " pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" Oct 30 18:50:28.871873 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:28.871846 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-credential-socket\") pod \"e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w\" (UID: \"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2\") " pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" Oct 30 18:50:28.871973 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:28.871954 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-istio-podinfo\") pod \"e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w\" (UID: \"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2\") " pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" Oct 30 18:50:28.872055 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:28.872039 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-istiod-ca-cert\") pod \"e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w\" (UID: \"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2\") " pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" Oct 30 18:50:28.872192 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:28.872173 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-workload-socket\") pod \"e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w\" (UID: \"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2\") " pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" Oct 30 18:50:28.977339 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:28.977247 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-credential-socket\") pod \"e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w\" (UID: \"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2\") " pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" Oct 30 18:50:29.009801 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:29.009773 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-istio-podinfo\") pod \"e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w\" (UID: \"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2\") " pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" Oct 30 18:50:29.010804 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:29.010776 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-istiod-ca-cert\") pod \"e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w\" (UID: \"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2\") " pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" Oct 30 18:50:29.028423 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:29.028397 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-workload-socket\") pod \"e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w\" (UID: \"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2\") " pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" Oct 30 18:50:29.029062 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:29.029040 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-istio-data\") pod \"e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w\" (UID: \"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2\") " pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" Oct 30 18:50:29.029681 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:29.029661 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v5ht\" (UniqueName: \"kubernetes.io/projected/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-kube-api-access-5v5ht\") pod \"e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w\" (UID: \"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2\") " pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" Oct 30 18:50:29.030174 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:29.020761 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-istio-podinfo\") pod \"e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w\" (UID: \"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2\") " pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" Oct 30 18:50:29.030826 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:29.028891 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-workload-socket\") pod \"e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w\" (UID: \"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2\") " pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" Oct 30 18:50:29.052528 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:28.977940 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-credential-socket\") pod \"e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w\" (UID: \"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2\") " pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" Oct 30 18:50:29.052955 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:29.029560 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-istio-data\") pod \"e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w\" (UID: \"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2\") " pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" Oct 30 18:50:29.053178 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:29.028207 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-istiod-ca-cert\") pod \"e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w\" (UID: \"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2\") " pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" Oct 30 18:50:29.053309 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:29.030362 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-workload-certs\") pod \"e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w\" (UID: \"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2\") " pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" Oct 30 18:50:29.053482 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:29.053462 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-istio-envoy\") pod \"e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w\" (UID: \"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2\") " pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" Oct 30 18:50:29.053969 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:29.053946 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-istio-token\") pod \"e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w\" (UID: \"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2\") " pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" Oct 30 18:50:29.054587 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:29.030714 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-workload-certs\") pod \"e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w\" (UID: \"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2\") " pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" Oct 30 18:50:29.057574 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:29.057553 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-istio-envoy\") pod \"e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w\" (UID: \"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2\") " pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" Oct 30 18:50:29.076632 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:29.075929 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v5ht\" (UniqueName: \"kubernetes.io/projected/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-kube-api-access-5v5ht\") pod \"e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w\" (UID: \"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2\") " pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" Oct 30 18:50:29.098326 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:29.097787 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-istio-token\") pod \"e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w\" (UID: \"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2\") " pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" Oct 30 18:50:29.098326 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:29.098096 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" Oct 30 18:50:29.712483 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:29.712449 4442 patch_prober.go:29] interesting pod/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Oct 30 18:50:29.712861 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:29.712502 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" podUID="fba1a472-f362-4900-a939-652f2354d12a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Oct 30 18:50:29.761083 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:50:29.761044 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7e7ff65_c137_48d2_a189_eb7f64bb1ff2.slice/crio-58add8e7335419c60bd614a71d6c517213eb0fd129b359db412f5179984937bb WatchSource:0}: Error finding container 58add8e7335419c60bd614a71d6c517213eb0fd129b359db412f5179984937bb: Status 404 returned error can't find the container with id 58add8e7335419c60bd614a71d6c517213eb0fd129b359db412f5179984937bb Oct 30 18:50:29.765041 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:29.765019 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w"] Oct 30 18:50:29.765331 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:29.765295 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:50:29.765395 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:29.765380 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:50:29.765432 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:29.765419 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:50:30.369396 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:30.369356 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" event={"ID":"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2","Type":"ContainerStarted","Data":"9ece5b1efeaf907e8e1f56ccd2da07a49a6b8207b3c212b1043c78ca79b7558d"} Oct 30 18:50:30.369610 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:30.369425 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" event={"ID":"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2","Type":"ContainerStarted","Data":"58add8e7335419c60bd614a71d6c517213eb0fd129b359db412f5179984937bb"} Oct 30 18:50:30.444159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:30.443903 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" podStartSLOduration=2.443879304 podStartE2EDuration="2.443879304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:30.425576004 +0000 UTC m=+3127.515638345" watchObservedRunningTime="2026-10-30 18:50:30.443879304 +0000 UTC m=+3127.533941646" Oct 30 18:50:30.712948 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:30.712865 4442 patch_prober.go:29] interesting pod/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Oct 30 18:50:30.712948 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:30.712918 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" podUID="fba1a472-f362-4900-a939-652f2354d12a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Oct 30 18:50:31.101781 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:31.101700 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" Oct 30 18:50:31.108967 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:31.108936 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" Oct 30 18:50:31.380213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:31.380182 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" Oct 30 18:50:31.381262 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:31.381240 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" Oct 30 18:50:31.712508 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:31.712429 4442 patch_prober.go:29] interesting pod/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Oct 30 18:50:31.712508 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:31.712478 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" podUID="fba1a472-f362-4900-a939-652f2354d12a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Oct 30 18:50:32.496072 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:32.496043 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" Oct 30 18:50:32.720767 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:32.720536 4442 patch_prober.go:29] interesting pod/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Oct 30 18:50:32.720767 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:32.720597 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" podUID="fba1a472-f362-4900-a939-652f2354d12a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Oct 30 18:50:33.723190 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:33.717872 4442 patch_prober.go:29] interesting pod/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Oct 30 18:50:33.723190 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:33.717942 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" podUID="fba1a472-f362-4900-a939-652f2354d12a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Oct 30 18:50:34.721089 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:34.721056 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" Oct 30 18:50:34.722578 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:34.722554 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" Oct 30 18:50:36.642156 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:36.634223 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-2pbp9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:50:36.642156 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:36.634282 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" podUID="418064ff-dabc-4b46-b2cf-133f7f20e9d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:50:37.224381 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:37.224345 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-z2dn9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:50:37.224965 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:37.224935 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" podUID="e1e466ee-ec7a-4640-bbc5-2f16795107cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:50:47.231950 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:47.231890 4442 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-176.ec2.internal" podUID="9e81bd78300430e51b01d30caa1123e7" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:50:47.565120 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:50:47.565026 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:50:59.124674 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:59.124642 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm"] Oct 30 18:50:59.130123 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:59.130101 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" Oct 30 18:50:59.166757 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:59.166715 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm"] Oct 30 18:50:59.173083 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:59.173056 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-8862aa-openshift-default-dockercfg-5svft\"" Oct 30 18:50:59.259482 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:59.259441 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b15a372b-64ad-449b-8f2c-c7b49dbe4600-workload-socket\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm\" (UID: \"b15a372b-64ad-449b-8f2c-c7b49dbe4600\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" Oct 30 18:50:59.259682 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:59.259571 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b15a372b-64ad-449b-8f2c-c7b49dbe4600-workload-certs\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm\" (UID: \"b15a372b-64ad-449b-8f2c-c7b49dbe4600\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" Oct 30 18:50:59.259748 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:59.259726 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b15a372b-64ad-449b-8f2c-c7b49dbe4600-istio-data\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm\" (UID: \"b15a372b-64ad-449b-8f2c-c7b49dbe4600\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" Oct 30 18:50:59.259889 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:59.259866 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b15a372b-64ad-449b-8f2c-c7b49dbe4600-istio-token\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm\" (UID: \"b15a372b-64ad-449b-8f2c-c7b49dbe4600\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" Oct 30 18:50:59.259986 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:59.259916 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvgsk\" (UniqueName: \"kubernetes.io/projected/b15a372b-64ad-449b-8f2c-c7b49dbe4600-kube-api-access-tvgsk\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm\" (UID: \"b15a372b-64ad-449b-8f2c-c7b49dbe4600\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" Oct 30 18:50:59.260041 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:59.259993 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b15a372b-64ad-449b-8f2c-c7b49dbe4600-istiod-ca-cert\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm\" (UID: \"b15a372b-64ad-449b-8f2c-c7b49dbe4600\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" Oct 30 18:50:59.260084 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:59.260049 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b15a372b-64ad-449b-8f2c-c7b49dbe4600-istio-podinfo\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm\" (UID: \"b15a372b-64ad-449b-8f2c-c7b49dbe4600\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" Oct 30 18:50:59.260127 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:59.260091 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b15a372b-64ad-449b-8f2c-c7b49dbe4600-istio-envoy\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm\" (UID: \"b15a372b-64ad-449b-8f2c-c7b49dbe4600\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" Oct 30 18:50:59.260263 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:59.260248 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b15a372b-64ad-449b-8f2c-c7b49dbe4600-credential-socket\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm\" (UID: \"b15a372b-64ad-449b-8f2c-c7b49dbe4600\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" Oct 30 18:50:59.365238 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:59.365199 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b15a372b-64ad-449b-8f2c-c7b49dbe4600-workload-socket\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm\" (UID: \"b15a372b-64ad-449b-8f2c-c7b49dbe4600\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" Oct 30 18:50:59.366200 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:59.366172 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b15a372b-64ad-449b-8f2c-c7b49dbe4600-workload-certs\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm\" (UID: \"b15a372b-64ad-449b-8f2c-c7b49dbe4600\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" Oct 30 18:50:59.366493 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:59.366472 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b15a372b-64ad-449b-8f2c-c7b49dbe4600-istio-data\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm\" (UID: \"b15a372b-64ad-449b-8f2c-c7b49dbe4600\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" Oct 30 18:50:59.373774 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:59.373748 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b15a372b-64ad-449b-8f2c-c7b49dbe4600-istio-token\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm\" (UID: \"b15a372b-64ad-449b-8f2c-c7b49dbe4600\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" Oct 30 18:50:59.373981 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:59.373963 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvgsk\" (UniqueName: \"kubernetes.io/projected/b15a372b-64ad-449b-8f2c-c7b49dbe4600-kube-api-access-tvgsk\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm\" (UID: \"b15a372b-64ad-449b-8f2c-c7b49dbe4600\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" Oct 30 18:50:59.374213 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:59.374118 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b15a372b-64ad-449b-8f2c-c7b49dbe4600-istiod-ca-cert\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm\" (UID: \"b15a372b-64ad-449b-8f2c-c7b49dbe4600\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" Oct 30 18:50:59.374380 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:59.374362 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b15a372b-64ad-449b-8f2c-c7b49dbe4600-istio-podinfo\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm\" (UID: \"b15a372b-64ad-449b-8f2c-c7b49dbe4600\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" Oct 30 18:50:59.374525 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:59.374511 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b15a372b-64ad-449b-8f2c-c7b49dbe4600-istio-envoy\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm\" (UID: \"b15a372b-64ad-449b-8f2c-c7b49dbe4600\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" Oct 30 18:50:59.392337 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:59.374771 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b15a372b-64ad-449b-8f2c-c7b49dbe4600-credential-socket\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm\" (UID: \"b15a372b-64ad-449b-8f2c-c7b49dbe4600\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" Oct 30 18:50:59.397897 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:59.366054 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b15a372b-64ad-449b-8f2c-c7b49dbe4600-workload-socket\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm\" (UID: \"b15a372b-64ad-449b-8f2c-c7b49dbe4600\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" Oct 30 18:50:59.398076 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:59.369495 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b15a372b-64ad-449b-8f2c-c7b49dbe4600-workload-certs\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm\" (UID: \"b15a372b-64ad-449b-8f2c-c7b49dbe4600\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" Oct 30 18:50:59.398207 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:59.376400 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b15a372b-64ad-449b-8f2c-c7b49dbe4600-istiod-ca-cert\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm\" (UID: \"b15a372b-64ad-449b-8f2c-c7b49dbe4600\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" Oct 30 18:50:59.398312 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:59.369745 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b15a372b-64ad-449b-8f2c-c7b49dbe4600-istio-data\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm\" (UID: \"b15a372b-64ad-449b-8f2c-c7b49dbe4600\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" Oct 30 18:50:59.398405 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:59.377328 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b15a372b-64ad-449b-8f2c-c7b49dbe4600-credential-socket\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm\" (UID: \"b15a372b-64ad-449b-8f2c-c7b49dbe4600\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" Oct 30 18:50:59.398503 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:59.396711 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b15a372b-64ad-449b-8f2c-c7b49dbe4600-istio-podinfo\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm\" (UID: \"b15a372b-64ad-449b-8f2c-c7b49dbe4600\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" Oct 30 18:50:59.405299 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:59.405277 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b15a372b-64ad-449b-8f2c-c7b49dbe4600-istio-token\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm\" (UID: \"b15a372b-64ad-449b-8f2c-c7b49dbe4600\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" Oct 30 18:50:59.436158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:59.433828 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvgsk\" (UniqueName: \"kubernetes.io/projected/b15a372b-64ad-449b-8f2c-c7b49dbe4600-kube-api-access-tvgsk\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm\" (UID: \"b15a372b-64ad-449b-8f2c-c7b49dbe4600\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" Oct 30 18:50:59.436158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:59.435577 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b15a372b-64ad-449b-8f2c-c7b49dbe4600-istio-envoy\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm\" (UID: \"b15a372b-64ad-449b-8f2c-c7b49dbe4600\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" Oct 30 18:50:59.476488 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:50:59.476451 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" Oct 30 18:51:00.057664 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:00.057635 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm"] Oct 30 18:51:00.058398 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:51:00.058373 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb15a372b_64ad_449b_8f2c_c7b49dbe4600.slice/crio-2181dd21bc42987f7b21f70af607a02e0a1340e4362bd3360c6c8b2c9c13b256 WatchSource:0}: Error finding container 2181dd21bc42987f7b21f70af607a02e0a1340e4362bd3360c6c8b2c9c13b256: Status 404 returned error can't find the container with id 2181dd21bc42987f7b21f70af607a02e0a1340e4362bd3360c6c8b2c9c13b256 Oct 30 18:51:00.061784 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:00.061750 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:51:00.061863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:00.061838 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:51:00.061908 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:00.061867 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:51:00.827309 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:00.827263 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" event={"ID":"b15a372b-64ad-449b-8f2c-c7b49dbe4600","Type":"ContainerStarted","Data":"2f400ecb7f14bb45abbd1106184a7c4b1b381edcdc0307e7d9755af418f0e4cd"} Oct 30 18:51:00.827818 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:00.827326 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" event={"ID":"b15a372b-64ad-449b-8f2c-c7b49dbe4600","Type":"ContainerStarted","Data":"2181dd21bc42987f7b21f70af607a02e0a1340e4362bd3360c6c8b2c9c13b256"} Oct 30 18:51:00.880090 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:00.880034 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" podStartSLOduration=1.880018824 podStartE2EDuration="1.880018824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:00.879400079 +0000 UTC m=+3157.969462418" watchObservedRunningTime="2026-10-30 18:51:00.880018824 +0000 UTC m=+3157.970081162" Oct 30 18:51:01.477744 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:01.477713 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" Oct 30 18:51:01.484190 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:01.484170 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" Oct 30 18:51:01.837124 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:01.837031 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" Oct 30 18:51:01.838438 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:01.838419 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" Oct 30 18:51:17.565590 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:51:17.565548 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:51:41.071070 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:41.071040 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x"] Oct 30 18:51:41.077049 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:41.077027 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x" Oct 30 18:51:41.085450 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:41.085420 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x"] Oct 30 18:51:41.103863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:41.103741 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-861bb3\"/\"agc-cred-3357f1dd84d4\"" Oct 30 18:51:41.103863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:41.103787 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-861bb3\"/\"payload-processing-external-model-plugins-e2e-extmod-a-861bb3\"" Oct 30 18:51:41.103863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:41.103793 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-861bb3\"/\"openshift-service-ca.crt\"" Oct 30 18:51:41.103863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:41.103814 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-861bb3\"/\"agc-cred-ab842f5d10de\"" Oct 30 18:51:41.103863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:41.103816 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-861bb3\"/\"payload-processing-external-model-e2e--dacb1c2b-dockercfg-vhv72\"" Oct 30 18:51:41.103863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:41.103748 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-861bb3\"/\"kube-root-ca.crt\"" Oct 30 18:51:41.103863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:41.103847 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-861bb3\"/\"routing-overlay\"" Oct 30 18:51:41.103863 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:41.103866 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-861bb3\"/\"agc-cred-245a97933c01\"" Oct 30 18:51:41.150529 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:41.150494 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c310be7-bed9-487f-b4bc-58cca0ea2703-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x\" (UID: \"1c310be7-bed9-487f-b4bc-58cca0ea2703\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x" Oct 30 18:51:41.150739 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:41.150563 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c310be7-bed9-487f-b4bc-58cca0ea2703-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x\" (UID: \"1c310be7-bed9-487f-b4bc-58cca0ea2703\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x" Oct 30 18:51:41.150739 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:41.150704 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q2rj\" (UniqueName: \"kubernetes.io/projected/1c310be7-bed9-487f-b4bc-58cca0ea2703-kube-api-access-7q2rj\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x\" (UID: \"1c310be7-bed9-487f-b4bc-58cca0ea2703\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x" Oct 30 18:51:41.150880 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:41.150822 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c310be7-bed9-487f-b4bc-58cca0ea2703-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x\" (UID: \"1c310be7-bed9-487f-b4bc-58cca0ea2703\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x" Oct 30 18:51:41.252591 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:41.252555 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c310be7-bed9-487f-b4bc-58cca0ea2703-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x\" (UID: \"1c310be7-bed9-487f-b4bc-58cca0ea2703\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x" Oct 30 18:51:41.252782 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:41.252638 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c310be7-bed9-487f-b4bc-58cca0ea2703-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x\" (UID: \"1c310be7-bed9-487f-b4bc-58cca0ea2703\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x" Oct 30 18:51:41.252782 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:41.252679 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q2rj\" (UniqueName: \"kubernetes.io/projected/1c310be7-bed9-487f-b4bc-58cca0ea2703-kube-api-access-7q2rj\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x\" (UID: \"1c310be7-bed9-487f-b4bc-58cca0ea2703\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x" Oct 30 18:51:41.252782 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:41.252747 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c310be7-bed9-487f-b4bc-58cca0ea2703-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x\" (UID: \"1c310be7-bed9-487f-b4bc-58cca0ea2703\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x" Oct 30 18:51:41.253390 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:41.253366 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c310be7-bed9-487f-b4bc-58cca0ea2703-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x\" (UID: \"1c310be7-bed9-487f-b4bc-58cca0ea2703\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x" Oct 30 18:51:41.253542 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:41.253522 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c310be7-bed9-487f-b4bc-58cca0ea2703-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x\" (UID: \"1c310be7-bed9-487f-b4bc-58cca0ea2703\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x" Oct 30 18:51:41.255553 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:41.255530 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c310be7-bed9-487f-b4bc-58cca0ea2703-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x\" (UID: \"1c310be7-bed9-487f-b4bc-58cca0ea2703\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x" Oct 30 18:51:41.271856 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:41.271828 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q2rj\" (UniqueName: \"kubernetes.io/projected/1c310be7-bed9-487f-b4bc-58cca0ea2703-kube-api-access-7q2rj\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x\" (UID: \"1c310be7-bed9-487f-b4bc-58cca0ea2703\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x" Oct 30 18:51:41.406773 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:41.406736 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x" Oct 30 18:51:41.894455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:41.894423 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-9b6ds94sp"] Oct 30 18:51:41.902557 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:41.902531 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-9b6ds94sp" Oct 30 18:51:41.920003 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:41.919980 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-9b6ds94sp"] Oct 30 18:51:41.921940 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:41.921871 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6018b4\"/\"agc-cred-45d23ae5f8fc\"" Oct 30 18:51:41.922104 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:41.922088 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6018b4\"/\"agc-cred-49968ea063aa\"" Oct 30 18:51:41.922233 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:41.922181 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6018b4\"/\"agc-cred-8403349d486c\"" Oct 30 18:51:41.922233 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:41.922229 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6018b4\"/\"openshift-service-ca.crt\"" Oct 30 18:51:41.922308 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:41.922238 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6018b4\"/\"payload-processing-external-model-e2e--6d8a874f-dockercfg-j6h5f\"" Oct 30 18:51:41.933749 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:41.933535 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6018b4\"/\"routing-overlay\"" Oct 30 18:51:41.933749 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:41.933616 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6018b4\"/\"payload-processing-external-model-plugins-e2e-extmod-b-6018b4\"" Oct 30 18:51:41.933749 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:41.933686 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6018b4\"/\"kube-root-ca.crt\"" Oct 30 18:51:42.008652 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:42.008623 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43442120-12b1-4c2f-b993-80490b3f415f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-9b6ds94sp\" (UID: \"43442120-12b1-4c2f-b993-80490b3f415f\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-9b6ds94sp" Oct 30 18:51:42.008831 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:42.008657 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5jgk\" (UniqueName: \"kubernetes.io/projected/43442120-12b1-4c2f-b993-80490b3f415f-kube-api-access-p5jgk\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-9b6ds94sp\" (UID: \"43442120-12b1-4c2f-b993-80490b3f415f\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-9b6ds94sp" Oct 30 18:51:42.008831 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:42.008815 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/43442120-12b1-4c2f-b993-80490b3f415f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-9b6ds94sp\" (UID: \"43442120-12b1-4c2f-b993-80490b3f415f\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-9b6ds94sp" Oct 30 18:51:42.008907 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:42.008870 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/43442120-12b1-4c2f-b993-80490b3f415f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-9b6ds94sp\" (UID: \"43442120-12b1-4c2f-b993-80490b3f415f\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-9b6ds94sp" Oct 30 18:51:42.110573 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:42.110493 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43442120-12b1-4c2f-b993-80490b3f415f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-9b6ds94sp\" (UID: \"43442120-12b1-4c2f-b993-80490b3f415f\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-9b6ds94sp" Oct 30 18:51:42.110573 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:42.110530 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5jgk\" (UniqueName: \"kubernetes.io/projected/43442120-12b1-4c2f-b993-80490b3f415f-kube-api-access-p5jgk\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-9b6ds94sp\" (UID: \"43442120-12b1-4c2f-b993-80490b3f415f\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-9b6ds94sp" Oct 30 18:51:42.111082 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:42.110667 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/43442120-12b1-4c2f-b993-80490b3f415f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-9b6ds94sp\" (UID: \"43442120-12b1-4c2f-b993-80490b3f415f\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-9b6ds94sp" Oct 30 18:51:42.111082 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:42.110791 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/43442120-12b1-4c2f-b993-80490b3f415f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-9b6ds94sp\" (UID: \"43442120-12b1-4c2f-b993-80490b3f415f\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-9b6ds94sp" Oct 30 18:51:42.111589 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:42.111545 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/43442120-12b1-4c2f-b993-80490b3f415f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-9b6ds94sp\" (UID: \"43442120-12b1-4c2f-b993-80490b3f415f\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-9b6ds94sp" Oct 30 18:51:42.112201 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:42.112176 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43442120-12b1-4c2f-b993-80490b3f415f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-9b6ds94sp\" (UID: \"43442120-12b1-4c2f-b993-80490b3f415f\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-9b6ds94sp" Oct 30 18:51:42.114263 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:42.114242 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/43442120-12b1-4c2f-b993-80490b3f415f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-9b6ds94sp\" (UID: \"43442120-12b1-4c2f-b993-80490b3f415f\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-9b6ds94sp" Oct 30 18:51:42.127176 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:51:42.126743 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c310be7_bed9_487f_b4bc_58cca0ea2703.slice/crio-acac2609343f5b3d9a93805ae0c30709b7586d88c79ffdfe74bf832dbf62a98f WatchSource:0}: Error finding container acac2609343f5b3d9a93805ae0c30709b7586d88c79ffdfe74bf832dbf62a98f: Status 404 returned error can't find the container with id acac2609343f5b3d9a93805ae0c30709b7586d88c79ffdfe74bf832dbf62a98f Oct 30 18:51:42.131012 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:42.130994 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x"] Oct 30 18:51:42.139118 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:42.139096 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5jgk\" (UniqueName: \"kubernetes.io/projected/43442120-12b1-4c2f-b993-80490b3f415f-kube-api-access-p5jgk\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-9b6ds94sp\" (UID: \"43442120-12b1-4c2f-b993-80490b3f415f\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-9b6ds94sp" Oct 30 18:51:42.221474 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:42.221447 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-9b6ds94sp" Oct 30 18:51:42.537501 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:51:42.537470 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43442120_12b1_4c2f_b993_80490b3f415f.slice/crio-6f7b811eee0ee6499571cfbdb6c2b3cf302727bd0d5b353cd8dcc657615b9218 WatchSource:0}: Error finding container 6f7b811eee0ee6499571cfbdb6c2b3cf302727bd0d5b353cd8dcc657615b9218: Status 404 returned error can't find the container with id 6f7b811eee0ee6499571cfbdb6c2b3cf302727bd0d5b353cd8dcc657615b9218 Oct 30 18:51:42.540412 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:42.540382 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-9b6ds94sp"] Oct 30 18:51:42.588867 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:42.588838 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x" event={"ID":"1c310be7-bed9-487f-b4bc-58cca0ea2703","Type":"ContainerStarted","Data":"e5daae1cc0f56e4b7375d19b4347b75e399c278d365fc5f73460b4fb3cc35ac8"} Oct 30 18:51:42.589033 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:42.588886 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x" event={"ID":"1c310be7-bed9-487f-b4bc-58cca0ea2703","Type":"ContainerStarted","Data":"acac2609343f5b3d9a93805ae0c30709b7586d88c79ffdfe74bf832dbf62a98f"} Oct 30 18:51:42.589033 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:42.588929 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x" Oct 30 18:51:42.590354 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:42.590330 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-9b6ds94sp" event={"ID":"43442120-12b1-4c2f-b993-80490b3f415f","Type":"ContainerStarted","Data":"6f7b811eee0ee6499571cfbdb6c2b3cf302727bd0d5b353cd8dcc657615b9218"} Oct 30 18:51:42.632796 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:42.632625 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x" podStartSLOduration=1.632608179 podStartE2EDuration="1.632608179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:42.630237811 +0000 UTC m=+3199.720300152" watchObservedRunningTime="2026-10-30 18:51:42.632608179 +0000 UTC m=+3199.722670519" Oct 30 18:51:43.602598 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:43.602386 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-9b6ds94sp" event={"ID":"43442120-12b1-4c2f-b993-80490b3f415f","Type":"ContainerStarted","Data":"16f3abd2bb8113270cf631d440d45d1a61e0a7adf91fa4be72ac99d70e0f6b4e"} Oct 30 18:51:43.646087 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:43.646017 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-9b6ds94sp" podStartSLOduration=2.645998801 podStartE2EDuration="2.645998801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:43.641484351 +0000 UTC m=+3200.731546768" watchObservedRunningTime="2026-10-30 18:51:43.645998801 +0000 UTC m=+3200.736061141" Oct 30 18:51:44.610872 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:44.610838 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-9b6ds94sp" Oct 30 18:51:45.576532 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:45.576496 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-551d0b-6686dd876d-qvsgq"] Oct 30 18:51:45.582695 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:45.582671 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-551d0b-6686dd876d-qvsgq" Oct 30 18:51:45.603873 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:45.603837 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-551d0b-6686dd876d-qvsgq"] Oct 30 18:51:45.605680 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:45.605655 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-551d0b-dockercfg-zfkb9\"" Oct 30 18:51:45.605826 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:45.605752 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-551d0b\"" Oct 30 18:51:45.688161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:45.681510 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knd28\" (UniqueName: \"kubernetes.io/projected/dd895ac2-c8b9-47df-93cd-0da009ab4c1e-kube-api-access-knd28\") pod \"payload-processing-e2e-inf-b-551d0b-6686dd876d-qvsgq\" (UID: \"dd895ac2-c8b9-47df-93cd-0da009ab4c1e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-551d0b-6686dd876d-qvsgq" Oct 30 18:51:45.688161 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:45.681669 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd895ac2-c8b9-47df-93cd-0da009ab4c1e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-551d0b-6686dd876d-qvsgq\" (UID: \"dd895ac2-c8b9-47df-93cd-0da009ab4c1e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-551d0b-6686dd876d-qvsgq" Oct 30 18:51:46.011228 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:46.009474 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd895ac2-c8b9-47df-93cd-0da009ab4c1e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-551d0b-6686dd876d-qvsgq\" (UID: \"dd895ac2-c8b9-47df-93cd-0da009ab4c1e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-551d0b-6686dd876d-qvsgq" Oct 30 18:51:46.011228 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:46.010234 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knd28\" (UniqueName: \"kubernetes.io/projected/dd895ac2-c8b9-47df-93cd-0da009ab4c1e-kube-api-access-knd28\") pod \"payload-processing-e2e-inf-b-551d0b-6686dd876d-qvsgq\" (UID: \"dd895ac2-c8b9-47df-93cd-0da009ab4c1e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-551d0b-6686dd876d-qvsgq" Oct 30 18:51:46.015355 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:46.015298 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd895ac2-c8b9-47df-93cd-0da009ab4c1e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-551d0b-6686dd876d-qvsgq\" (UID: \"dd895ac2-c8b9-47df-93cd-0da009ab4c1e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-551d0b-6686dd876d-qvsgq" Oct 30 18:51:46.084508 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:46.084473 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knd28\" (UniqueName: \"kubernetes.io/projected/dd895ac2-c8b9-47df-93cd-0da009ab4c1e-kube-api-access-knd28\") pod \"payload-processing-e2e-inf-b-551d0b-6686dd876d-qvsgq\" (UID: \"dd895ac2-c8b9-47df-93cd-0da009ab4c1e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-551d0b-6686dd876d-qvsgq" Oct 30 18:51:46.204599 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:46.204568 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-551d0b-6686dd876d-qvsgq" Oct 30 18:51:46.513143 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:51:46.513092 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd895ac2_c8b9_47df_93cd_0da009ab4c1e.slice/crio-f8e2260a4bbbba6017e612339d0501802b3937409527a5e28742e5df1ce6d43c WatchSource:0}: Error finding container f8e2260a4bbbba6017e612339d0501802b3937409527a5e28742e5df1ce6d43c: Status 404 returned error can't find the container with id f8e2260a4bbbba6017e612339d0501802b3937409527a5e28742e5df1ce6d43c Oct 30 18:51:46.514450 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:46.514398 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-551d0b-6686dd876d-qvsgq"] Oct 30 18:51:46.647436 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:46.647400 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-551d0b-6686dd876d-qvsgq" event={"ID":"dd895ac2-c8b9-47df-93cd-0da009ab4c1e","Type":"ContainerStarted","Data":"9d3c41c9902d6e1db4462b26df0b49780580fcd329c3006f0c5a0c642347f813"} Oct 30 18:51:46.647598 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:46.647461 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-551d0b-6686dd876d-qvsgq" event={"ID":"dd895ac2-c8b9-47df-93cd-0da009ab4c1e","Type":"ContainerStarted","Data":"f8e2260a4bbbba6017e612339d0501802b3937409527a5e28742e5df1ce6d43c"} Oct 30 18:51:46.647598 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:46.647582 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-551d0b-6686dd876d-qvsgq" Oct 30 18:51:46.686035 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:46.685979 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-551d0b-6686dd876d-qvsgq" podStartSLOduration=1.685964582 podStartE2EDuration="1.685964582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:46.684220067 +0000 UTC m=+3203.774282406" watchObservedRunningTime="2026-10-30 18:51:46.685964582 +0000 UTC m=+3203.776026975" Oct 30 18:51:47.562462 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:51:47.562427 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:51:53.606011 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:53.605983 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x" Oct 30 18:51:55.658366 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:55.658327 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-9b6ds94sp" Oct 30 18:51:57.660242 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:51:57.660210 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-551d0b-6686dd876d-qvsgq" Oct 30 18:52:06.974341 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:06.974306 4442 generic.go:356] "Generic (PLEG): container finished" podID="c7e7ff65-c137-48d2-a189-eb7f64bb1ff2" containerID="9ece5b1efeaf907e8e1f56ccd2da07a49a6b8207b3c212b1043c78ca79b7558d" exitCode=0 Oct 30 18:52:06.974341 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:06.974333 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" event={"ID":"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2","Type":"ContainerDied","Data":"9ece5b1efeaf907e8e1f56ccd2da07a49a6b8207b3c212b1043c78ca79b7558d"} Oct 30 18:52:06.974880 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:06.974811 4442 scope.go:122] "RemoveContainer" containerID="9ece5b1efeaf907e8e1f56ccd2da07a49a6b8207b3c212b1043c78ca79b7558d" Oct 30 18:52:06.977081 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:06.977044 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:52:06.977259 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:06.977159 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:52:06.977259 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:06.977214 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:52:07.985948 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:07.985910 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" event={"ID":"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2","Type":"ContainerStarted","Data":"91593f9a61c1578c25f2243d3802cdf9a82469710a7eb79d449bc50d34981656"} Oct 30 18:52:08.104925 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:08.104885 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" Oct 30 18:52:08.105223 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:08.105199 4442 patch_prober.go:29] interesting pod/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body= Oct 30 18:52:08.105323 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:08.105264 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" podUID="c7e7ff65-c137-48d2-a189-eb7f64bb1ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" Oct 30 18:52:09.115170 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:09.113683 4442 patch_prober.go:29] interesting pod/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body= Oct 30 18:52:09.115170 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:09.113745 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" podUID="c7e7ff65-c137-48d2-a189-eb7f64bb1ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" Oct 30 18:52:10.101203 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:10.101167 4442 patch_prober.go:29] interesting pod/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body= Oct 30 18:52:10.101394 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:10.101221 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" podUID="c7e7ff65-c137-48d2-a189-eb7f64bb1ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" Oct 30 18:52:11.101514 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:11.101478 4442 patch_prober.go:29] interesting pod/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body= Oct 30 18:52:11.101927 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:11.101535 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" podUID="c7e7ff65-c137-48d2-a189-eb7f64bb1ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" Oct 30 18:52:12.101947 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:12.101910 4442 patch_prober.go:29] interesting pod/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body= Oct 30 18:52:12.102533 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:12.102104 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" podUID="c7e7ff65-c137-48d2-a189-eb7f64bb1ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" Oct 30 18:52:13.102025 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:13.101980 4442 patch_prober.go:29] interesting pod/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body= Oct 30 18:52:13.102603 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:13.102052 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" podUID="c7e7ff65-c137-48d2-a189-eb7f64bb1ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" Oct 30 18:52:14.106600 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:14.106571 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" Oct 30 18:52:14.107100 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:14.107081 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" Oct 30 18:52:14.107981 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:14.107961 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" Oct 30 18:52:17.562652 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:52:17.562605 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:52:46.441538 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:46.441505 4442 generic.go:356] "Generic (PLEG): container finished" podID="b15a372b-64ad-449b-8f2c-c7b49dbe4600" containerID="2f400ecb7f14bb45abbd1106184a7c4b1b381edcdc0307e7d9755af418f0e4cd" exitCode=0 Oct 30 18:52:46.442173 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:46.441585 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" event={"ID":"b15a372b-64ad-449b-8f2c-c7b49dbe4600","Type":"ContainerDied","Data":"2f400ecb7f14bb45abbd1106184a7c4b1b381edcdc0307e7d9755af418f0e4cd"} Oct 30 18:52:46.442173 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:46.442077 4442 scope.go:122] "RemoveContainer" containerID="2f400ecb7f14bb45abbd1106184a7c4b1b381edcdc0307e7d9755af418f0e4cd" Oct 30 18:52:46.444759 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:46.444720 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:52:46.444899 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:46.444813 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:52:46.444899 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:46.444859 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:52:46.837895 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:46.837807 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" Oct 30 18:52:47.460590 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:47.460547 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" event={"ID":"b15a372b-64ad-449b-8f2c-c7b49dbe4600","Type":"ContainerStarted","Data":"20151f26f52bf7959f6411cb8f2ba1b8ad0ba6f5579d1969b47e1e4b272d859d"} Oct 30 18:52:47.477788 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:47.477755 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" Oct 30 18:52:47.479325 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:47.479304 4442 patch_prober.go:29] interesting pod/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body= Oct 30 18:52:47.479389 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:47.479351 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" podUID="b15a372b-64ad-449b-8f2c-c7b49dbe4600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" Oct 30 18:52:47.581583 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:52:47.581548 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:52:48.477624 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:48.477593 4442 patch_prober.go:29] interesting pod/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body= Oct 30 18:52:48.478121 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:48.477652 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" podUID="b15a372b-64ad-449b-8f2c-c7b49dbe4600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" Oct 30 18:52:49.477589 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:49.477554 4442 patch_prober.go:29] interesting pod/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body= Oct 30 18:52:49.477809 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:49.477606 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" podUID="b15a372b-64ad-449b-8f2c-c7b49dbe4600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" Oct 30 18:52:50.477817 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:50.477781 4442 patch_prober.go:29] interesting pod/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body= Oct 30 18:52:50.478240 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:50.477846 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" podUID="b15a372b-64ad-449b-8f2c-c7b49dbe4600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" Oct 30 18:52:51.480159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:51.479568 4442 patch_prober.go:29] interesting pod/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body= Oct 30 18:52:51.480159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:51.479648 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" podUID="b15a372b-64ad-449b-8f2c-c7b49dbe4600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" Oct 30 18:52:52.477498 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:52.477467 4442 patch_prober.go:29] interesting pod/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body= Oct 30 18:52:52.477690 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:52.477530 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" podUID="b15a372b-64ad-449b-8f2c-c7b49dbe4600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" Oct 30 18:52:53.484693 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:53.484661 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" Oct 30 18:52:53.485403 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:53.485376 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" Oct 30 18:52:53.486147 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:52:53.486110 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" Oct 30 18:53:02.993375 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:02.993328 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd"] Oct 30 18:53:03.006174 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:02.999705 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" Oct 30 18:53:03.024251 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:03.024221 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f37ca7-openshift-default-dockercfg-fgcsx\"" Oct 30 18:53:03.034174 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:03.034123 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd"] Oct 30 18:53:03.086790 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:03.086757 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d8d84cf-1334-4038-8b24-8cbec03e6882-credential-socket\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd\" (UID: \"6d8d84cf-1334-4038-8b24-8cbec03e6882\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" Oct 30 18:53:03.086790 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:03.086796 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d8d84cf-1334-4038-8b24-8cbec03e6882-istio-data\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd\" (UID: \"6d8d84cf-1334-4038-8b24-8cbec03e6882\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" Oct 30 18:53:03.087013 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:03.086989 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d8d84cf-1334-4038-8b24-8cbec03e6882-istio-token\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd\" (UID: \"6d8d84cf-1334-4038-8b24-8cbec03e6882\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" Oct 30 18:53:03.087094 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:03.087076 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d8d84cf-1334-4038-8b24-8cbec03e6882-workload-socket\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd\" (UID: \"6d8d84cf-1334-4038-8b24-8cbec03e6882\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" Oct 30 18:53:03.087230 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:03.087211 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d8d84cf-1334-4038-8b24-8cbec03e6882-istio-envoy\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd\" (UID: \"6d8d84cf-1334-4038-8b24-8cbec03e6882\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" Oct 30 18:53:03.087397 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:03.087381 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdhjc\" (UniqueName: \"kubernetes.io/projected/6d8d84cf-1334-4038-8b24-8cbec03e6882-kube-api-access-pdhjc\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd\" (UID: \"6d8d84cf-1334-4038-8b24-8cbec03e6882\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" Oct 30 18:53:03.087459 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:03.087445 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d8d84cf-1334-4038-8b24-8cbec03e6882-istio-podinfo\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd\" (UID: \"6d8d84cf-1334-4038-8b24-8cbec03e6882\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" Oct 30 18:53:03.087564 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:03.087547 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d8d84cf-1334-4038-8b24-8cbec03e6882-workload-certs\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd\" (UID: \"6d8d84cf-1334-4038-8b24-8cbec03e6882\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" Oct 30 18:53:03.087657 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:03.087590 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d8d84cf-1334-4038-8b24-8cbec03e6882-istiod-ca-cert\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd\" (UID: \"6d8d84cf-1334-4038-8b24-8cbec03e6882\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" Oct 30 18:53:03.197205 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:03.195607 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d8d84cf-1334-4038-8b24-8cbec03e6882-credential-socket\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd\" (UID: \"6d8d84cf-1334-4038-8b24-8cbec03e6882\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" Oct 30 18:53:03.197205 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:03.195765 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d8d84cf-1334-4038-8b24-8cbec03e6882-istio-data\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd\" (UID: \"6d8d84cf-1334-4038-8b24-8cbec03e6882\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" Oct 30 18:53:03.197205 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:03.195834 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d8d84cf-1334-4038-8b24-8cbec03e6882-istio-token\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd\" (UID: \"6d8d84cf-1334-4038-8b24-8cbec03e6882\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" Oct 30 18:53:03.197205 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:03.196048 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d8d84cf-1334-4038-8b24-8cbec03e6882-workload-socket\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd\" (UID: \"6d8d84cf-1334-4038-8b24-8cbec03e6882\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" Oct 30 18:53:03.197205 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:03.196295 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d8d84cf-1334-4038-8b24-8cbec03e6882-istio-envoy\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd\" (UID: \"6d8d84cf-1334-4038-8b24-8cbec03e6882\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" Oct 30 18:53:03.197205 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:03.196593 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdhjc\" (UniqueName: \"kubernetes.io/projected/6d8d84cf-1334-4038-8b24-8cbec03e6882-kube-api-access-pdhjc\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd\" (UID: \"6d8d84cf-1334-4038-8b24-8cbec03e6882\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" Oct 30 18:53:03.197205 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:03.196665 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d8d84cf-1334-4038-8b24-8cbec03e6882-istio-podinfo\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd\" (UID: \"6d8d84cf-1334-4038-8b24-8cbec03e6882\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" Oct 30 18:53:03.197205 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:03.196782 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d8d84cf-1334-4038-8b24-8cbec03e6882-workload-certs\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd\" (UID: \"6d8d84cf-1334-4038-8b24-8cbec03e6882\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" Oct 30 18:53:03.197205 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:03.196819 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d8d84cf-1334-4038-8b24-8cbec03e6882-istiod-ca-cert\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd\" (UID: \"6d8d84cf-1334-4038-8b24-8cbec03e6882\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" Oct 30 18:53:03.219067 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:03.197932 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d8d84cf-1334-4038-8b24-8cbec03e6882-istiod-ca-cert\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd\" (UID: \"6d8d84cf-1334-4038-8b24-8cbec03e6882\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" Oct 30 18:53:03.219067 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:03.198287 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d8d84cf-1334-4038-8b24-8cbec03e6882-credential-socket\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd\" (UID: \"6d8d84cf-1334-4038-8b24-8cbec03e6882\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" Oct 30 18:53:03.219067 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:03.198536 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d8d84cf-1334-4038-8b24-8cbec03e6882-istio-data\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd\" (UID: \"6d8d84cf-1334-4038-8b24-8cbec03e6882\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" Oct 30 18:53:03.219067 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:03.199095 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d8d84cf-1334-4038-8b24-8cbec03e6882-workload-socket\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd\" (UID: \"6d8d84cf-1334-4038-8b24-8cbec03e6882\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" Oct 30 18:53:03.241260 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:03.232623 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d8d84cf-1334-4038-8b24-8cbec03e6882-workload-certs\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd\" (UID: \"6d8d84cf-1334-4038-8b24-8cbec03e6882\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" Oct 30 18:53:03.263218 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:03.262760 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d8d84cf-1334-4038-8b24-8cbec03e6882-istio-podinfo\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd\" (UID: \"6d8d84cf-1334-4038-8b24-8cbec03e6882\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" Oct 30 18:53:03.277157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:03.270591 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d8d84cf-1334-4038-8b24-8cbec03e6882-istio-envoy\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd\" (UID: \"6d8d84cf-1334-4038-8b24-8cbec03e6882\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" Oct 30 18:53:03.310162 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:03.300981 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdhjc\" (UniqueName: \"kubernetes.io/projected/6d8d84cf-1334-4038-8b24-8cbec03e6882-kube-api-access-pdhjc\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd\" (UID: \"6d8d84cf-1334-4038-8b24-8cbec03e6882\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" Oct 30 18:53:03.329157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:03.322158 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d8d84cf-1334-4038-8b24-8cbec03e6882-istio-token\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd\" (UID: \"6d8d84cf-1334-4038-8b24-8cbec03e6882\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" Oct 30 18:53:03.352162 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:03.343919 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" Oct 30 18:53:03.956782 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:03.956715 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd"] Oct 30 18:53:03.957278 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:53:03.957250 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d8d84cf_1334_4038_8b24_8cbec03e6882.slice/crio-814b606c0517cf6a063c1c6095963226d07c5f0954278b162d5034e5d386ecaa WatchSource:0}: Error finding container 814b606c0517cf6a063c1c6095963226d07c5f0954278b162d5034e5d386ecaa: Status 404 returned error can't find the container with id 814b606c0517cf6a063c1c6095963226d07c5f0954278b162d5034e5d386ecaa Oct 30 18:53:03.961588 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:03.961543 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:53:03.961726 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:03.961634 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:53:03.961726 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:03.961683 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:53:04.912635 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:04.912545 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" event={"ID":"6d8d84cf-1334-4038-8b24-8cbec03e6882","Type":"ContainerStarted","Data":"7ffd7490b3dcbf7b355af0ac00e48d063fa223293713c5445ac06795e4a88324"} Oct 30 18:53:04.912635 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:04.912607 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" event={"ID":"6d8d84cf-1334-4038-8b24-8cbec03e6882","Type":"ContainerStarted","Data":"814b606c0517cf6a063c1c6095963226d07c5f0954278b162d5034e5d386ecaa"} Oct 30 18:53:04.976992 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:04.976933 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" podStartSLOduration=2.9769145139999997 podStartE2EDuration="2.976914514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:04.964917189 +0000 UTC m=+3282.054979529" watchObservedRunningTime="2026-10-30 18:53:04.976914514 +0000 UTC m=+3282.066976854" Oct 30 18:53:05.345179 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:05.345078 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" Oct 30 18:53:05.352013 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:05.351987 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" Oct 30 18:53:05.921825 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:05.921792 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" Oct 30 18:53:05.922948 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:05.922930 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" Oct 30 18:53:07.856598 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:07.856557 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466"] Oct 30 18:53:07.863535 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:07.863506 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" Oct 30 18:53:07.881106 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:07.881080 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-551d0b-openshift-default-dockercfg-r95tk\"" Oct 30 18:53:07.905174 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:07.901853 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466"] Oct 30 18:53:08.004620 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:08.004586 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138a0f6a-a94a-4112-b13c-baf9877782c2-istio-envoy\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466\" (UID: \"138a0f6a-a94a-4112-b13c-baf9877782c2\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" Oct 30 18:53:08.004620 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:08.004622 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138a0f6a-a94a-4112-b13c-baf9877782c2-workload-certs\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466\" (UID: \"138a0f6a-a94a-4112-b13c-baf9877782c2\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" Oct 30 18:53:08.004832 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:08.004705 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138a0f6a-a94a-4112-b13c-baf9877782c2-credential-socket\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466\" (UID: \"138a0f6a-a94a-4112-b13c-baf9877782c2\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" Oct 30 18:53:08.004832 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:08.004817 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138a0f6a-a94a-4112-b13c-baf9877782c2-istio-data\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466\" (UID: \"138a0f6a-a94a-4112-b13c-baf9877782c2\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" Oct 30 18:53:08.004924 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:08.004907 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138a0f6a-a94a-4112-b13c-baf9877782c2-istio-podinfo\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466\" (UID: \"138a0f6a-a94a-4112-b13c-baf9877782c2\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" Oct 30 18:53:08.004992 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:08.004980 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138a0f6a-a94a-4112-b13c-baf9877782c2-workload-socket\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466\" (UID: \"138a0f6a-a94a-4112-b13c-baf9877782c2\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" Oct 30 18:53:08.005029 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:08.005020 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2xqs\" (UniqueName: \"kubernetes.io/projected/138a0f6a-a94a-4112-b13c-baf9877782c2-kube-api-access-r2xqs\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466\" (UID: \"138a0f6a-a94a-4112-b13c-baf9877782c2\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" Oct 30 18:53:08.005102 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:08.005090 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138a0f6a-a94a-4112-b13c-baf9877782c2-istiod-ca-cert\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466\" (UID: \"138a0f6a-a94a-4112-b13c-baf9877782c2\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" Oct 30 18:53:08.005172 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:08.005124 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/138a0f6a-a94a-4112-b13c-baf9877782c2-istio-token\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466\" (UID: \"138a0f6a-a94a-4112-b13c-baf9877782c2\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" Oct 30 18:53:08.107068 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:08.106989 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138a0f6a-a94a-4112-b13c-baf9877782c2-istio-data\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466\" (UID: \"138a0f6a-a94a-4112-b13c-baf9877782c2\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" Oct 30 18:53:08.107068 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:08.107057 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138a0f6a-a94a-4112-b13c-baf9877782c2-istio-podinfo\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466\" (UID: \"138a0f6a-a94a-4112-b13c-baf9877782c2\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" Oct 30 18:53:08.107326 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:08.107196 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138a0f6a-a94a-4112-b13c-baf9877782c2-workload-socket\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466\" (UID: \"138a0f6a-a94a-4112-b13c-baf9877782c2\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" Oct 30 18:53:08.107326 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:08.107256 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2xqs\" (UniqueName: \"kubernetes.io/projected/138a0f6a-a94a-4112-b13c-baf9877782c2-kube-api-access-r2xqs\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466\" (UID: \"138a0f6a-a94a-4112-b13c-baf9877782c2\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" Oct 30 18:53:08.107454 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:08.107347 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138a0f6a-a94a-4112-b13c-baf9877782c2-istiod-ca-cert\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466\" (UID: \"138a0f6a-a94a-4112-b13c-baf9877782c2\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" Oct 30 18:53:08.107454 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:08.107394 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/138a0f6a-a94a-4112-b13c-baf9877782c2-istio-token\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466\" (UID: \"138a0f6a-a94a-4112-b13c-baf9877782c2\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" Oct 30 18:53:08.107562 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:08.107539 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138a0f6a-a94a-4112-b13c-baf9877782c2-istio-data\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466\" (UID: \"138a0f6a-a94a-4112-b13c-baf9877782c2\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" Oct 30 18:53:08.107626 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:08.107594 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138a0f6a-a94a-4112-b13c-baf9877782c2-istio-envoy\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466\" (UID: \"138a0f6a-a94a-4112-b13c-baf9877782c2\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" Oct 30 18:53:08.107699 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:08.107624 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138a0f6a-a94a-4112-b13c-baf9877782c2-workload-certs\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466\" (UID: \"138a0f6a-a94a-4112-b13c-baf9877782c2\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" Oct 30 18:53:08.107699 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:08.107677 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138a0f6a-a94a-4112-b13c-baf9877782c2-credential-socket\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466\" (UID: \"138a0f6a-a94a-4112-b13c-baf9877782c2\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" Oct 30 18:53:08.107959 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:08.107928 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138a0f6a-a94a-4112-b13c-baf9877782c2-workload-socket\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466\" (UID: \"138a0f6a-a94a-4112-b13c-baf9877782c2\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" Oct 30 18:53:08.108200 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:08.108172 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138a0f6a-a94a-4112-b13c-baf9877782c2-credential-socket\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466\" (UID: \"138a0f6a-a94a-4112-b13c-baf9877782c2\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" Oct 30 18:53:08.108284 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:08.108235 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138a0f6a-a94a-4112-b13c-baf9877782c2-workload-certs\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466\" (UID: \"138a0f6a-a94a-4112-b13c-baf9877782c2\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" Oct 30 18:53:08.108547 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:08.108505 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138a0f6a-a94a-4112-b13c-baf9877782c2-istiod-ca-cert\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466\" (UID: \"138a0f6a-a94a-4112-b13c-baf9877782c2\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" Oct 30 18:53:08.110175 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:08.110118 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138a0f6a-a94a-4112-b13c-baf9877782c2-istio-envoy\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466\" (UID: \"138a0f6a-a94a-4112-b13c-baf9877782c2\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" Oct 30 18:53:08.110274 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:08.110208 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138a0f6a-a94a-4112-b13c-baf9877782c2-istio-podinfo\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466\" (UID: \"138a0f6a-a94a-4112-b13c-baf9877782c2\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" Oct 30 18:53:08.133507 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:08.133473 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/138a0f6a-a94a-4112-b13c-baf9877782c2-istio-token\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466\" (UID: \"138a0f6a-a94a-4112-b13c-baf9877782c2\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" Oct 30 18:53:08.134120 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:08.134101 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2xqs\" (UniqueName: \"kubernetes.io/projected/138a0f6a-a94a-4112-b13c-baf9877782c2-kube-api-access-r2xqs\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466\" (UID: \"138a0f6a-a94a-4112-b13c-baf9877782c2\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" Oct 30 18:53:08.190168 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:08.189458 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" Oct 30 18:53:08.563851 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:53:08.563814 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod138a0f6a_a94a_4112_b13c_baf9877782c2.slice/crio-02a81f84ce2a0d4eec3df8d36b6d1f3ce347ad8e4e44593dbdec2b38a7313f27 WatchSource:0}: Error finding container 02a81f84ce2a0d4eec3df8d36b6d1f3ce347ad8e4e44593dbdec2b38a7313f27: Status 404 returned error can't find the container with id 02a81f84ce2a0d4eec3df8d36b6d1f3ce347ad8e4e44593dbdec2b38a7313f27 Oct 30 18:53:08.564976 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:08.564944 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466"] Oct 30 18:53:08.567225 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:08.567179 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:53:08.567345 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:08.567259 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:53:08.567345 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:08.567288 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:53:08.965914 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:08.965874 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" event={"ID":"138a0f6a-a94a-4112-b13c-baf9877782c2","Type":"ContainerStarted","Data":"61c93894c3c8d6d5ce0fec5ec1b6d3f084f8bd763b2a9d47a62e13a06d1ea609"} Oct 30 18:53:08.966478 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:08.965943 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" event={"ID":"138a0f6a-a94a-4112-b13c-baf9877782c2","Type":"ContainerStarted","Data":"02a81f84ce2a0d4eec3df8d36b6d1f3ce347ad8e4e44593dbdec2b38a7313f27"} Oct 30 18:53:09.024346 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:09.024283 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" podStartSLOduration=2.024265919 podStartE2EDuration="2.024265919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:09.018469247 +0000 UTC m=+3286.108531587" watchObservedRunningTime="2026-10-30 18:53:09.024265919 +0000 UTC m=+3286.114328260" Oct 30 18:53:09.191357 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:09.191322 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" Oct 30 18:53:09.192917 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:09.192864 4442 patch_prober.go:29] interesting pod/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Oct 30 18:53:09.193072 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:09.192928 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" podUID="138a0f6a-a94a-4112-b13c-baf9877782c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Oct 30 18:53:10.192123 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:10.192090 4442 patch_prober.go:29] interesting pod/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Oct 30 18:53:10.192580 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:10.192160 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" podUID="138a0f6a-a94a-4112-b13c-baf9877782c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Oct 30 18:53:11.191621 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:11.191590 4442 patch_prober.go:29] interesting pod/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Oct 30 18:53:11.191804 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:11.191642 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" podUID="138a0f6a-a94a-4112-b13c-baf9877782c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Oct 30 18:53:12.202156 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:12.201593 4442 patch_prober.go:29] interesting pod/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Oct 30 18:53:12.202156 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:12.201660 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" podUID="138a0f6a-a94a-4112-b13c-baf9877782c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Oct 30 18:53:13.191585 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:13.191538 4442 patch_prober.go:29] interesting pod/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Oct 30 18:53:13.191805 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:13.191589 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" podUID="138a0f6a-a94a-4112-b13c-baf9877782c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Oct 30 18:53:14.192631 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:14.192595 4442 patch_prober.go:29] interesting pod/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Oct 30 18:53:14.193270 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:14.193237 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" podUID="138a0f6a-a94a-4112-b13c-baf9877782c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Oct 30 18:53:15.191027 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:15.190990 4442 patch_prober.go:29] interesting pod/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Oct 30 18:53:15.191260 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:15.191045 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" podUID="138a0f6a-a94a-4112-b13c-baf9877782c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Oct 30 18:53:16.197045 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:16.197011 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" Oct 30 18:53:16.197618 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:16.197352 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" Oct 30 18:53:16.198048 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:16.198027 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" Oct 30 18:53:17.563507 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:53:17.563465 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:53:23.838892 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:23.838861 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52rn6_516c6b6c-fc40-49cf-93b5-1c4471b02cf0/ovn-acl-logging/0.log" Oct 30 18:53:23.841637 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:53:23.841613 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52rn6_516c6b6c-fc40-49cf-93b5-1c4471b02cf0/ovn-acl-logging/0.log" Oct 30 18:53:47.561513 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:53:47.561470 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:54:07.160016 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:07.159981 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-551d0b/test-model-551d0b-kserve-65db7ffb5-b828f"] Oct 30 18:54:07.166782 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:07.166755 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-551d0b/test-model-551d0b-kserve-65db7ffb5-b828f" Oct 30 18:54:07.191016 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:07.190974 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-551d0b/test-model-551d0b-kserve-65db7ffb5-b828f"] Oct 30 18:54:07.195561 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:07.195535 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-551d0b\"/\"default-dockercfg-wmrst\"" Oct 30 18:54:07.195837 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:07.195813 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-551d0b\"/\"kube-root-ca.crt\"" Oct 30 18:54:07.196392 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:07.196368 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-551d0b\"/\"test-model-551d0b-kserve-self-signed-certs\"" Oct 30 18:54:07.197282 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:07.196446 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-551d0b\"/\"openshift-service-ca.crt\"" Oct 30 18:54:07.223454 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:07.223409 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ce11bc9-b3ab-4302-8338-4fa4d3db2afb-home\") pod \"test-model-551d0b-kserve-65db7ffb5-b828f\" (UID: \"2ce11bc9-b3ab-4302-8338-4fa4d3db2afb\") " pod="ai-tenant-e2e-inf-b-551d0b/test-model-551d0b-kserve-65db7ffb5-b828f" Oct 30 18:54:07.223759 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:07.223490 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ce11bc9-b3ab-4302-8338-4fa4d3db2afb-tls-certs\") pod \"test-model-551d0b-kserve-65db7ffb5-b828f\" (UID: \"2ce11bc9-b3ab-4302-8338-4fa4d3db2afb\") " pod="ai-tenant-e2e-inf-b-551d0b/test-model-551d0b-kserve-65db7ffb5-b828f" Oct 30 18:54:07.223759 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:07.223564 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ce11bc9-b3ab-4302-8338-4fa4d3db2afb-model-cache\") pod \"test-model-551d0b-kserve-65db7ffb5-b828f\" (UID: \"2ce11bc9-b3ab-4302-8338-4fa4d3db2afb\") " pod="ai-tenant-e2e-inf-b-551d0b/test-model-551d0b-kserve-65db7ffb5-b828f" Oct 30 18:54:07.223759 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:07.223623 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x47sr\" (UniqueName: \"kubernetes.io/projected/2ce11bc9-b3ab-4302-8338-4fa4d3db2afb-kube-api-access-x47sr\") pod \"test-model-551d0b-kserve-65db7ffb5-b828f\" (UID: \"2ce11bc9-b3ab-4302-8338-4fa4d3db2afb\") " pod="ai-tenant-e2e-inf-b-551d0b/test-model-551d0b-kserve-65db7ffb5-b828f" Oct 30 18:54:07.224117 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:07.223797 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ce11bc9-b3ab-4302-8338-4fa4d3db2afb-dshm\") pod \"test-model-551d0b-kserve-65db7ffb5-b828f\" (UID: \"2ce11bc9-b3ab-4302-8338-4fa4d3db2afb\") " pod="ai-tenant-e2e-inf-b-551d0b/test-model-551d0b-kserve-65db7ffb5-b828f" Oct 30 18:54:07.224117 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:07.223880 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ce11bc9-b3ab-4302-8338-4fa4d3db2afb-tmp-dir\") pod \"test-model-551d0b-kserve-65db7ffb5-b828f\" (UID: \"2ce11bc9-b3ab-4302-8338-4fa4d3db2afb\") " pod="ai-tenant-e2e-inf-b-551d0b/test-model-551d0b-kserve-65db7ffb5-b828f" Oct 30 18:54:07.325687 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:07.325656 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ce11bc9-b3ab-4302-8338-4fa4d3db2afb-tmp-dir\") pod \"test-model-551d0b-kserve-65db7ffb5-b828f\" (UID: \"2ce11bc9-b3ab-4302-8338-4fa4d3db2afb\") " pod="ai-tenant-e2e-inf-b-551d0b/test-model-551d0b-kserve-65db7ffb5-b828f" Oct 30 18:54:07.325885 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:07.325860 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ce11bc9-b3ab-4302-8338-4fa4d3db2afb-home\") pod \"test-model-551d0b-kserve-65db7ffb5-b828f\" (UID: \"2ce11bc9-b3ab-4302-8338-4fa4d3db2afb\") " pod="ai-tenant-e2e-inf-b-551d0b/test-model-551d0b-kserve-65db7ffb5-b828f" Oct 30 18:54:07.325959 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:07.325937 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ce11bc9-b3ab-4302-8338-4fa4d3db2afb-tls-certs\") pod \"test-model-551d0b-kserve-65db7ffb5-b828f\" (UID: \"2ce11bc9-b3ab-4302-8338-4fa4d3db2afb\") " pod="ai-tenant-e2e-inf-b-551d0b/test-model-551d0b-kserve-65db7ffb5-b828f" Oct 30 18:54:07.326032 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:07.325973 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ce11bc9-b3ab-4302-8338-4fa4d3db2afb-model-cache\") pod \"test-model-551d0b-kserve-65db7ffb5-b828f\" (UID: \"2ce11bc9-b3ab-4302-8338-4fa4d3db2afb\") " pod="ai-tenant-e2e-inf-b-551d0b/test-model-551d0b-kserve-65db7ffb5-b828f" Oct 30 18:54:07.326092 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:07.326029 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x47sr\" (UniqueName: \"kubernetes.io/projected/2ce11bc9-b3ab-4302-8338-4fa4d3db2afb-kube-api-access-x47sr\") pod \"test-model-551d0b-kserve-65db7ffb5-b828f\" (UID: \"2ce11bc9-b3ab-4302-8338-4fa4d3db2afb\") " pod="ai-tenant-e2e-inf-b-551d0b/test-model-551d0b-kserve-65db7ffb5-b828f" Oct 30 18:54:07.326232 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:07.326214 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ce11bc9-b3ab-4302-8338-4fa4d3db2afb-dshm\") pod \"test-model-551d0b-kserve-65db7ffb5-b828f\" (UID: \"2ce11bc9-b3ab-4302-8338-4fa4d3db2afb\") " pod="ai-tenant-e2e-inf-b-551d0b/test-model-551d0b-kserve-65db7ffb5-b828f" Oct 30 18:54:07.326337 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:07.326318 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ce11bc9-b3ab-4302-8338-4fa4d3db2afb-tmp-dir\") pod \"test-model-551d0b-kserve-65db7ffb5-b828f\" (UID: \"2ce11bc9-b3ab-4302-8338-4fa4d3db2afb\") " pod="ai-tenant-e2e-inf-b-551d0b/test-model-551d0b-kserve-65db7ffb5-b828f" Oct 30 18:54:07.326679 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:07.326657 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ce11bc9-b3ab-4302-8338-4fa4d3db2afb-model-cache\") pod \"test-model-551d0b-kserve-65db7ffb5-b828f\" (UID: \"2ce11bc9-b3ab-4302-8338-4fa4d3db2afb\") " pod="ai-tenant-e2e-inf-b-551d0b/test-model-551d0b-kserve-65db7ffb5-b828f" Oct 30 18:54:07.326679 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:07.326666 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ce11bc9-b3ab-4302-8338-4fa4d3db2afb-home\") pod \"test-model-551d0b-kserve-65db7ffb5-b828f\" (UID: \"2ce11bc9-b3ab-4302-8338-4fa4d3db2afb\") " pod="ai-tenant-e2e-inf-b-551d0b/test-model-551d0b-kserve-65db7ffb5-b828f" Oct 30 18:54:07.328954 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:07.328931 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ce11bc9-b3ab-4302-8338-4fa4d3db2afb-dshm\") pod \"test-model-551d0b-kserve-65db7ffb5-b828f\" (UID: \"2ce11bc9-b3ab-4302-8338-4fa4d3db2afb\") " pod="ai-tenant-e2e-inf-b-551d0b/test-model-551d0b-kserve-65db7ffb5-b828f" Oct 30 18:54:07.329418 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:07.329398 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ce11bc9-b3ab-4302-8338-4fa4d3db2afb-tls-certs\") pod \"test-model-551d0b-kserve-65db7ffb5-b828f\" (UID: \"2ce11bc9-b3ab-4302-8338-4fa4d3db2afb\") " pod="ai-tenant-e2e-inf-b-551d0b/test-model-551d0b-kserve-65db7ffb5-b828f" Oct 30 18:54:07.356180 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:07.356111 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x47sr\" (UniqueName: \"kubernetes.io/projected/2ce11bc9-b3ab-4302-8338-4fa4d3db2afb-kube-api-access-x47sr\") pod \"test-model-551d0b-kserve-65db7ffb5-b828f\" (UID: \"2ce11bc9-b3ab-4302-8338-4fa4d3db2afb\") " pod="ai-tenant-e2e-inf-b-551d0b/test-model-551d0b-kserve-65db7ffb5-b828f" Oct 30 18:54:07.519210 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:07.515236 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-551d0b/test-model-551d0b-kserve-65db7ffb5-b828f" Oct 30 18:54:07.989606 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:07.989391 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-551d0b/test-model-551d0b-kserve-65db7ffb5-b828f"] Oct 30 18:54:08.999101 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:08.999064 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-551d0b/test-model-551d0b-kserve-65db7ffb5-b828f" event={"ID":"2ce11bc9-b3ab-4302-8338-4fa4d3db2afb","Type":"ContainerStarted","Data":"5d1e353a8ff4ca413e7790a9440a52cd8e1f91766aa5424f6fbb8d2c7a4d4896"} Oct 30 18:54:08.999546 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:08.999109 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-551d0b/test-model-551d0b-kserve-65db7ffb5-b828f" event={"ID":"2ce11bc9-b3ab-4302-8338-4fa4d3db2afb","Type":"ContainerStarted","Data":"a0d093c2b2d91ebbc030e6a6d8a503ea2ca5227b5584a5e37a50ddd480c5222a"} Oct 30 18:54:09.045302 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:09.045239 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-551d0b/test-model-551d0b-kserve-65db7ffb5-b828f" podStartSLOduration=2.045217857 podStartE2EDuration="2.045217857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:09.038524242 +0000 UTC m=+3346.128586582" watchObservedRunningTime="2026-10-30 18:54:09.045217857 +0000 UTC m=+3346.135280197" Oct 30 18:54:09.516201 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:09.516173 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-551d0b/test-model-551d0b-kserve-65db7ffb5-b828f" Oct 30 18:54:17.784707 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:17.784667 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-551d0b/test-model-551d0b-kserve-65db7ffb5-b828f" Oct 30 18:54:17.785267 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:17.784724 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475wp9ng"] Oct 30 18:54:17.793636 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:17.793609 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-551d0b/test-model-551d0b-kserve-65db7ffb5-b828f" Oct 30 18:54:17.793801 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:17.793708 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475wp9ng" Oct 30 18:54:17.800212 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:17.800186 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475wp9ng"] Oct 30 18:54:17.828239 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:17.828208 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f23f60cb-e0e8-492b-93c4-8a04ff1f3ace-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-d475wp9ng\" (UID: \"f23f60cb-e0e8-492b-93c4-8a04ff1f3ace\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475wp9ng" Oct 30 18:54:17.828414 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:17.828392 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f23f60cb-e0e8-492b-93c4-8a04ff1f3ace-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-d475wp9ng\" (UID: \"f23f60cb-e0e8-492b-93c4-8a04ff1f3ace\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475wp9ng" Oct 30 18:54:17.828612 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:17.828597 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99mlv\" (UniqueName: \"kubernetes.io/projected/f23f60cb-e0e8-492b-93c4-8a04ff1f3ace-kube-api-access-99mlv\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-d475wp9ng\" (UID: \"f23f60cb-e0e8-492b-93c4-8a04ff1f3ace\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475wp9ng" Oct 30 18:54:17.828806 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:17.828785 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f23f60cb-e0e8-492b-93c4-8a04ff1f3ace-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-d475wp9ng\" (UID: \"f23f60cb-e0e8-492b-93c4-8a04ff1f3ace\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475wp9ng" Oct 30 18:54:17.833426 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:54:17.833397 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:54:17.930592 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:17.930559 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f23f60cb-e0e8-492b-93c4-8a04ff1f3ace-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-d475wp9ng\" (UID: \"f23f60cb-e0e8-492b-93c4-8a04ff1f3ace\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475wp9ng" Oct 30 18:54:17.930746 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:17.930626 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99mlv\" (UniqueName: \"kubernetes.io/projected/f23f60cb-e0e8-492b-93c4-8a04ff1f3ace-kube-api-access-99mlv\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-d475wp9ng\" (UID: \"f23f60cb-e0e8-492b-93c4-8a04ff1f3ace\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475wp9ng" Oct 30 18:54:17.930746 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:17.930676 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f23f60cb-e0e8-492b-93c4-8a04ff1f3ace-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-d475wp9ng\" (UID: \"f23f60cb-e0e8-492b-93c4-8a04ff1f3ace\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475wp9ng" Oct 30 18:54:17.930884 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:17.930866 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f23f60cb-e0e8-492b-93c4-8a04ff1f3ace-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-d475wp9ng\" (UID: \"f23f60cb-e0e8-492b-93c4-8a04ff1f3ace\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475wp9ng" Oct 30 18:54:17.931660 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:17.931629 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f23f60cb-e0e8-492b-93c4-8a04ff1f3ace-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-d475wp9ng\" (UID: \"f23f60cb-e0e8-492b-93c4-8a04ff1f3ace\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475wp9ng" Oct 30 18:54:17.931827 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:17.931778 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f23f60cb-e0e8-492b-93c4-8a04ff1f3ace-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-d475wp9ng\" (UID: \"f23f60cb-e0e8-492b-93c4-8a04ff1f3ace\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475wp9ng" Oct 30 18:54:17.933676 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:17.933649 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f23f60cb-e0e8-492b-93c4-8a04ff1f3ace-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-d475wp9ng\" (UID: \"f23f60cb-e0e8-492b-93c4-8a04ff1f3ace\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475wp9ng" Oct 30 18:54:17.951399 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:17.951370 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99mlv\" (UniqueName: \"kubernetes.io/projected/f23f60cb-e0e8-492b-93c4-8a04ff1f3ace-kube-api-access-99mlv\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-d475wp9ng\" (UID: \"f23f60cb-e0e8-492b-93c4-8a04ff1f3ace\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475wp9ng" Oct 30 18:54:18.117004 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:18.116981 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475wp9ng" Oct 30 18:54:18.438219 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:54:18.425202 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf23f60cb_e0e8_492b_93c4_8a04ff1f3ace.slice/crio-1ad3d0e938b52bc21564a99442426346d1109c30b0d94003b197f9bb586f7e05 WatchSource:0}: Error finding container 1ad3d0e938b52bc21564a99442426346d1109c30b0d94003b197f9bb586f7e05: Status 404 returned error can't find the container with id 1ad3d0e938b52bc21564a99442426346d1109c30b0d94003b197f9bb586f7e05 Oct 30 18:54:18.438219 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:18.427024 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475wp9ng"] Oct 30 18:54:18.528446 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:18.528416 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-551d0b/test-model-551d0b-kserve-65db7ffb5-b828f" Oct 30 18:54:19.079762 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:19.079727 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475wp9ng" event={"ID":"f23f60cb-e0e8-492b-93c4-8a04ff1f3ace","Type":"ContainerStarted","Data":"96e0169ffbbbc632f40f1b154549f23214158207707cac5376eaf92a3b1df38e"} Oct 30 18:54:19.080407 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:19.079787 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475wp9ng" event={"ID":"f23f60cb-e0e8-492b-93c4-8a04ff1f3ace","Type":"ContainerStarted","Data":"1ad3d0e938b52bc21564a99442426346d1109c30b0d94003b197f9bb586f7e05"} Oct 30 18:54:19.117886 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:19.117832 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475wp9ng" podStartSLOduration=2.117818124 podStartE2EDuration="2.117818124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:19.112561205 +0000 UTC m=+3356.202623534" watchObservedRunningTime="2026-10-30 18:54:19.117818124 +0000 UTC m=+3356.207880464" Oct 30 18:54:20.084271 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:20.084240 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475wp9ng" Oct 30 18:54:26.100308 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:54:26.100274 4442 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-861bb3/agc-cred-245a97933c01: secret "agc-cred-245a97933c01" not found Oct 30 18:54:26.100308 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:54:26.100310 4442 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x: secret "agc-cred-245a97933c01" not found Oct 30 18:54:26.101470 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:54:26.100416 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c310be7-bed9-487f-b4bc-58cca0ea2703-provider-credentials podName:1c310be7-bed9-487f-b4bc-58cca0ea2703 nodeName:}" failed. No retries permitted until 2026-10-30 18:54:26.600394718 +0000 UTC m=+3363.690457056 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1c310be7-bed9-487f-b4bc-58cca0ea2703-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x" (UID: "1c310be7-bed9-487f-b4bc-58cca0ea2703") : secret "agc-cred-245a97933c01" not found Oct 30 18:54:26.609990 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:54:26.609962 4442 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-861bb3/agc-cred-245a97933c01: secret "agc-cred-245a97933c01" not found Oct 30 18:54:26.609990 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:54:26.609990 4442 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x: secret "agc-cred-245a97933c01" not found Oct 30 18:54:26.610238 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:54:26.610043 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c310be7-bed9-487f-b4bc-58cca0ea2703-provider-credentials podName:1c310be7-bed9-487f-b4bc-58cca0ea2703 nodeName:}" failed. No retries permitted until 2026-10-30 18:54:27.610028237 +0000 UTC m=+3364.700090554 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1c310be7-bed9-487f-b4bc-58cca0ea2703-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x" (UID: "1c310be7-bed9-487f-b4bc-58cca0ea2703") : secret "agc-cred-245a97933c01" not found Oct 30 18:54:27.629242 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:54:27.629209 4442 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-861bb3/agc-cred-245a97933c01: secret "agc-cred-245a97933c01" not found Oct 30 18:54:27.629242 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:54:27.629242 4442 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x: secret "agc-cred-245a97933c01" not found Oct 30 18:54:27.629781 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:54:27.629311 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c310be7-bed9-487f-b4bc-58cca0ea2703-provider-credentials podName:1c310be7-bed9-487f-b4bc-58cca0ea2703 nodeName:}" failed. No retries permitted until 2026-10-30 18:54:29.629291181 +0000 UTC m=+3366.719353518 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1c310be7-bed9-487f-b4bc-58cca0ea2703-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x" (UID: "1c310be7-bed9-487f-b4bc-58cca0ea2703") : secret "agc-cred-245a97933c01" not found Oct 30 18:54:29.503868 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:29.503834 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-9b6ds94sp"] Oct 30 18:54:29.680700 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:54:29.680664 4442 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-861bb3/agc-cred-245a97933c01: secret "agc-cred-245a97933c01" not found Oct 30 18:54:29.680700 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:54:29.680698 4442 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x: secret "agc-cred-245a97933c01" not found Oct 30 18:54:29.680964 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:54:29.680764 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c310be7-bed9-487f-b4bc-58cca0ea2703-provider-credentials podName:1c310be7-bed9-487f-b4bc-58cca0ea2703 nodeName:}" failed. No retries permitted until 2026-10-30 18:54:33.680746566 +0000 UTC m=+3370.770808896 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1c310be7-bed9-487f-b4bc-58cca0ea2703-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x" (UID: "1c310be7-bed9-487f-b4bc-58cca0ea2703") : secret "agc-cred-245a97933c01" not found Oct 30 18:54:31.096946 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:31.096919 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475wp9ng" Oct 30 18:54:31.177942 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:31.177895 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x"] Oct 30 18:54:33.702363 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:54:33.702332 4442 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-861bb3/agc-cred-245a97933c01: secret "agc-cred-245a97933c01" not found Oct 30 18:54:33.702363 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:54:33.702364 4442 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x: secret "agc-cred-245a97933c01" not found Oct 30 18:54:33.702914 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:54:33.702432 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c310be7-bed9-487f-b4bc-58cca0ea2703-provider-credentials podName:1c310be7-bed9-487f-b4bc-58cca0ea2703 nodeName:}" failed. No retries permitted until 2026-10-30 18:54:41.70241119 +0000 UTC m=+3378.792473540 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1c310be7-bed9-487f-b4bc-58cca0ea2703-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x" (UID: "1c310be7-bed9-487f-b4bc-58cca0ea2703") : secret "agc-cred-245a97933c01" not found Oct 30 18:54:34.504196 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:34.504085 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-9b6ds94sp" podUID="43442120-12b1-4c2f-b993-80490b3f415f" containerName="payload-processing" containerID="cri-o://16f3abd2bb8113270cf631d440d45d1a61e0a7adf91fa4be72ac99d70e0f6b4e" gracePeriod=25 Oct 30 18:54:34.806174 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:34.806145 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-9b6ds94sp" Oct 30 18:54:34.921942 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:34.921911 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43442120-12b1-4c2f-b993-80490b3f415f-kube-api-access-p5jgk\" (UniqueName: \"kubernetes.io/projected/43442120-12b1-4c2f-b993-80490b3f415f-kube-api-access-p5jgk\") pod \"43442120-12b1-4c2f-b993-80490b3f415f\" (UID: \"43442120-12b1-4c2f-b993-80490b3f415f\") " Oct 30 18:54:34.922176 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:34.922074 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43442120-12b1-4c2f-b993-80490b3f415f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/43442120-12b1-4c2f-b993-80490b3f415f-routing-overlay-volume\") pod \"43442120-12b1-4c2f-b993-80490b3f415f\" (UID: \"43442120-12b1-4c2f-b993-80490b3f415f\") " Oct 30 18:54:34.922335 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:34.922321 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43442120-12b1-4c2f-b993-80490b3f415f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/43442120-12b1-4c2f-b993-80490b3f415f-provider-credentials\") pod \"43442120-12b1-4c2f-b993-80490b3f415f\" (UID: \"43442120-12b1-4c2f-b993-80490b3f415f\") " Oct 30 18:54:34.922420 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:34.922377 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43442120-12b1-4c2f-b993-80490b3f415f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43442120-12b1-4c2f-b993-80490b3f415f-plugins-config-volume\") pod \"43442120-12b1-4c2f-b993-80490b3f415f\" (UID: \"43442120-12b1-4c2f-b993-80490b3f415f\") " Oct 30 18:54:34.922525 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:34.922502 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43442120-12b1-4c2f-b993-80490b3f415f-routing-overlay-volume" pod "43442120-12b1-4c2f-b993-80490b3f415f" (UID: "43442120-12b1-4c2f-b993-80490b3f415f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:54:34.922879 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:34.922856 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43442120-12b1-4c2f-b993-80490b3f415f-plugins-config-volume" pod "43442120-12b1-4c2f-b993-80490b3f415f" (UID: "43442120-12b1-4c2f-b993-80490b3f415f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:54:34.923070 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:34.923056 4442 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/43442120-12b1-4c2f-b993-80490b3f415f-routing-overlay-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:54:34.923111 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:34.923073 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43442120-12b1-4c2f-b993-80490b3f415f-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:54:34.924666 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:34.924635 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43442120-12b1-4c2f-b993-80490b3f415f-provider-credentials" pod "43442120-12b1-4c2f-b993-80490b3f415f" (UID: "43442120-12b1-4c2f-b993-80490b3f415f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:54:34.924776 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:34.924749 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43442120-12b1-4c2f-b993-80490b3f415f-kube-api-access-p5jgk" pod "43442120-12b1-4c2f-b993-80490b3f415f" (UID: "43442120-12b1-4c2f-b993-80490b3f415f"). InnerVolumeSpecName "kube-api-access-p5jgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:54:35.025521 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:35.025437 4442 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/43442120-12b1-4c2f-b993-80490b3f415f-provider-credentials\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:54:35.025521 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:35.025465 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5jgk\" (UniqueName: \"kubernetes.io/projected/43442120-12b1-4c2f-b993-80490b3f415f-kube-api-access-p5jgk\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:54:35.323215 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:35.320972 4442 generic.go:356] "Generic (PLEG): container finished" podID="43442120-12b1-4c2f-b993-80490b3f415f" containerID="16f3abd2bb8113270cf631d440d45d1a61e0a7adf91fa4be72ac99d70e0f6b4e" exitCode=0 Oct 30 18:54:35.323215 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:35.321192 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-9b6ds94sp" event={"ID":"43442120-12b1-4c2f-b993-80490b3f415f","Type":"ContainerDied","Data":"16f3abd2bb8113270cf631d440d45d1a61e0a7adf91fa4be72ac99d70e0f6b4e"} Oct 30 18:54:35.323215 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:35.321258 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-9b6ds94sp" event={"ID":"43442120-12b1-4c2f-b993-80490b3f415f","Type":"ContainerDied","Data":"6f7b811eee0ee6499571cfbdb6c2b3cf302727bd0d5b353cd8dcc657615b9218"} Oct 30 18:54:35.323215 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:35.321288 4442 scope.go:122] "RemoveContainer" containerID="16f3abd2bb8113270cf631d440d45d1a61e0a7adf91fa4be72ac99d70e0f6b4e" Oct 30 18:54:35.323215 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:35.321470 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-9b6ds94sp" Oct 30 18:54:35.360338 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:35.356311 4442 scope.go:122] "RemoveContainer" containerID="16f3abd2bb8113270cf631d440d45d1a61e0a7adf91fa4be72ac99d70e0f6b4e" Oct 30 18:54:35.373157 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:54:35.371349 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16f3abd2bb8113270cf631d440d45d1a61e0a7adf91fa4be72ac99d70e0f6b4e\": container with ID starting with 16f3abd2bb8113270cf631d440d45d1a61e0a7adf91fa4be72ac99d70e0f6b4e not found: ID does not exist" containerID="16f3abd2bb8113270cf631d440d45d1a61e0a7adf91fa4be72ac99d70e0f6b4e" Oct 30 18:54:35.373157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:35.371399 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16f3abd2bb8113270cf631d440d45d1a61e0a7adf91fa4be72ac99d70e0f6b4e"} err="failed to get container status \"16f3abd2bb8113270cf631d440d45d1a61e0a7adf91fa4be72ac99d70e0f6b4e\": rpc error: code = NotFound desc = could not find container \"16f3abd2bb8113270cf631d440d45d1a61e0a7adf91fa4be72ac99d70e0f6b4e\": container with ID starting with 16f3abd2bb8113270cf631d440d45d1a61e0a7adf91fa4be72ac99d70e0f6b4e not found: ID does not exist" Oct 30 18:54:35.486157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:35.476233 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-9b6ds94sp"] Oct 30 18:54:35.529211 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:35.528776 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-9b6ds94sp"] Oct 30 18:54:36.187156 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:36.181189 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x" podUID="1c310be7-bed9-487f-b4bc-58cca0ea2703" containerName="payload-processing" containerID="cri-o://e5daae1cc0f56e4b7375d19b4347b75e399c278d365fc5f73460b4fb3cc35ac8" gracePeriod=25 Oct 30 18:54:36.692360 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:36.692332 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x" Oct 30 18:54:36.836396 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:36.836228 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c310be7-bed9-487f-b4bc-58cca0ea2703-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c310be7-bed9-487f-b4bc-58cca0ea2703-plugins-config-volume\") pod \"1c310be7-bed9-487f-b4bc-58cca0ea2703\" (UID: \"1c310be7-bed9-487f-b4bc-58cca0ea2703\") " Oct 30 18:54:36.836396 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:36.836328 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c310be7-bed9-487f-b4bc-58cca0ea2703-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c310be7-bed9-487f-b4bc-58cca0ea2703-provider-credentials\") pod \"1c310be7-bed9-487f-b4bc-58cca0ea2703\" (UID: \"1c310be7-bed9-487f-b4bc-58cca0ea2703\") " Oct 30 18:54:36.836396 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:36.836381 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c310be7-bed9-487f-b4bc-58cca0ea2703-kube-api-access-7q2rj\" (UniqueName: \"kubernetes.io/projected/1c310be7-bed9-487f-b4bc-58cca0ea2703-kube-api-access-7q2rj\") pod \"1c310be7-bed9-487f-b4bc-58cca0ea2703\" (UID: \"1c310be7-bed9-487f-b4bc-58cca0ea2703\") " Oct 30 18:54:36.836709 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:36.836495 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c310be7-bed9-487f-b4bc-58cca0ea2703-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c310be7-bed9-487f-b4bc-58cca0ea2703-routing-overlay-volume\") pod \"1c310be7-bed9-487f-b4bc-58cca0ea2703\" (UID: \"1c310be7-bed9-487f-b4bc-58cca0ea2703\") " Oct 30 18:54:36.837732 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:36.837702 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c310be7-bed9-487f-b4bc-58cca0ea2703-routing-overlay-volume" pod "1c310be7-bed9-487f-b4bc-58cca0ea2703" (UID: "1c310be7-bed9-487f-b4bc-58cca0ea2703"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:54:36.838701 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:36.838663 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c310be7-bed9-487f-b4bc-58cca0ea2703-plugins-config-volume" pod "1c310be7-bed9-487f-b4bc-58cca0ea2703" (UID: "1c310be7-bed9-487f-b4bc-58cca0ea2703"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:54:36.840505 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:36.840484 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c310be7-bed9-487f-b4bc-58cca0ea2703-provider-credentials" pod "1c310be7-bed9-487f-b4bc-58cca0ea2703" (UID: "1c310be7-bed9-487f-b4bc-58cca0ea2703"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:54:36.840738 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:36.840691 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c310be7-bed9-487f-b4bc-58cca0ea2703-kube-api-access-7q2rj" pod "1c310be7-bed9-487f-b4bc-58cca0ea2703" (UID: "1c310be7-bed9-487f-b4bc-58cca0ea2703"). InnerVolumeSpecName "kube-api-access-7q2rj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:54:36.949269 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:36.949219 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c310be7-bed9-487f-b4bc-58cca0ea2703-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:54:36.949269 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:36.949255 4442 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c310be7-bed9-487f-b4bc-58cca0ea2703-provider-credentials\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:54:36.949269 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:36.949271 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q2rj\" (UniqueName: \"kubernetes.io/projected/1c310be7-bed9-487f-b4bc-58cca0ea2703-kube-api-access-7q2rj\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:54:36.949617 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:36.949297 4442 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c310be7-bed9-487f-b4bc-58cca0ea2703-routing-overlay-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:54:37.131204 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:37.128619 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43442120-12b1-4c2f-b993-80490b3f415f" path="/var/lib/kubelet/pods/43442120-12b1-4c2f-b993-80490b3f415f/volumes" Oct 30 18:54:37.503503 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:37.503405 4442 generic.go:356] "Generic (PLEG): container finished" podID="1c310be7-bed9-487f-b4bc-58cca0ea2703" containerID="e5daae1cc0f56e4b7375d19b4347b75e399c278d365fc5f73460b4fb3cc35ac8" exitCode=0 Oct 30 18:54:37.505604 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:37.505576 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x" Oct 30 18:54:37.508680 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:37.505818 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x" event={"ID":"1c310be7-bed9-487f-b4bc-58cca0ea2703","Type":"ContainerDied","Data":"e5daae1cc0f56e4b7375d19b4347b75e399c278d365fc5f73460b4fb3cc35ac8"} Oct 30 18:54:37.508680 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:37.508550 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x" event={"ID":"1c310be7-bed9-487f-b4bc-58cca0ea2703","Type":"ContainerDied","Data":"acac2609343f5b3d9a93805ae0c30709b7586d88c79ffdfe74bf832dbf62a98f"} Oct 30 18:54:37.508680 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:37.508587 4442 scope.go:122] "RemoveContainer" containerID="e5daae1cc0f56e4b7375d19b4347b75e399c278d365fc5f73460b4fb3cc35ac8" Oct 30 18:54:37.621283 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:37.621249 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x"] Oct 30 18:54:37.662218 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:37.662186 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6f5b2g45x"] Oct 30 18:54:37.767286 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:37.750719 4442 scope.go:122] "RemoveContainer" containerID="e5daae1cc0f56e4b7375d19b4347b75e399c278d365fc5f73460b4fb3cc35ac8" Oct 30 18:54:37.768686 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:54:37.768450 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5daae1cc0f56e4b7375d19b4347b75e399c278d365fc5f73460b4fb3cc35ac8\": container with ID starting with e5daae1cc0f56e4b7375d19b4347b75e399c278d365fc5f73460b4fb3cc35ac8 not found: ID does not exist" containerID="e5daae1cc0f56e4b7375d19b4347b75e399c278d365fc5f73460b4fb3cc35ac8" Oct 30 18:54:37.792460 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:37.792249 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5daae1cc0f56e4b7375d19b4347b75e399c278d365fc5f73460b4fb3cc35ac8"} err="failed to get container status \"e5daae1cc0f56e4b7375d19b4347b75e399c278d365fc5f73460b4fb3cc35ac8\": rpc error: code = NotFound desc = could not find container \"e5daae1cc0f56e4b7375d19b4347b75e399c278d365fc5f73460b4fb3cc35ac8\": container with ID starting with e5daae1cc0f56e4b7375d19b4347b75e399c278d365fc5f73460b4fb3cc35ac8 not found: ID does not exist" Oct 30 18:54:39.077295 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:39.077218 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c310be7-bed9-487f-b4bc-58cca0ea2703" path="/var/lib/kubelet/pods/1c310be7-bed9-487f-b4bc-58cca0ea2703/volumes" Oct 30 18:54:40.833696 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:40.830193 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdmbkvd"] Oct 30 18:54:40.833696 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:40.831510 4442 cpu_manager.go:414] "containerMap: removing container" podUID="43442120-12b1-4c2f-b993-80490b3f415f" containerName="payload-processing" Oct 30 18:54:40.833696 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:40.831531 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43442120-12b1-4c2f-b993-80490b3f415f" containerName="payload-processing" Oct 30 18:54:40.833696 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:40.831564 4442 cpu_manager.go:414] "containerMap: removing container" podUID="1c310be7-bed9-487f-b4bc-58cca0ea2703" containerName="payload-processing" Oct 30 18:54:40.833696 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:40.831574 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c310be7-bed9-487f-b4bc-58cca0ea2703" containerName="payload-processing" Oct 30 18:54:40.833696 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:40.831821 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="43442120-12b1-4c2f-b993-80490b3f415f" containerName="payload-processing" Oct 30 18:54:40.833696 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:40.831840 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c310be7-bed9-487f-b4bc-58cca0ea2703" containerName="payload-processing" Oct 30 18:54:40.838377 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:40.838339 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdmbkvd" Oct 30 18:54:40.883482 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:40.883410 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdmbkvd"] Oct 30 18:54:40.913485 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:40.913046 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-6ffdmbkvd\" (UID: \"a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdmbkvd" Oct 30 18:54:40.913485 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:40.913111 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhtpq\" (UniqueName: \"kubernetes.io/projected/a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864-kube-api-access-fhtpq\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-6ffdmbkvd\" (UID: \"a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdmbkvd" Oct 30 18:54:40.913485 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:40.913250 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-6ffdmbkvd\" (UID: \"a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdmbkvd" Oct 30 18:54:40.913485 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:40.913458 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-6ffdmbkvd\" (UID: \"a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdmbkvd" Oct 30 18:54:41.015080 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:41.015045 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-6ffdmbkvd\" (UID: \"a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdmbkvd" Oct 30 18:54:41.015289 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:41.015213 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-6ffdmbkvd\" (UID: \"a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdmbkvd" Oct 30 18:54:41.015289 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:41.015258 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhtpq\" (UniqueName: \"kubernetes.io/projected/a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864-kube-api-access-fhtpq\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-6ffdmbkvd\" (UID: \"a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdmbkvd" Oct 30 18:54:41.015377 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:41.015351 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-6ffdmbkvd\" (UID: \"a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdmbkvd" Oct 30 18:54:41.016268 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:41.016238 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-6ffdmbkvd\" (UID: \"a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdmbkvd" Oct 30 18:54:41.016418 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:41.016246 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-6ffdmbkvd\" (UID: \"a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdmbkvd" Oct 30 18:54:41.017897 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:41.017871 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-6ffdmbkvd\" (UID: \"a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdmbkvd" Oct 30 18:54:41.036192 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:41.036165 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhtpq\" (UniqueName: \"kubernetes.io/projected/a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864-kube-api-access-fhtpq\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-6ffdmbkvd\" (UID: \"a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdmbkvd" Oct 30 18:54:41.274160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:41.273453 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdmbkvd" Oct 30 18:54:41.885208 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:41.879384 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789f79ngk"] Oct 30 18:54:41.900166 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:41.896794 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789f79ngk"] Oct 30 18:54:41.900166 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:41.896912 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789f79ngk" Oct 30 18:54:41.919182 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:41.914363 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6018b4\"/\"agc-cred-8403349d486c\"" Oct 30 18:54:41.919182 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:41.915528 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6018b4\"/\"payload-processing-external-model-e2e--6d8a874f-dockercfg-j6h5f\"" Oct 30 18:54:41.930982 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:41.930959 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6018b4\"/\"routing-overlay\"" Oct 30 18:54:41.931389 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:41.931264 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6018b4\"/\"kube-root-ca.crt\"" Oct 30 18:54:41.931483 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:41.931350 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6018b4\"/\"openshift-service-ca.crt\"" Oct 30 18:54:41.934614 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:41.934377 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6018b4\"/\"payload-processing-external-model-plugins-e2e-extmod-b-6018b4\"" Oct 30 18:54:41.966907 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:41.966477 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e4979818-bca0-49e1-aebd-bdc7bdfafe3e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-789f79ngk\" (UID: \"e4979818-bca0-49e1-aebd-bdc7bdfafe3e\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789f79ngk" Oct 30 18:54:41.966907 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:41.966622 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxqjn\" (UniqueName: \"kubernetes.io/projected/e4979818-bca0-49e1-aebd-bdc7bdfafe3e-kube-api-access-vxqjn\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-789f79ngk\" (UID: \"e4979818-bca0-49e1-aebd-bdc7bdfafe3e\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789f79ngk" Oct 30 18:54:41.966907 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:41.966645 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4979818-bca0-49e1-aebd-bdc7bdfafe3e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-789f79ngk\" (UID: \"e4979818-bca0-49e1-aebd-bdc7bdfafe3e\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789f79ngk" Oct 30 18:54:41.966907 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:41.966727 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e4979818-bca0-49e1-aebd-bdc7bdfafe3e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-789f79ngk\" (UID: \"e4979818-bca0-49e1-aebd-bdc7bdfafe3e\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789f79ngk" Oct 30 18:54:42.068309 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:42.068274 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxqjn\" (UniqueName: \"kubernetes.io/projected/e4979818-bca0-49e1-aebd-bdc7bdfafe3e-kube-api-access-vxqjn\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-789f79ngk\" (UID: \"e4979818-bca0-49e1-aebd-bdc7bdfafe3e\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789f79ngk" Oct 30 18:54:42.068309 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:42.068322 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4979818-bca0-49e1-aebd-bdc7bdfafe3e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-789f79ngk\" (UID: \"e4979818-bca0-49e1-aebd-bdc7bdfafe3e\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789f79ngk" Oct 30 18:54:42.068552 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:42.068413 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e4979818-bca0-49e1-aebd-bdc7bdfafe3e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-789f79ngk\" (UID: \"e4979818-bca0-49e1-aebd-bdc7bdfafe3e\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789f79ngk" Oct 30 18:54:42.068552 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:42.068524 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e4979818-bca0-49e1-aebd-bdc7bdfafe3e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-789f79ngk\" (UID: \"e4979818-bca0-49e1-aebd-bdc7bdfafe3e\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789f79ngk" Oct 30 18:54:42.068674 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:54:42.068650 4442 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6018b4/agc-cred-8403349d486c: secret "agc-cred-8403349d486c" not found Oct 30 18:54:42.068733 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:54:42.068673 4442 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789f79ngk: secret "agc-cred-8403349d486c" not found Oct 30 18:54:42.068795 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:54:42.068776 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4979818-bca0-49e1-aebd-bdc7bdfafe3e-provider-credentials podName:e4979818-bca0-49e1-aebd-bdc7bdfafe3e nodeName:}" failed. No retries permitted until 2026-10-30 18:54:42.568752472 +0000 UTC m=+3379.658814804 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e4979818-bca0-49e1-aebd-bdc7bdfafe3e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-6018b4-789f79ngk" (UID: "e4979818-bca0-49e1-aebd-bdc7bdfafe3e") : secret "agc-cred-8403349d486c" not found Oct 30 18:54:42.069169 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:42.069147 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4979818-bca0-49e1-aebd-bdc7bdfafe3e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-789f79ngk\" (UID: \"e4979818-bca0-49e1-aebd-bdc7bdfafe3e\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789f79ngk" Oct 30 18:54:42.069254 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:42.069172 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e4979818-bca0-49e1-aebd-bdc7bdfafe3e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-789f79ngk\" (UID: \"e4979818-bca0-49e1-aebd-bdc7bdfafe3e\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789f79ngk" Oct 30 18:54:42.092825 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:42.092803 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxqjn\" (UniqueName: \"kubernetes.io/projected/e4979818-bca0-49e1-aebd-bdc7bdfafe3e-kube-api-access-vxqjn\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-789f79ngk\" (UID: \"e4979818-bca0-49e1-aebd-bdc7bdfafe3e\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789f79ngk" Oct 30 18:54:42.122645 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:54:42.122606 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2abfeaa_13f4_4e3e_8d8a_b3d535d9d864.slice/crio-9638faf3ddbcce77ef6c5c9ca1b624fb01ecc42347211276e64428f339cf3ba2 WatchSource:0}: Error finding container 9638faf3ddbcce77ef6c5c9ca1b624fb01ecc42347211276e64428f339cf3ba2: Status 404 returned error can't find the container with id 9638faf3ddbcce77ef6c5c9ca1b624fb01ecc42347211276e64428f339cf3ba2 Oct 30 18:54:42.141094 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:42.141026 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdmbkvd"] Oct 30 18:54:42.185476 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:42.185448 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475wp9ng"] Oct 30 18:54:42.191663 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:42.191634 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdmbkvd"] Oct 30 18:54:42.587303 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:42.580710 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e4979818-bca0-49e1-aebd-bdc7bdfafe3e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-789f79ngk\" (UID: \"e4979818-bca0-49e1-aebd-bdc7bdfafe3e\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789f79ngk" Oct 30 18:54:42.587303 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:54:42.581325 4442 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6018b4/agc-cred-8403349d486c: secret "agc-cred-8403349d486c" not found Oct 30 18:54:42.587303 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:54:42.581348 4442 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789f79ngk: secret "agc-cred-8403349d486c" not found Oct 30 18:54:42.587303 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:54:42.581414 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4979818-bca0-49e1-aebd-bdc7bdfafe3e-provider-credentials podName:e4979818-bca0-49e1-aebd-bdc7bdfafe3e nodeName:}" failed. No retries permitted until 2026-10-30 18:54:43.581393159 +0000 UTC m=+3380.671455480 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e4979818-bca0-49e1-aebd-bdc7bdfafe3e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-6018b4-789f79ngk" (UID: "e4979818-bca0-49e1-aebd-bdc7bdfafe3e") : secret "agc-cred-8403349d486c" not found Oct 30 18:54:42.964733 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:42.964691 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdmbkvd" event={"ID":"a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864","Type":"ContainerStarted","Data":"32487b080a963bf47626781d8ee4810f46bdd5a5fa3d5457a8f6ebd45dc9e2bd"} Oct 30 18:54:42.965235 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:42.964761 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdmbkvd" event={"ID":"a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864","Type":"ContainerStarted","Data":"9638faf3ddbcce77ef6c5c9ca1b624fb01ecc42347211276e64428f339cf3ba2"} Oct 30 18:54:42.965235 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:42.964793 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdmbkvd" Oct 30 18:54:43.003367 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:43.003308 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdmbkvd" podStartSLOduration=3.003290328 podStartE2EDuration="3.003290328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:43.002646886 +0000 UTC m=+3380.092709219" watchObservedRunningTime="2026-10-30 18:54:43.003290328 +0000 UTC m=+3380.093352667" Oct 30 18:54:43.076185 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:43.076118 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789f79ngk"] Oct 30 18:54:43.076677 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:54:43.076657 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789f79ngk" podUID="e4979818-bca0-49e1-aebd-bdc7bdfafe3e" Oct 30 18:54:43.619160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:43.616639 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e4979818-bca0-49e1-aebd-bdc7bdfafe3e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-789f79ngk\" (UID: \"e4979818-bca0-49e1-aebd-bdc7bdfafe3e\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789f79ngk" Oct 30 18:54:43.619160 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:54:43.617165 4442 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6018b4/agc-cred-8403349d486c: secret "agc-cred-8403349d486c" not found Oct 30 18:54:43.619160 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:54:43.617185 4442 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789f79ngk: secret "agc-cred-8403349d486c" not found Oct 30 18:54:43.619160 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:54:43.617249 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4979818-bca0-49e1-aebd-bdc7bdfafe3e-provider-credentials podName:e4979818-bca0-49e1-aebd-bdc7bdfafe3e nodeName:}" failed. No retries permitted until 2026-10-30 18:54:45.617230062 +0000 UTC m=+3382.707292382 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e4979818-bca0-49e1-aebd-bdc7bdfafe3e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-6018b4-789f79ngk" (UID: "e4979818-bca0-49e1-aebd-bdc7bdfafe3e") : secret "agc-cred-8403349d486c" not found Oct 30 18:54:43.979534 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:43.979448 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789f79ngk" Oct 30 18:54:43.988264 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:43.988236 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789f79ngk" Oct 30 18:54:44.144164 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:44.139780 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4979818-bca0-49e1-aebd-bdc7bdfafe3e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e4979818-bca0-49e1-aebd-bdc7bdfafe3e-routing-overlay-volume\") pod \"e4979818-bca0-49e1-aebd-bdc7bdfafe3e\" (UID: \"e4979818-bca0-49e1-aebd-bdc7bdfafe3e\") " Oct 30 18:54:44.144164 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:44.139915 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4979818-bca0-49e1-aebd-bdc7bdfafe3e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4979818-bca0-49e1-aebd-bdc7bdfafe3e-plugins-config-volume\") pod \"e4979818-bca0-49e1-aebd-bdc7bdfafe3e\" (UID: \"e4979818-bca0-49e1-aebd-bdc7bdfafe3e\") " Oct 30 18:54:44.144164 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:44.140291 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4979818-bca0-49e1-aebd-bdc7bdfafe3e-kube-api-access-vxqjn\" (UniqueName: \"kubernetes.io/projected/e4979818-bca0-49e1-aebd-bdc7bdfafe3e-kube-api-access-vxqjn\") pod \"e4979818-bca0-49e1-aebd-bdc7bdfafe3e\" (UID: \"e4979818-bca0-49e1-aebd-bdc7bdfafe3e\") " Oct 30 18:54:44.163019 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:44.157387 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4979818-bca0-49e1-aebd-bdc7bdfafe3e-plugins-config-volume" pod "e4979818-bca0-49e1-aebd-bdc7bdfafe3e" (UID: "e4979818-bca0-49e1-aebd-bdc7bdfafe3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:54:44.163019 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:44.157807 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4979818-bca0-49e1-aebd-bdc7bdfafe3e-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:54:44.163019 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:44.158067 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4979818-bca0-49e1-aebd-bdc7bdfafe3e-routing-overlay-volume" pod "e4979818-bca0-49e1-aebd-bdc7bdfafe3e" (UID: "e4979818-bca0-49e1-aebd-bdc7bdfafe3e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:54:44.186127 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:44.185583 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4979818-bca0-49e1-aebd-bdc7bdfafe3e-kube-api-access-vxqjn" pod "e4979818-bca0-49e1-aebd-bdc7bdfafe3e" (UID: "e4979818-bca0-49e1-aebd-bdc7bdfafe3e"). InnerVolumeSpecName "kube-api-access-vxqjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:54:44.259160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:44.259065 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxqjn\" (UniqueName: \"kubernetes.io/projected/e4979818-bca0-49e1-aebd-bdc7bdfafe3e-kube-api-access-vxqjn\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:54:44.259160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:44.259091 4442 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e4979818-bca0-49e1-aebd-bdc7bdfafe3e-routing-overlay-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:54:44.988120 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:44.988083 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789f79ngk" Oct 30 18:54:45.076824 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:45.076786 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789f79ngk"] Oct 30 18:54:45.076824 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:45.076824 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789f79ngk"] Oct 30 18:54:45.174601 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:45.174565 4442 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e4979818-bca0-49e1-aebd-bdc7bdfafe3e-provider-credentials\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:54:47.071491 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:47.071454 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4979818-bca0-49e1-aebd-bdc7bdfafe3e" path="/var/lib/kubelet/pods/e4979818-bca0-49e1-aebd-bdc7bdfafe3e/volumes" Oct 30 18:54:47.186455 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:47.186417 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475wp9ng" podUID="f23f60cb-e0e8-492b-93c4-8a04ff1f3ace" containerName="payload-processing" containerID="cri-o://96e0169ffbbbc632f40f1b154549f23214158207707cac5376eaf92a3b1df38e" gracePeriod=25 Oct 30 18:54:47.231999 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:47.231956 4442 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-176.ec2.internal" podUID="9e81bd78300430e51b01d30caa1123e7" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:54:47.492027 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:47.491999 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475wp9ng" Oct 30 18:54:47.572474 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:54:47.572435 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:54:47.639019 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:47.638923 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f23f60cb-e0e8-492b-93c4-8a04ff1f3ace-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f23f60cb-e0e8-492b-93c4-8a04ff1f3ace-routing-overlay-volume\") pod \"f23f60cb-e0e8-492b-93c4-8a04ff1f3ace\" (UID: \"f23f60cb-e0e8-492b-93c4-8a04ff1f3ace\") " Oct 30 18:54:47.639019 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:47.638998 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f23f60cb-e0e8-492b-93c4-8a04ff1f3ace-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f23f60cb-e0e8-492b-93c4-8a04ff1f3ace-provider-credentials\") pod \"f23f60cb-e0e8-492b-93c4-8a04ff1f3ace\" (UID: \"f23f60cb-e0e8-492b-93c4-8a04ff1f3ace\") " Oct 30 18:54:47.639297 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:47.639092 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f23f60cb-e0e8-492b-93c4-8a04ff1f3ace-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f23f60cb-e0e8-492b-93c4-8a04ff1f3ace-plugins-config-volume\") pod \"f23f60cb-e0e8-492b-93c4-8a04ff1f3ace\" (UID: \"f23f60cb-e0e8-492b-93c4-8a04ff1f3ace\") " Oct 30 18:54:47.639358 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:47.639317 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f23f60cb-e0e8-492b-93c4-8a04ff1f3ace-routing-overlay-volume" pod "f23f60cb-e0e8-492b-93c4-8a04ff1f3ace" (UID: "f23f60cb-e0e8-492b-93c4-8a04ff1f3ace"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:54:47.639446 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:47.639429 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f23f60cb-e0e8-492b-93c4-8a04ff1f3ace-kube-api-access-99mlv\" (UniqueName: \"kubernetes.io/projected/f23f60cb-e0e8-492b-93c4-8a04ff1f3ace-kube-api-access-99mlv\") pod \"f23f60cb-e0e8-492b-93c4-8a04ff1f3ace\" (UID: \"f23f60cb-e0e8-492b-93c4-8a04ff1f3ace\") " Oct 30 18:54:47.639565 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:47.639540 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f23f60cb-e0e8-492b-93c4-8a04ff1f3ace-plugins-config-volume" pod "f23f60cb-e0e8-492b-93c4-8a04ff1f3ace" (UID: "f23f60cb-e0e8-492b-93c4-8a04ff1f3ace"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:54:47.640385 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:47.640359 4442 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f23f60cb-e0e8-492b-93c4-8a04ff1f3ace-routing-overlay-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:54:47.640517 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:47.640394 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f23f60cb-e0e8-492b-93c4-8a04ff1f3ace-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:54:47.641606 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:47.641584 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f23f60cb-e0e8-492b-93c4-8a04ff1f3ace-provider-credentials" pod "f23f60cb-e0e8-492b-93c4-8a04ff1f3ace" (UID: "f23f60cb-e0e8-492b-93c4-8a04ff1f3ace"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:54:47.642303 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:47.642271 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f23f60cb-e0e8-492b-93c4-8a04ff1f3ace-kube-api-access-99mlv" pod "f23f60cb-e0e8-492b-93c4-8a04ff1f3ace" (UID: "f23f60cb-e0e8-492b-93c4-8a04ff1f3ace"). InnerVolumeSpecName "kube-api-access-99mlv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:54:47.747154 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:47.743955 4442 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f23f60cb-e0e8-492b-93c4-8a04ff1f3ace-provider-credentials\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:54:47.747154 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:47.744003 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99mlv\" (UniqueName: \"kubernetes.io/projected/f23f60cb-e0e8-492b-93c4-8a04ff1f3ace-kube-api-access-99mlv\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:54:47.965757 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:47.965658 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdmbkvd" podUID="a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864" containerName="payload-processing" containerID="cri-o://32487b080a963bf47626781d8ee4810f46bdd5a5fa3d5457a8f6ebd45dc9e2bd" gracePeriod=25 Oct 30 18:54:48.089280 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:48.089240 4442 generic.go:356] "Generic (PLEG): container finished" podID="f23f60cb-e0e8-492b-93c4-8a04ff1f3ace" containerID="96e0169ffbbbc632f40f1b154549f23214158207707cac5376eaf92a3b1df38e" exitCode=0 Oct 30 18:54:48.089280 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:48.089282 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475wp9ng" Oct 30 18:54:48.089799 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:48.089327 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475wp9ng" event={"ID":"f23f60cb-e0e8-492b-93c4-8a04ff1f3ace","Type":"ContainerDied","Data":"96e0169ffbbbc632f40f1b154549f23214158207707cac5376eaf92a3b1df38e"} Oct 30 18:54:48.089799 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:48.089382 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475wp9ng" event={"ID":"f23f60cb-e0e8-492b-93c4-8a04ff1f3ace","Type":"ContainerDied","Data":"1ad3d0e938b52bc21564a99442426346d1109c30b0d94003b197f9bb586f7e05"} Oct 30 18:54:48.089799 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:48.089409 4442 scope.go:122] "RemoveContainer" containerID="96e0169ffbbbc632f40f1b154549f23214158207707cac5376eaf92a3b1df38e" Oct 30 18:54:48.091849 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:48.091821 4442 generic.go:356] "Generic (PLEG): container finished" podID="a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864" containerID="32487b080a963bf47626781d8ee4810f46bdd5a5fa3d5457a8f6ebd45dc9e2bd" exitCode=0 Oct 30 18:54:48.092283 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:48.092128 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdmbkvd" event={"ID":"a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864","Type":"ContainerDied","Data":"32487b080a963bf47626781d8ee4810f46bdd5a5fa3d5457a8f6ebd45dc9e2bd"} Oct 30 18:54:48.106673 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:48.106653 4442 scope.go:122] "RemoveContainer" containerID="96e0169ffbbbc632f40f1b154549f23214158207707cac5376eaf92a3b1df38e" Oct 30 18:54:48.107697 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:54:48.106951 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96e0169ffbbbc632f40f1b154549f23214158207707cac5376eaf92a3b1df38e\": container with ID starting with 96e0169ffbbbc632f40f1b154549f23214158207707cac5376eaf92a3b1df38e not found: ID does not exist" containerID="96e0169ffbbbc632f40f1b154549f23214158207707cac5376eaf92a3b1df38e" Oct 30 18:54:48.107697 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:48.106978 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96e0169ffbbbc632f40f1b154549f23214158207707cac5376eaf92a3b1df38e"} err="failed to get container status \"96e0169ffbbbc632f40f1b154549f23214158207707cac5376eaf92a3b1df38e\": rpc error: code = NotFound desc = could not find container \"96e0169ffbbbc632f40f1b154549f23214158207707cac5376eaf92a3b1df38e\": container with ID starting with 96e0169ffbbbc632f40f1b154549f23214158207707cac5376eaf92a3b1df38e not found: ID does not exist" Oct 30 18:54:48.146926 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:48.146897 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475wp9ng"] Oct 30 18:54:48.168954 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:48.165504 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475wp9ng"] Oct 30 18:54:48.288071 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:48.288039 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdmbkvd" Oct 30 18:54:48.371914 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:48.358727 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864-kube-api-access-fhtpq\" (UniqueName: \"kubernetes.io/projected/a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864-kube-api-access-fhtpq\") pod \"a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864\" (UID: \"a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864\") " Oct 30 18:54:48.372180 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:48.372098 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864-routing-overlay-volume\") pod \"a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864\" (UID: \"a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864\") " Oct 30 18:54:48.372322 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:48.372300 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864-plugins-config-volume\") pod \"a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864\" (UID: \"a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864\") " Oct 30 18:54:48.372432 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:48.372414 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864-provider-credentials\") pod \"a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864\" (UID: \"a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864\") " Oct 30 18:54:48.372883 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:48.372854 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864-routing-overlay-volume" pod "a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864" (UID: "a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:54:48.373657 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:48.373629 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864-plugins-config-volume" pod "a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864" (UID: "a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:54:48.373833 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:48.373803 4442 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864-routing-overlay-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:54:48.376347 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:48.376310 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864-kube-api-access-fhtpq" pod "a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864" (UID: "a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864"). InnerVolumeSpecName "kube-api-access-fhtpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:54:48.378183 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:48.376478 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864-provider-credentials" pod "a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864" (UID: "a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:54:48.475378 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:48.475339 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:54:48.475378 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:48.475368 4442 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864-provider-credentials\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:54:48.475378 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:48.475379 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhtpq\" (UniqueName: \"kubernetes.io/projected/a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864-kube-api-access-fhtpq\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:54:49.071957 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:49.071917 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f23f60cb-e0e8-492b-93c4-8a04ff1f3ace" path="/var/lib/kubelet/pods/f23f60cb-e0e8-492b-93c4-8a04ff1f3ace/volumes" Oct 30 18:54:49.103576 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:49.103549 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdmbkvd" Oct 30 18:54:49.104120 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:49.104086 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdmbkvd" event={"ID":"a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864","Type":"ContainerDied","Data":"9638faf3ddbcce77ef6c5c9ca1b624fb01ecc42347211276e64428f339cf3ba2"} Oct 30 18:54:49.104280 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:49.104216 4442 scope.go:122] "RemoveContainer" containerID="32487b080a963bf47626781d8ee4810f46bdd5a5fa3d5457a8f6ebd45dc9e2bd" Oct 30 18:54:49.161573 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:49.161544 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdmbkvd"] Oct 30 18:54:49.183513 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:49.183488 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdmbkvd"] Oct 30 18:54:51.074339 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:51.074302 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864" path="/var/lib/kubelet/pods/a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864/volumes" Oct 30 18:54:53.190764 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:53.190724 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9"] Oct 30 18:54:53.191849 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:53.191827 4442 cpu_manager.go:414] "containerMap: removing container" podUID="a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864" containerName="payload-processing" Oct 30 18:54:53.191849 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:53.191848 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864" containerName="payload-processing" Oct 30 18:54:53.192014 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:53.191874 4442 cpu_manager.go:414] "containerMap: removing container" podUID="f23f60cb-e0e8-492b-93c4-8a04ff1f3ace" containerName="payload-processing" Oct 30 18:54:53.192014 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:53.191880 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f23f60cb-e0e8-492b-93c4-8a04ff1f3ace" containerName="payload-processing" Oct 30 18:54:53.192014 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:53.191995 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2abfeaa-13f4-4e3e-8d8a-b3d535d9d864" containerName="payload-processing" Oct 30 18:54:53.192014 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:53.192007 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="f23f60cb-e0e8-492b-93c4-8a04ff1f3ace" containerName="payload-processing" Oct 30 18:54:53.203182 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:53.196621 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" Oct 30 18:54:53.216907 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:53.216871 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9"] Oct 30 18:54:53.270889 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:53.270851 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/553c4b32-dcd3-494b-9cfc-5014c97c1543-istiod-ca-cert\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9\" (UID: \"553c4b32-dcd3-494b-9cfc-5014c97c1543\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" Oct 30 18:54:53.271094 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:53.270905 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/553c4b32-dcd3-494b-9cfc-5014c97c1543-istio-envoy\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9\" (UID: \"553c4b32-dcd3-494b-9cfc-5014c97c1543\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" Oct 30 18:54:53.271094 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:53.271037 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/553c4b32-dcd3-494b-9cfc-5014c97c1543-workload-socket\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9\" (UID: \"553c4b32-dcd3-494b-9cfc-5014c97c1543\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" Oct 30 18:54:53.271245 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:53.271092 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/553c4b32-dcd3-494b-9cfc-5014c97c1543-workload-certs\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9\" (UID: \"553c4b32-dcd3-494b-9cfc-5014c97c1543\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" Oct 30 18:54:53.271245 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:53.271205 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/553c4b32-dcd3-494b-9cfc-5014c97c1543-istio-data\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9\" (UID: \"553c4b32-dcd3-494b-9cfc-5014c97c1543\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" Oct 30 18:54:53.271381 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:53.271257 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/553c4b32-dcd3-494b-9cfc-5014c97c1543-credential-socket\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9\" (UID: \"553c4b32-dcd3-494b-9cfc-5014c97c1543\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" Oct 30 18:54:53.271381 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:53.271296 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/553c4b32-dcd3-494b-9cfc-5014c97c1543-istio-token\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9\" (UID: \"553c4b32-dcd3-494b-9cfc-5014c97c1543\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" Oct 30 18:54:53.271381 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:53.271313 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/553c4b32-dcd3-494b-9cfc-5014c97c1543-istio-podinfo\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9\" (UID: \"553c4b32-dcd3-494b-9cfc-5014c97c1543\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" Oct 30 18:54:53.271381 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:53.271353 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnv5t\" (UniqueName: \"kubernetes.io/projected/553c4b32-dcd3-494b-9cfc-5014c97c1543-kube-api-access-gnv5t\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9\" (UID: \"553c4b32-dcd3-494b-9cfc-5014c97c1543\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" Oct 30 18:54:53.372360 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:53.372319 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/553c4b32-dcd3-494b-9cfc-5014c97c1543-workload-certs\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9\" (UID: \"553c4b32-dcd3-494b-9cfc-5014c97c1543\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" Oct 30 18:54:53.372571 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:53.372400 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/553c4b32-dcd3-494b-9cfc-5014c97c1543-istio-data\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9\" (UID: \"553c4b32-dcd3-494b-9cfc-5014c97c1543\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" Oct 30 18:54:53.372571 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:53.372454 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/553c4b32-dcd3-494b-9cfc-5014c97c1543-credential-socket\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9\" (UID: \"553c4b32-dcd3-494b-9cfc-5014c97c1543\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" Oct 30 18:54:53.372571 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:53.372487 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/553c4b32-dcd3-494b-9cfc-5014c97c1543-istio-token\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9\" (UID: \"553c4b32-dcd3-494b-9cfc-5014c97c1543\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" Oct 30 18:54:53.372571 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:53.372505 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/553c4b32-dcd3-494b-9cfc-5014c97c1543-istio-podinfo\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9\" (UID: \"553c4b32-dcd3-494b-9cfc-5014c97c1543\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" Oct 30 18:54:53.372571 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:53.372546 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnv5t\" (UniqueName: \"kubernetes.io/projected/553c4b32-dcd3-494b-9cfc-5014c97c1543-kube-api-access-gnv5t\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9\" (UID: \"553c4b32-dcd3-494b-9cfc-5014c97c1543\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" Oct 30 18:54:53.372933 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:53.372579 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/553c4b32-dcd3-494b-9cfc-5014c97c1543-istiod-ca-cert\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9\" (UID: \"553c4b32-dcd3-494b-9cfc-5014c97c1543\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" Oct 30 18:54:53.372933 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:53.372627 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/553c4b32-dcd3-494b-9cfc-5014c97c1543-istio-envoy\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9\" (UID: \"553c4b32-dcd3-494b-9cfc-5014c97c1543\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" Oct 30 18:54:53.372933 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:53.372734 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/553c4b32-dcd3-494b-9cfc-5014c97c1543-workload-socket\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9\" (UID: \"553c4b32-dcd3-494b-9cfc-5014c97c1543\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" Oct 30 18:54:53.372933 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:53.372772 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/553c4b32-dcd3-494b-9cfc-5014c97c1543-workload-certs\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9\" (UID: \"553c4b32-dcd3-494b-9cfc-5014c97c1543\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" Oct 30 18:54:53.372933 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:53.372790 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/553c4b32-dcd3-494b-9cfc-5014c97c1543-istio-data\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9\" (UID: \"553c4b32-dcd3-494b-9cfc-5014c97c1543\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" Oct 30 18:54:53.373391 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:53.373370 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/553c4b32-dcd3-494b-9cfc-5014c97c1543-workload-socket\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9\" (UID: \"553c4b32-dcd3-494b-9cfc-5014c97c1543\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" Oct 30 18:54:53.373479 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:53.373437 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/553c4b32-dcd3-494b-9cfc-5014c97c1543-credential-socket\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9\" (UID: \"553c4b32-dcd3-494b-9cfc-5014c97c1543\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" Oct 30 18:54:53.373545 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:53.373483 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/553c4b32-dcd3-494b-9cfc-5014c97c1543-istiod-ca-cert\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9\" (UID: \"553c4b32-dcd3-494b-9cfc-5014c97c1543\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" Oct 30 18:54:53.375480 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:53.375456 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/553c4b32-dcd3-494b-9cfc-5014c97c1543-istio-podinfo\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9\" (UID: \"553c4b32-dcd3-494b-9cfc-5014c97c1543\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" Oct 30 18:54:53.375599 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:53.375515 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/553c4b32-dcd3-494b-9cfc-5014c97c1543-istio-envoy\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9\" (UID: \"553c4b32-dcd3-494b-9cfc-5014c97c1543\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" Oct 30 18:54:53.391653 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:53.391627 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/553c4b32-dcd3-494b-9cfc-5014c97c1543-istio-token\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9\" (UID: \"553c4b32-dcd3-494b-9cfc-5014c97c1543\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" Oct 30 18:54:53.392175 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:53.392150 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnv5t\" (UniqueName: \"kubernetes.io/projected/553c4b32-dcd3-494b-9cfc-5014c97c1543-kube-api-access-gnv5t\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9\" (UID: \"553c4b32-dcd3-494b-9cfc-5014c97c1543\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" Oct 30 18:54:53.521870 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:53.521790 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" Oct 30 18:54:53.863736 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:54:53.863697 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod553c4b32_dcd3_494b_9cfc_5014c97c1543.slice/crio-de5eff019c3a5ebe3d86d33844c535d6a0ffc18b3a758cc46a6d30f29250d555 WatchSource:0}: Error finding container de5eff019c3a5ebe3d86d33844c535d6a0ffc18b3a758cc46a6d30f29250d555: Status 404 returned error can't find the container with id de5eff019c3a5ebe3d86d33844c535d6a0ffc18b3a758cc46a6d30f29250d555 Oct 30 18:54:53.867208 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:53.867180 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9"] Oct 30 18:54:53.867683 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:53.867642 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:54:53.867754 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:53.867735 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:54:53.867798 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:53.867778 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:54:54.202316 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:54.202280 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" event={"ID":"553c4b32-dcd3-494b-9cfc-5014c97c1543","Type":"ContainerStarted","Data":"715472ff24cafa7875efa47b7a4f983a1354fb73767e9e4ee3d41dd4eff6f75a"} Oct 30 18:54:54.202715 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:54.202334 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" event={"ID":"553c4b32-dcd3-494b-9cfc-5014c97c1543","Type":"ContainerStarted","Data":"de5eff019c3a5ebe3d86d33844c535d6a0ffc18b3a758cc46a6d30f29250d555"} Oct 30 18:54:54.267971 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:54.267908 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" podStartSLOduration=1.267892216 podStartE2EDuration="1.267892216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:54.254991298 +0000 UTC m=+3391.345053660" watchObservedRunningTime="2026-10-30 18:54:54.267892216 +0000 UTC m=+3391.357954555" Oct 30 18:54:54.522763 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:54.522642 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" Oct 30 18:54:54.524867 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:54.524830 4442 patch_prober.go:29] interesting pod/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" start-of-body= Oct 30 18:54:54.525034 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:54.524893 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" podUID="553c4b32-dcd3-494b-9cfc-5014c97c1543" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" Oct 30 18:54:55.522558 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:55.522522 4442 patch_prober.go:29] interesting pod/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" start-of-body= Oct 30 18:54:55.522980 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:55.522617 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" podUID="553c4b32-dcd3-494b-9cfc-5014c97c1543" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" Oct 30 18:54:56.530170 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:56.529533 4442 patch_prober.go:29] interesting pod/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" start-of-body= Oct 30 18:54:56.530170 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:56.529592 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" podUID="553c4b32-dcd3-494b-9cfc-5014c97c1543" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" Oct 30 18:54:57.523156 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:57.523094 4442 patch_prober.go:29] interesting pod/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" start-of-body= Oct 30 18:54:57.523366 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:57.523205 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" podUID="553c4b32-dcd3-494b-9cfc-5014c97c1543" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" Oct 30 18:54:58.522537 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:58.522501 4442 patch_prober.go:29] interesting pod/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" start-of-body= Oct 30 18:54:58.522981 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:58.522570 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" podUID="553c4b32-dcd3-494b-9cfc-5014c97c1543" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" Oct 30 18:54:59.522716 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:59.522670 4442 patch_prober.go:29] interesting pod/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" start-of-body= Oct 30 18:54:59.523157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:59.522750 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" podUID="553c4b32-dcd3-494b-9cfc-5014c97c1543" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" Oct 30 18:54:59.676908 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:59.676875 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw"] Oct 30 18:54:59.686663 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:59.683446 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw"] Oct 30 18:54:59.686663 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:59.683565 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" Oct 30 18:54:59.700450 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:59.700423 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-869b55-openshift-default-dockercfg-l4fqf\"" Oct 30 18:54:59.755442 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:59.755404 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29ce0e02-4041-4ecb-bb5f-c885099d00b7-istiod-ca-cert\") pod \"e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw\" (UID: \"29ce0e02-4041-4ecb-bb5f-c885099d00b7\") " pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" Oct 30 18:54:59.755641 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:59.755483 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29ce0e02-4041-4ecb-bb5f-c885099d00b7-istio-data\") pod \"e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw\" (UID: \"29ce0e02-4041-4ecb-bb5f-c885099d00b7\") " pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" Oct 30 18:54:59.755641 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:59.755559 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29ce0e02-4041-4ecb-bb5f-c885099d00b7-istio-envoy\") pod \"e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw\" (UID: \"29ce0e02-4041-4ecb-bb5f-c885099d00b7\") " pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" Oct 30 18:54:59.755641 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:59.755611 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdffg\" (UniqueName: \"kubernetes.io/projected/29ce0e02-4041-4ecb-bb5f-c885099d00b7-kube-api-access-sdffg\") pod \"e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw\" (UID: \"29ce0e02-4041-4ecb-bb5f-c885099d00b7\") " pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" Oct 30 18:54:59.755783 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:59.755651 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29ce0e02-4041-4ecb-bb5f-c885099d00b7-istio-podinfo\") pod \"e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw\" (UID: \"29ce0e02-4041-4ecb-bb5f-c885099d00b7\") " pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" Oct 30 18:54:59.755836 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:59.755778 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29ce0e02-4041-4ecb-bb5f-c885099d00b7-workload-certs\") pod \"e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw\" (UID: \"29ce0e02-4041-4ecb-bb5f-c885099d00b7\") " pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" Oct 30 18:54:59.755908 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:59.755880 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29ce0e02-4041-4ecb-bb5f-c885099d00b7-istio-token\") pod \"e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw\" (UID: \"29ce0e02-4041-4ecb-bb5f-c885099d00b7\") " pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" Oct 30 18:54:59.756043 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:59.755970 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ce0e02-4041-4ecb-bb5f-c885099d00b7-credential-socket\") pod \"e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw\" (UID: \"29ce0e02-4041-4ecb-bb5f-c885099d00b7\") " pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" Oct 30 18:54:59.756110 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:59.756095 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ce0e02-4041-4ecb-bb5f-c885099d00b7-workload-socket\") pod \"e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw\" (UID: \"29ce0e02-4041-4ecb-bb5f-c885099d00b7\") " pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" Oct 30 18:54:59.871223 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:59.867414 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29ce0e02-4041-4ecb-bb5f-c885099d00b7-istio-envoy\") pod \"e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw\" (UID: \"29ce0e02-4041-4ecb-bb5f-c885099d00b7\") " pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" Oct 30 18:54:59.871223 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:59.867490 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdffg\" (UniqueName: \"kubernetes.io/projected/29ce0e02-4041-4ecb-bb5f-c885099d00b7-kube-api-access-sdffg\") pod \"e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw\" (UID: \"29ce0e02-4041-4ecb-bb5f-c885099d00b7\") " pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" Oct 30 18:54:59.871223 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:59.867531 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29ce0e02-4041-4ecb-bb5f-c885099d00b7-istio-podinfo\") pod \"e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw\" (UID: \"29ce0e02-4041-4ecb-bb5f-c885099d00b7\") " pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" Oct 30 18:54:59.871223 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:59.867653 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29ce0e02-4041-4ecb-bb5f-c885099d00b7-workload-certs\") pod \"e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw\" (UID: \"29ce0e02-4041-4ecb-bb5f-c885099d00b7\") " pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" Oct 30 18:54:59.871223 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:59.867749 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29ce0e02-4041-4ecb-bb5f-c885099d00b7-istio-token\") pod \"e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw\" (UID: \"29ce0e02-4041-4ecb-bb5f-c885099d00b7\") " pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" Oct 30 18:54:59.871223 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:59.867826 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ce0e02-4041-4ecb-bb5f-c885099d00b7-credential-socket\") pod \"e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw\" (UID: \"29ce0e02-4041-4ecb-bb5f-c885099d00b7\") " pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" Oct 30 18:54:59.871223 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:59.867930 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ce0e02-4041-4ecb-bb5f-c885099d00b7-workload-socket\") pod \"e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw\" (UID: \"29ce0e02-4041-4ecb-bb5f-c885099d00b7\") " pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" Oct 30 18:54:59.871223 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:59.868014 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29ce0e02-4041-4ecb-bb5f-c885099d00b7-istiod-ca-cert\") pod \"e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw\" (UID: \"29ce0e02-4041-4ecb-bb5f-c885099d00b7\") " pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" Oct 30 18:54:59.871223 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:59.868071 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29ce0e02-4041-4ecb-bb5f-c885099d00b7-istio-data\") pod \"e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw\" (UID: \"29ce0e02-4041-4ecb-bb5f-c885099d00b7\") " pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" Oct 30 18:54:59.899155 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:59.893628 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29ce0e02-4041-4ecb-bb5f-c885099d00b7-workload-certs\") pod \"e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw\" (UID: \"29ce0e02-4041-4ecb-bb5f-c885099d00b7\") " pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" Oct 30 18:54:59.899155 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:59.896044 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29ce0e02-4041-4ecb-bb5f-c885099d00b7-istio-data\") pod \"e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw\" (UID: \"29ce0e02-4041-4ecb-bb5f-c885099d00b7\") " pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" Oct 30 18:54:59.899155 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:59.896269 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ce0e02-4041-4ecb-bb5f-c885099d00b7-credential-socket\") pod \"e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw\" (UID: \"29ce0e02-4041-4ecb-bb5f-c885099d00b7\") " pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" Oct 30 18:54:59.899155 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:59.896501 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ce0e02-4041-4ecb-bb5f-c885099d00b7-workload-socket\") pod \"e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw\" (UID: \"29ce0e02-4041-4ecb-bb5f-c885099d00b7\") " pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" Oct 30 18:54:59.899155 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:59.897816 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29ce0e02-4041-4ecb-bb5f-c885099d00b7-istiod-ca-cert\") pod \"e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw\" (UID: \"29ce0e02-4041-4ecb-bb5f-c885099d00b7\") " pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" Oct 30 18:54:59.949155 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:59.935067 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdffg\" (UniqueName: \"kubernetes.io/projected/29ce0e02-4041-4ecb-bb5f-c885099d00b7-kube-api-access-sdffg\") pod \"e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw\" (UID: \"29ce0e02-4041-4ecb-bb5f-c885099d00b7\") " pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" Oct 30 18:54:59.949155 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:59.943776 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29ce0e02-4041-4ecb-bb5f-c885099d00b7-istio-podinfo\") pod \"e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw\" (UID: \"29ce0e02-4041-4ecb-bb5f-c885099d00b7\") " pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" Oct 30 18:54:59.949155 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:59.944424 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29ce0e02-4041-4ecb-bb5f-c885099d00b7-istio-token\") pod \"e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw\" (UID: \"29ce0e02-4041-4ecb-bb5f-c885099d00b7\") " pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" Oct 30 18:54:59.972158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:54:59.959059 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29ce0e02-4041-4ecb-bb5f-c885099d00b7-istio-envoy\") pod \"e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw\" (UID: \"29ce0e02-4041-4ecb-bb5f-c885099d00b7\") " pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" Oct 30 18:55:00.016647 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:00.015219 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" Oct 30 18:55:00.523099 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:00.523016 4442 patch_prober.go:29] interesting pod/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" start-of-body= Oct 30 18:55:00.523099 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:00.523064 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" podUID="553c4b32-dcd3-494b-9cfc-5014c97c1543" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" Oct 30 18:55:00.655782 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:00.655684 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw"] Oct 30 18:55:00.659296 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:00.659254 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:55:00.659434 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:00.659351 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:55:00.659434 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:00.659395 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:55:01.355522 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:01.355485 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" event={"ID":"29ce0e02-4041-4ecb-bb5f-c885099d00b7","Type":"ContainerStarted","Data":"71f4dbf20f87947e8cfbd5fcbf4435ae557edb9a8619c06379cd334fcc223379"} Oct 30 18:55:01.355722 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:01.355543 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" event={"ID":"29ce0e02-4041-4ecb-bb5f-c885099d00b7","Type":"ContainerStarted","Data":"5864beb8eed8f9e9fd71d32e77b649823cd77a4dc05277bfeda807e0ae9923f8"} Oct 30 18:55:01.401334 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:01.401285 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" podStartSLOduration=2.4012721040000002 podStartE2EDuration="2.401272104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:01.396364343 +0000 UTC m=+3398.486426683" watchObservedRunningTime="2026-10-30 18:55:01.401272104 +0000 UTC m=+3398.491334443" Oct 30 18:55:01.607860 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:01.607741 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" Oct 30 18:55:01.609890 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:01.609861 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" Oct 30 18:55:01.612653 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:01.612626 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" Oct 30 18:55:02.016356 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:02.016318 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" Oct 30 18:55:02.024302 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:02.024270 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" Oct 30 18:55:02.366678 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:02.366636 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" Oct 30 18:55:02.367657 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:02.367631 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" Oct 30 18:55:05.745450 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:05.745417 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-869b55-b745cd4f5-7smvc"] Oct 30 18:55:05.751228 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:05.751205 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-869b55-b745cd4f5-7smvc" Oct 30 18:55:05.771045 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:05.771000 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-869b55-b745cd4f5-7smvc"] Oct 30 18:55:05.773668 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:05.773604 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-869b55\"" Oct 30 18:55:05.773785 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:05.773703 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-869b55-dockercfg-llldk\"" Oct 30 18:55:05.829158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:05.824376 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-756zj\" (UniqueName: \"kubernetes.io/projected/0709501c-2bbd-4926-8679-0e65a78e09ac-kube-api-access-756zj\") pod \"payload-pre-processing-e2e-inf-a-869b55-b745cd4f5-7smvc\" (UID: \"0709501c-2bbd-4926-8679-0e65a78e09ac\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-869b55-b745cd4f5-7smvc" Oct 30 18:55:05.829158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:05.824760 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0709501c-2bbd-4926-8679-0e65a78e09ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-869b55-b745cd4f5-7smvc\" (UID: \"0709501c-2bbd-4926-8679-0e65a78e09ac\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-869b55-b745cd4f5-7smvc" Oct 30 18:55:05.957226 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:05.954447 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-756zj\" (UniqueName: \"kubernetes.io/projected/0709501c-2bbd-4926-8679-0e65a78e09ac-kube-api-access-756zj\") pod \"payload-pre-processing-e2e-inf-a-869b55-b745cd4f5-7smvc\" (UID: \"0709501c-2bbd-4926-8679-0e65a78e09ac\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-869b55-b745cd4f5-7smvc" Oct 30 18:55:05.957226 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:05.954602 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0709501c-2bbd-4926-8679-0e65a78e09ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-869b55-b745cd4f5-7smvc\" (UID: \"0709501c-2bbd-4926-8679-0e65a78e09ac\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-869b55-b745cd4f5-7smvc" Oct 30 18:55:05.957226 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:05.955756 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0709501c-2bbd-4926-8679-0e65a78e09ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-869b55-b745cd4f5-7smvc\" (UID: \"0709501c-2bbd-4926-8679-0e65a78e09ac\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-869b55-b745cd4f5-7smvc" Oct 30 18:55:06.208162 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:06.207925 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-756zj\" (UniqueName: \"kubernetes.io/projected/0709501c-2bbd-4926-8679-0e65a78e09ac-kube-api-access-756zj\") pod \"payload-pre-processing-e2e-inf-a-869b55-b745cd4f5-7smvc\" (UID: \"0709501c-2bbd-4926-8679-0e65a78e09ac\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-869b55-b745cd4f5-7smvc" Oct 30 18:55:06.265017 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:06.264733 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-869b55-b745cd4f5-7smvc" Oct 30 18:55:06.707368 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:06.706412 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-869b55-b745cd4f5-7smvc"] Oct 30 18:55:07.467384 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:07.467349 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-869b55-b745cd4f5-7smvc" event={"ID":"0709501c-2bbd-4926-8679-0e65a78e09ac","Type":"ContainerStarted","Data":"908cb5d05e08bba2a38762ed63e2737dea0985078888c26d9f7fbe407bb312e7"} Oct 30 18:55:07.467833 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:07.467418 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-869b55-b745cd4f5-7smvc" event={"ID":"0709501c-2bbd-4926-8679-0e65a78e09ac","Type":"ContainerStarted","Data":"f05e2872002277c537b2dc811f1fa329a662680521aa27428edb20a473382b5a"} Oct 30 18:55:07.467833 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:07.467451 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-869b55-b745cd4f5-7smvc" Oct 30 18:55:07.504827 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:07.504772 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-869b55-b745cd4f5-7smvc" podStartSLOduration=2.504758242 podStartE2EDuration="2.504758242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:55:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:07.504208929 +0000 UTC m=+3404.594271269" watchObservedRunningTime="2026-10-30 18:55:07.504758242 +0000 UTC m=+3404.594820580" Oct 30 18:55:17.588938 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:55:17.588893 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:55:18.485110 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:18.485021 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-869b55-b745cd4f5-7smvc" Oct 30 18:55:21.964395 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:21.964351 4442 patch_prober.go:29] interesting pod/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:55:21.965020 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:21.964990 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" podUID="6d8d84cf-1334-4038-8b24-8cbec03e6882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:55:23.695563 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:23.695530 4442 generic.go:356] "Generic (PLEG): container finished" podID="6d8d84cf-1334-4038-8b24-8cbec03e6882" containerID="7ffd7490b3dcbf7b355af0ac00e48d063fa223293713c5445ac06795e4a88324" exitCode=0 Oct 30 18:55:23.696023 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:23.695602 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" event={"ID":"6d8d84cf-1334-4038-8b24-8cbec03e6882","Type":"ContainerDied","Data":"7ffd7490b3dcbf7b355af0ac00e48d063fa223293713c5445ac06795e4a88324"} Oct 30 18:55:23.696239 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:23.696222 4442 scope.go:122] "RemoveContainer" containerID="7ffd7490b3dcbf7b355af0ac00e48d063fa223293713c5445ac06795e4a88324" Oct 30 18:55:23.698651 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:23.698621 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:55:23.698761 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:23.698722 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:55:23.698761 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:23.698758 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:55:24.706980 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:24.706945 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" event={"ID":"6d8d84cf-1334-4038-8b24-8cbec03e6882","Type":"ContainerStarted","Data":"a7bb5aa4b35ee7536b17eac0e1f9d3bccf57f8b470396a4befb164ad7965c24e"} Oct 30 18:55:25.356382 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:25.356349 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" Oct 30 18:55:25.356675 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:25.356651 4442 patch_prober.go:29] interesting pod/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Oct 30 18:55:25.356786 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:25.356708 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" podUID="6d8d84cf-1334-4038-8b24-8cbec03e6882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Oct 30 18:55:26.344585 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:26.344548 4442 patch_prober.go:29] interesting pod/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Oct 30 18:55:26.373187 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:26.373126 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" podUID="6d8d84cf-1334-4038-8b24-8cbec03e6882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Oct 30 18:55:27.345601 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:27.345515 4442 patch_prober.go:29] interesting pod/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Oct 30 18:55:27.346188 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:27.346156 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" podUID="6d8d84cf-1334-4038-8b24-8cbec03e6882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Oct 30 18:55:28.348160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:28.347702 4442 patch_prober.go:29] interesting pod/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Oct 30 18:55:28.348160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:28.347780 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" podUID="6d8d84cf-1334-4038-8b24-8cbec03e6882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Oct 30 18:55:29.350614 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:29.346538 4442 patch_prober.go:29] interesting pod/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Oct 30 18:55:29.350614 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:29.346608 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" podUID="6d8d84cf-1334-4038-8b24-8cbec03e6882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Oct 30 18:55:30.347608 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:30.347566 4442 patch_prober.go:29] interesting pod/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Oct 30 18:55:30.347911 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:30.347880 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" podUID="6d8d84cf-1334-4038-8b24-8cbec03e6882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Oct 30 18:55:31.427984 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:31.427783 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" Oct 30 18:55:31.429160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:31.429118 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" Oct 30 18:55:31.430227 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:55:31.430204 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" Oct 30 18:55:47.562326 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:55:47.562286 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:56:17.562679 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:56:17.562590 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:56:39.148988 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:56:39.148945 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t"] Oct 30 18:56:39.155236 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:56:39.155213 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" Oct 30 18:56:39.178484 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:56:39.178454 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t"] Oct 30 18:56:39.190214 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:56:39.190192 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-9f6b7c-openshift-default-dockercfg-mpxqb\"" Oct 30 18:56:39.546035 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:56:39.452696 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-istio-token\") pod \"e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t\" (UID: \"5ba882f0-916a-4651-8ff9-1e8fe8d9e021\") " pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" Oct 30 18:56:39.546360 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:56:39.546331 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-istio-data\") pod \"e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t\" (UID: \"5ba882f0-916a-4651-8ff9-1e8fe8d9e021\") " pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" Oct 30 18:56:39.546538 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:56:39.546519 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-workload-certs\") pod \"e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t\" (UID: \"5ba882f0-916a-4651-8ff9-1e8fe8d9e021\") " pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" Oct 30 18:56:39.550162 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:56:39.546738 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-istio-envoy\") pod \"e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t\" (UID: \"5ba882f0-916a-4651-8ff9-1e8fe8d9e021\") " pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" Oct 30 18:56:39.561294 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:56:39.561262 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-istiod-ca-cert\") pod \"e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t\" (UID: \"5ba882f0-916a-4651-8ff9-1e8fe8d9e021\") " pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" Oct 30 18:56:39.561615 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:56:39.561592 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-workload-socket\") pod \"e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t\" (UID: \"5ba882f0-916a-4651-8ff9-1e8fe8d9e021\") " pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" Oct 30 18:56:39.587159 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:56:39.561785 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmfm5\" (UniqueName: \"kubernetes.io/projected/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-kube-api-access-mmfm5\") pod \"e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t\" (UID: \"5ba882f0-916a-4651-8ff9-1e8fe8d9e021\") " pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" Oct 30 18:56:39.625158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:56:39.620726 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-credential-socket\") pod \"e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t\" (UID: \"5ba882f0-916a-4651-8ff9-1e8fe8d9e021\") " pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" Oct 30 18:56:39.625158 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:56:39.620853 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-istio-podinfo\") pod \"e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t\" (UID: \"5ba882f0-916a-4651-8ff9-1e8fe8d9e021\") " pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" Oct 30 18:56:39.724249 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:56:39.722361 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-credential-socket\") pod \"e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t\" (UID: \"5ba882f0-916a-4651-8ff9-1e8fe8d9e021\") " pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" Oct 30 18:56:39.724249 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:56:39.722453 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-istio-podinfo\") pod \"e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t\" (UID: \"5ba882f0-916a-4651-8ff9-1e8fe8d9e021\") " pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" Oct 30 18:56:39.724249 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:56:39.722667 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-istio-token\") pod \"e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t\" (UID: \"5ba882f0-916a-4651-8ff9-1e8fe8d9e021\") " pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" Oct 30 18:56:39.724249 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:56:39.722693 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-istio-data\") pod \"e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t\" (UID: \"5ba882f0-916a-4651-8ff9-1e8fe8d9e021\") " pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" Oct 30 18:56:39.724249 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:56:39.722741 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-workload-certs\") pod \"e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t\" (UID: \"5ba882f0-916a-4651-8ff9-1e8fe8d9e021\") " pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" Oct 30 18:56:39.724249 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:56:39.722825 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-istio-envoy\") pod \"e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t\" (UID: \"5ba882f0-916a-4651-8ff9-1e8fe8d9e021\") " pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" Oct 30 18:56:39.724249 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:56:39.722851 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-istiod-ca-cert\") pod \"e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t\" (UID: \"5ba882f0-916a-4651-8ff9-1e8fe8d9e021\") " pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" Oct 30 18:56:39.724249 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:56:39.722877 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-workload-socket\") pod \"e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t\" (UID: \"5ba882f0-916a-4651-8ff9-1e8fe8d9e021\") " pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" Oct 30 18:56:39.724249 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:56:39.722911 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmfm5\" (UniqueName: \"kubernetes.io/projected/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-kube-api-access-mmfm5\") pod \"e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t\" (UID: \"5ba882f0-916a-4651-8ff9-1e8fe8d9e021\") " pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" Oct 30 18:56:39.724249 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:56:39.723729 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-credential-socket\") pod \"e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t\" (UID: \"5ba882f0-916a-4651-8ff9-1e8fe8d9e021\") " pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" Oct 30 18:56:39.725369 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:56:39.725344 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-workload-certs\") pod \"e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t\" (UID: \"5ba882f0-916a-4651-8ff9-1e8fe8d9e021\") " pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" Oct 30 18:56:39.728188 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:56:39.726788 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-istiod-ca-cert\") pod \"e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t\" (UID: \"5ba882f0-916a-4651-8ff9-1e8fe8d9e021\") " pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" Oct 30 18:56:39.728395 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:56:39.727465 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-workload-socket\") pod \"e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t\" (UID: \"5ba882f0-916a-4651-8ff9-1e8fe8d9e021\") " pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" Oct 30 18:56:39.728552 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:56:39.728085 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-istio-data\") pod \"e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t\" (UID: \"5ba882f0-916a-4651-8ff9-1e8fe8d9e021\") " pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" Oct 30 18:56:39.749685 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:56:39.749651 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-istio-podinfo\") pod \"e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t\" (UID: \"5ba882f0-916a-4651-8ff9-1e8fe8d9e021\") " pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" Oct 30 18:56:39.750106 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:56:39.750085 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-istio-envoy\") pod \"e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t\" (UID: \"5ba882f0-916a-4651-8ff9-1e8fe8d9e021\") " pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" Oct 30 18:56:39.769900 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:56:39.769865 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmfm5\" (UniqueName: \"kubernetes.io/projected/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-kube-api-access-mmfm5\") pod \"e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t\" (UID: \"5ba882f0-916a-4651-8ff9-1e8fe8d9e021\") " pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" Oct 30 18:56:39.771790 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:56:39.771756 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-istio-token\") pod \"e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t\" (UID: \"5ba882f0-916a-4651-8ff9-1e8fe8d9e021\") " pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" Oct 30 18:56:39.780716 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:56:39.780686 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" Oct 30 18:56:40.131742 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:56:40.131712 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ba882f0_916a_4651_8ff9_1e8fe8d9e021.slice/crio-49df48a5e5b6a9f1bb5119df516589e65afda5cce8d091d59d3e836201aa6296 WatchSource:0}: Error finding container 49df48a5e5b6a9f1bb5119df516589e65afda5cce8d091d59d3e836201aa6296: Status 404 returned error can't find the container with id 49df48a5e5b6a9f1bb5119df516589e65afda5cce8d091d59d3e836201aa6296 Oct 30 18:56:40.134308 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:56:40.134284 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t"] Oct 30 18:56:40.136641 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:56:40.136609 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:56:40.136762 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:56:40.136695 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:56:40.136762 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:56:40.136743 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:56:40.844741 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:56:40.844701 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" event={"ID":"5ba882f0-916a-4651-8ff9-1e8fe8d9e021","Type":"ContainerStarted","Data":"701d8ba2128526b0148de4a9c69ecf43f6126e34c7665e54f82af7f51f3e8fd1"} Oct 30 18:56:40.845201 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:56:40.844778 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" event={"ID":"5ba882f0-916a-4651-8ff9-1e8fe8d9e021","Type":"ContainerStarted","Data":"49df48a5e5b6a9f1bb5119df516589e65afda5cce8d091d59d3e836201aa6296"} Oct 30 18:56:40.914469 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:56:40.914219 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" podStartSLOduration=1.9141989700000002 podStartE2EDuration="1.91419897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:40.899628797 +0000 UTC m=+3497.989691136" watchObservedRunningTime="2026-10-30 18:56:40.91419897 +0000 UTC m=+3498.004261315" Oct 30 18:56:41.782007 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:56:41.781979 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" Oct 30 18:56:41.788410 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:56:41.788382 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" Oct 30 18:56:41.853889 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:56:41.853853 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" Oct 30 18:56:41.854818 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:56:41.854798 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" Oct 30 18:56:47.562274 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:56:47.562236 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:57:17.579532 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:57:17.579491 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:57:23.750909 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:57:23.750872 4442 generic.go:356] "Generic (PLEG): container finished" podID="29ce0e02-4041-4ecb-bb5f-c885099d00b7" containerID="71f4dbf20f87947e8cfbd5fcbf4435ae557edb9a8619c06379cd334fcc223379" exitCode=0 Oct 30 18:57:23.751518 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:57:23.750952 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" event={"ID":"29ce0e02-4041-4ecb-bb5f-c885099d00b7","Type":"ContainerDied","Data":"71f4dbf20f87947e8cfbd5fcbf4435ae557edb9a8619c06379cd334fcc223379"} Oct 30 18:57:23.751518 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:57:23.751446 4442 scope.go:122] "RemoveContainer" containerID="71f4dbf20f87947e8cfbd5fcbf4435ae557edb9a8619c06379cd334fcc223379" Oct 30 18:57:23.768306 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:57:23.768254 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:57:23.770985 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:57:23.770948 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:57:23.771119 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:57:23.771032 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:57:24.774840 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:57:24.774803 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" event={"ID":"29ce0e02-4041-4ecb-bb5f-c885099d00b7","Type":"ContainerStarted","Data":"58b26f522793c6b618d5a220043e9f1e6d9b804468523ef5e80262da8bb8aae1"} Oct 30 18:57:25.016381 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:57:25.016350 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" Oct 30 18:57:25.016603 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:57:25.016584 4442 patch_prober.go:29] interesting pod/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body= Oct 30 18:57:25.016667 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:57:25.016628 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" podUID="29ce0e02-4041-4ecb-bb5f-c885099d00b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" Oct 30 18:57:26.016526 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:57:26.016478 4442 patch_prober.go:29] interesting pod/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body= Oct 30 18:57:26.016996 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:57:26.016544 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" podUID="29ce0e02-4041-4ecb-bb5f-c885099d00b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" Oct 30 18:57:27.016925 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:57:27.016887 4442 patch_prober.go:29] interesting pod/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body= Oct 30 18:57:27.017390 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:57:27.016966 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" podUID="29ce0e02-4041-4ecb-bb5f-c885099d00b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" Oct 30 18:57:28.028157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:57:28.019215 4442 patch_prober.go:29] interesting pod/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body= Oct 30 18:57:28.028157 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:57:28.019292 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" podUID="29ce0e02-4041-4ecb-bb5f-c885099d00b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" Oct 30 18:57:29.016827 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:57:29.016689 4442 patch_prober.go:29] interesting pod/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body= Oct 30 18:57:29.016827 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:57:29.016757 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" podUID="29ce0e02-4041-4ecb-bb5f-c885099d00b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" Oct 30 18:57:30.016815 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:57:30.016781 4442 patch_prober.go:29] interesting pod/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body= Oct 30 18:57:30.017323 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:57:30.016836 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" podUID="29ce0e02-4041-4ecb-bb5f-c885099d00b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" Oct 30 18:57:31.022291 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:57:31.022264 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" Oct 30 18:57:31.022767 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:57:31.022618 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" Oct 30 18:57:31.023458 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:57:31.023433 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" Oct 30 18:57:47.633108 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:57:47.633068 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:58:17.564804 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:58:17.564770 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:58:23.820701 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:58:23.820656 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8" podUID="1e2901e8-4919-4e5c-88c7-39126a7cca56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:58:24.110883 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:58:24.110791 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52rn6_516c6b6c-fc40-49cf-93b5-1c4471b02cf0/ovn-acl-logging/0.log" Oct 30 18:58:24.113714 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:58:24.113692 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52rn6_516c6b6c-fc40-49cf-93b5-1c4471b02cf0/ovn-acl-logging/0.log" Oct 30 18:58:34.467420 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:58:34.467385 4442 generic.go:356] "Generic (PLEG): container finished" podID="5ba882f0-916a-4651-8ff9-1e8fe8d9e021" containerID="701d8ba2128526b0148de4a9c69ecf43f6126e34c7665e54f82af7f51f3e8fd1" exitCode=0 Oct 30 18:58:34.467985 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:58:34.467459 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" event={"ID":"5ba882f0-916a-4651-8ff9-1e8fe8d9e021","Type":"ContainerDied","Data":"701d8ba2128526b0148de4a9c69ecf43f6126e34c7665e54f82af7f51f3e8fd1"} Oct 30 18:58:34.467985 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:58:34.467949 4442 scope.go:122] "RemoveContainer" containerID="701d8ba2128526b0148de4a9c69ecf43f6126e34c7665e54f82af7f51f3e8fd1" Oct 30 18:58:34.470271 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:58:34.470242 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:58:34.470376 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:58:34.470318 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:58:34.470376 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:58:34.470351 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:58:35.482280 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:58:35.482247 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" event={"ID":"5ba882f0-916a-4651-8ff9-1e8fe8d9e021","Type":"ContainerStarted","Data":"bfd655fb9d40b44edbaf88bc1903e789af11b5e5d7f0b773783f48e8b364fe72"} Oct 30 18:58:35.781358 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:58:35.781273 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" Oct 30 18:58:35.781663 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:58:35.781635 4442 patch_prober.go:29] interesting pod/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" start-of-body= Oct 30 18:58:35.781727 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:58:35.781703 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" podUID="5ba882f0-916a-4651-8ff9-1e8fe8d9e021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" Oct 30 18:58:36.782056 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:58:36.782023 4442 patch_prober.go:29] interesting pod/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" start-of-body= Oct 30 18:58:36.782495 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:58:36.782092 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" podUID="5ba882f0-916a-4651-8ff9-1e8fe8d9e021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" Oct 30 18:58:37.782284 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:58:37.782249 4442 patch_prober.go:29] interesting pod/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" start-of-body= Oct 30 18:58:37.782777 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:58:37.782307 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" podUID="5ba882f0-916a-4651-8ff9-1e8fe8d9e021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" Oct 30 18:58:38.781610 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:58:38.781572 4442 patch_prober.go:29] interesting pod/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" start-of-body= Oct 30 18:58:38.781793 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:58:38.781643 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" podUID="5ba882f0-916a-4651-8ff9-1e8fe8d9e021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" Oct 30 18:58:39.785585 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:58:39.785546 4442 patch_prober.go:29] interesting pod/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" start-of-body= Oct 30 18:58:39.786312 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:58:39.786275 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" podUID="5ba882f0-916a-4651-8ff9-1e8fe8d9e021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" Oct 30 18:58:40.782292 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:58:40.782247 4442 patch_prober.go:29] interesting pod/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" start-of-body= Oct 30 18:58:40.782496 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:58:40.782330 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" podUID="5ba882f0-916a-4651-8ff9-1e8fe8d9e021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" Oct 30 18:58:41.787653 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:58:41.787622 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" Oct 30 18:58:41.788073 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:58:41.788051 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" Oct 30 18:58:41.788758 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:58:41.788735 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" Oct 30 18:58:47.561616 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:58:47.561577 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:59:17.563326 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:59:17.563286 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:59:27.973604 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:27.973556 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8"] Oct 30 18:59:27.974199 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:27.973791 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" podUID="e8ac1217-0b8a-4d72-907f-c70814b60a55" containerName="istio-proxy" containerID="cri-o://41add33f7103e2c9fe91e53405c66e78c993031def0b6872ad593dcddd3a88c9" gracePeriod=30 Oct 30 18:59:33.298818 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:33.298788 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" Oct 30 18:59:33.311707 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:33.311671 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8ac1217-0b8a-4d72-907f-c70814b60a55-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8ac1217-0b8a-4d72-907f-c70814b60a55-istiod-ca-cert\") pod \"e8ac1217-0b8a-4d72-907f-c70814b60a55\" (UID: \"e8ac1217-0b8a-4d72-907f-c70814b60a55\") " Oct 30 18:59:33.311875 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:33.311726 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8ac1217-0b8a-4d72-907f-c70814b60a55-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8ac1217-0b8a-4d72-907f-c70814b60a55-workload-certs\") pod \"e8ac1217-0b8a-4d72-907f-c70814b60a55\" (UID: \"e8ac1217-0b8a-4d72-907f-c70814b60a55\") " Oct 30 18:59:33.311875 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:33.311769 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8ac1217-0b8a-4d72-907f-c70814b60a55-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8ac1217-0b8a-4d72-907f-c70814b60a55-workload-socket\") pod \"e8ac1217-0b8a-4d72-907f-c70814b60a55\" (UID: \"e8ac1217-0b8a-4d72-907f-c70814b60a55\") " Oct 30 18:59:33.311875 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:33.311804 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8ac1217-0b8a-4d72-907f-c70814b60a55-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8ac1217-0b8a-4d72-907f-c70814b60a55-istio-envoy\") pod \"e8ac1217-0b8a-4d72-907f-c70814b60a55\" (UID: \"e8ac1217-0b8a-4d72-907f-c70814b60a55\") " Oct 30 18:59:33.312062 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:33.311937 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8ac1217-0b8a-4d72-907f-c70814b60a55-kube-api-access-d4pj8\" (UniqueName: \"kubernetes.io/projected/e8ac1217-0b8a-4d72-907f-c70814b60a55-kube-api-access-d4pj8\") pod \"e8ac1217-0b8a-4d72-907f-c70814b60a55\" (UID: \"e8ac1217-0b8a-4d72-907f-c70814b60a55\") " Oct 30 18:59:33.312062 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:33.311989 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8ac1217-0b8a-4d72-907f-c70814b60a55-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8ac1217-0b8a-4d72-907f-c70814b60a55-istio-data\") pod \"e8ac1217-0b8a-4d72-907f-c70814b60a55\" (UID: \"e8ac1217-0b8a-4d72-907f-c70814b60a55\") " Oct 30 18:59:33.312062 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:33.312027 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8ac1217-0b8a-4d72-907f-c70814b60a55-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8ac1217-0b8a-4d72-907f-c70814b60a55-istio-podinfo\") pod \"e8ac1217-0b8a-4d72-907f-c70814b60a55\" (UID: \"e8ac1217-0b8a-4d72-907f-c70814b60a55\") " Oct 30 18:59:33.312062 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:33.312038 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8ac1217-0b8a-4d72-907f-c70814b60a55-istiod-ca-cert" pod "e8ac1217-0b8a-4d72-907f-c70814b60a55" (UID: "e8ac1217-0b8a-4d72-907f-c70814b60a55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:59:33.312062 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:33.312035 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8ac1217-0b8a-4d72-907f-c70814b60a55-workload-certs" pod "e8ac1217-0b8a-4d72-907f-c70814b60a55" (UID: "e8ac1217-0b8a-4d72-907f-c70814b60a55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:59:33.312376 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:33.312119 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8ac1217-0b8a-4d72-907f-c70814b60a55-istio-token\" (UniqueName: \"kubernetes.io/projected/e8ac1217-0b8a-4d72-907f-c70814b60a55-istio-token\") pod \"e8ac1217-0b8a-4d72-907f-c70814b60a55\" (UID: \"e8ac1217-0b8a-4d72-907f-c70814b60a55\") " Oct 30 18:59:33.312376 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:33.312222 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8ac1217-0b8a-4d72-907f-c70814b60a55-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8ac1217-0b8a-4d72-907f-c70814b60a55-credential-socket\") pod \"e8ac1217-0b8a-4d72-907f-c70814b60a55\" (UID: \"e8ac1217-0b8a-4d72-907f-c70814b60a55\") " Oct 30 18:59:33.312702 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:33.312552 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8ac1217-0b8a-4d72-907f-c70814b60a55-istio-data" pod "e8ac1217-0b8a-4d72-907f-c70814b60a55" (UID: "e8ac1217-0b8a-4d72-907f-c70814b60a55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:59:33.312702 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:33.312667 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8ac1217-0b8a-4d72-907f-c70814b60a55-credential-socket" pod "e8ac1217-0b8a-4d72-907f-c70814b60a55" (UID: "e8ac1217-0b8a-4d72-907f-c70814b60a55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:59:33.312702 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:33.312677 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8ac1217-0b8a-4d72-907f-c70814b60a55-workload-socket" pod "e8ac1217-0b8a-4d72-907f-c70814b60a55" (UID: "e8ac1217-0b8a-4d72-907f-c70814b60a55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:59:33.313302 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:33.313280 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8ac1217-0b8a-4d72-907f-c70814b60a55-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:59:33.313399 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:33.313307 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8ac1217-0b8a-4d72-907f-c70814b60a55-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:59:33.313399 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:33.313323 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8ac1217-0b8a-4d72-907f-c70814b60a55-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:59:33.313399 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:33.313339 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8ac1217-0b8a-4d72-907f-c70814b60a55-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:59:33.313399 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:33.313350 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8ac1217-0b8a-4d72-907f-c70814b60a55-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:59:33.314681 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:33.314593 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8ac1217-0b8a-4d72-907f-c70814b60a55-istio-envoy" pod "e8ac1217-0b8a-4d72-907f-c70814b60a55" (UID: "e8ac1217-0b8a-4d72-907f-c70814b60a55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:59:33.315600 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:33.315552 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8ac1217-0b8a-4d72-907f-c70814b60a55-istio-token" pod "e8ac1217-0b8a-4d72-907f-c70814b60a55" (UID: "e8ac1217-0b8a-4d72-907f-c70814b60a55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:59:33.315700 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:33.315597 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8ac1217-0b8a-4d72-907f-c70814b60a55-istio-podinfo" pod "e8ac1217-0b8a-4d72-907f-c70814b60a55" (UID: "e8ac1217-0b8a-4d72-907f-c70814b60a55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:59:33.315823 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:33.315806 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8ac1217-0b8a-4d72-907f-c70814b60a55-kube-api-access-d4pj8" pod "e8ac1217-0b8a-4d72-907f-c70814b60a55" (UID: "e8ac1217-0b8a-4d72-907f-c70814b60a55"). InnerVolumeSpecName "kube-api-access-d4pj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:59:33.415648 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:33.415599 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4pj8\" (UniqueName: \"kubernetes.io/projected/e8ac1217-0b8a-4d72-907f-c70814b60a55-kube-api-access-d4pj8\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:59:33.415648 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:33.415641 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8ac1217-0b8a-4d72-907f-c70814b60a55-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:59:33.415906 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:33.415658 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8ac1217-0b8a-4d72-907f-c70814b60a55-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:59:33.415906 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:33.415672 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8ac1217-0b8a-4d72-907f-c70814b60a55-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:59:33.686171 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:33.682437 4442 generic.go:356] "Generic (PLEG): container finished" podID="e8ac1217-0b8a-4d72-907f-c70814b60a55" containerID="41add33f7103e2c9fe91e53405c66e78c993031def0b6872ad593dcddd3a88c9" exitCode=0 Oct 30 18:59:33.686171 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:33.682774 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" event={"ID":"e8ac1217-0b8a-4d72-907f-c70814b60a55","Type":"ContainerDied","Data":"41add33f7103e2c9fe91e53405c66e78c993031def0b6872ad593dcddd3a88c9"} Oct 30 18:59:33.686171 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:33.682825 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" event={"ID":"e8ac1217-0b8a-4d72-907f-c70814b60a55","Type":"ContainerDied","Data":"45d2ef2aa9572bd2d1041e3d204b93c621b4b757d35a9049d0bfd8e84bf61fde"} Oct 30 18:59:33.686171 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:33.682852 4442 scope.go:122] "RemoveContainer" containerID="41add33f7103e2c9fe91e53405c66e78c993031def0b6872ad593dcddd3a88c9" Oct 30 18:59:33.686171 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:33.683019 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8" Oct 30 18:59:33.785633 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:33.785604 4442 scope.go:122] "RemoveContainer" containerID="41add33f7103e2c9fe91e53405c66e78c993031def0b6872ad593dcddd3a88c9" Oct 30 18:59:33.789402 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:59:33.789373 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41add33f7103e2c9fe91e53405c66e78c993031def0b6872ad593dcddd3a88c9\": container with ID starting with 41add33f7103e2c9fe91e53405c66e78c993031def0b6872ad593dcddd3a88c9 not found: ID does not exist" containerID="41add33f7103e2c9fe91e53405c66e78c993031def0b6872ad593dcddd3a88c9" Oct 30 18:59:33.789600 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:33.789570 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41add33f7103e2c9fe91e53405c66e78c993031def0b6872ad593dcddd3a88c9"} err="failed to get container status \"41add33f7103e2c9fe91e53405c66e78c993031def0b6872ad593dcddd3a88c9\": rpc error: code = NotFound desc = could not find container \"41add33f7103e2c9fe91e53405c66e78c993031def0b6872ad593dcddd3a88c9\": container with ID starting with 41add33f7103e2c9fe91e53405c66e78c993031def0b6872ad593dcddd3a88c9 not found: ID does not exist" Oct 30 18:59:33.953804 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:33.953712 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8"] Oct 30 18:59:34.008714 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:34.008679 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-rvgh8"] Oct 30 18:59:35.161880 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:35.151316 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8ac1217-0b8a-4d72-907f-c70814b60a55" path="/var/lib/kubelet/pods/e8ac1217-0b8a-4d72-907f-c70814b60a55/volumes" Oct 30 18:59:35.574581 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:35.573842 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr"] Oct 30 18:59:35.574581 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:35.574234 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" podUID="fba1a472-f362-4900-a939-652f2354d12a" containerName="istio-proxy" containerID="cri-o://2d185dfc549dc350b24d608c838089c6cd0a11b95bbefb0adf0b101f7172c56d" gracePeriod=30 Oct 30 18:59:38.639649 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:38.639609 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adc05f24-qlkw6"] Oct 30 18:59:38.640529 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:38.640512 4442 cpu_manager.go:414] "containerMap: removing container" podUID="e8ac1217-0b8a-4d72-907f-c70814b60a55" containerName="istio-proxy" Oct 30 18:59:38.640529 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:38.640529 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8ac1217-0b8a-4d72-907f-c70814b60a55" containerName="istio-proxy" Oct 30 18:59:38.640676 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:38.640666 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8ac1217-0b8a-4d72-907f-c70814b60a55" containerName="istio-proxy" Oct 30 18:59:38.646150 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:38.646117 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adc05f24-qlkw6" Oct 30 18:59:38.664619 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:38.664595 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adc05f24-qlkw6"] Oct 30 18:59:38.669118 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:38.669092 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:59:38.669272 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:38.669155 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:59:38.681608 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:38.681559 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a54f191-8b3f-4d88-87fb-c84c9e76efe8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adc05f24-qlkw6\" (UID: \"3a54f191-8b3f-4d88-87fb-c84c9e76efe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adc05f24-qlkw6" Oct 30 18:59:38.681834 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:38.681813 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a54f191-8b3f-4d88-87fb-c84c9e76efe8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adc05f24-qlkw6\" (UID: \"3a54f191-8b3f-4d88-87fb-c84c9e76efe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adc05f24-qlkw6" Oct 30 18:59:38.681973 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:38.681956 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxggf\" (UniqueName: \"kubernetes.io/projected/3a54f191-8b3f-4d88-87fb-c84c9e76efe8-kube-api-access-vxggf\") pod \"maas-api-revoke-sub-adc05f24-qlkw6\" (UID: \"3a54f191-8b3f-4d88-87fb-c84c9e76efe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adc05f24-qlkw6" Oct 30 18:59:38.783903 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:38.783872 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxggf\" (UniqueName: \"kubernetes.io/projected/3a54f191-8b3f-4d88-87fb-c84c9e76efe8-kube-api-access-vxggf\") pod \"maas-api-revoke-sub-adc05f24-qlkw6\" (UID: \"3a54f191-8b3f-4d88-87fb-c84c9e76efe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adc05f24-qlkw6" Oct 30 18:59:38.784097 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:38.784028 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a54f191-8b3f-4d88-87fb-c84c9e76efe8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adc05f24-qlkw6\" (UID: \"3a54f191-8b3f-4d88-87fb-c84c9e76efe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adc05f24-qlkw6" Oct 30 18:59:38.784196 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:38.784159 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a54f191-8b3f-4d88-87fb-c84c9e76efe8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adc05f24-qlkw6\" (UID: \"3a54f191-8b3f-4d88-87fb-c84c9e76efe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adc05f24-qlkw6" Oct 30 18:59:38.784715 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:38.784683 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a54f191-8b3f-4d88-87fb-c84c9e76efe8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adc05f24-qlkw6\" (UID: \"3a54f191-8b3f-4d88-87fb-c84c9e76efe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adc05f24-qlkw6" Oct 30 18:59:38.786712 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:38.786684 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a54f191-8b3f-4d88-87fb-c84c9e76efe8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adc05f24-qlkw6\" (UID: \"3a54f191-8b3f-4d88-87fb-c84c9e76efe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adc05f24-qlkw6" Oct 30 18:59:38.803822 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:38.803791 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxggf\" (UniqueName: \"kubernetes.io/projected/3a54f191-8b3f-4d88-87fb-c84c9e76efe8-kube-api-access-vxggf\") pod \"maas-api-revoke-sub-adc05f24-qlkw6\" (UID: \"3a54f191-8b3f-4d88-87fb-c84c9e76efe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adc05f24-qlkw6" Oct 30 18:59:38.966058 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:38.965976 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adc05f24-qlkw6" Oct 30 18:59:39.280667 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:59:39.280636 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a54f191_8b3f_4d88_87fb_c84c9e76efe8.slice/crio-bb5046c407354253ef77b2cafc219ed4ca7ebbd574710d36ef39362a47c76732 WatchSource:0}: Error finding container bb5046c407354253ef77b2cafc219ed4ca7ebbd574710d36ef39362a47c76732: Status 404 returned error can't find the container with id bb5046c407354253ef77b2cafc219ed4ca7ebbd574710d36ef39362a47c76732 Oct 30 18:59:39.281603 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:39.281577 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adc05f24-qlkw6"] Oct 30 18:59:40.085452 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:40.085365 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adc05f24-qlkw6" event={"ID":"3a54f191-8b3f-4d88-87fb-c84c9e76efe8","Type":"ContainerStarted","Data":"63e206950b02387b2cceae2d0b8e44941be2f6a8b45932b0878d4ff8963d64dd"} Oct 30 18:59:40.085452 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:40.085414 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adc05f24-qlkw6" event={"ID":"3a54f191-8b3f-4d88-87fb-c84c9e76efe8","Type":"ContainerStarted","Data":"bb5046c407354253ef77b2cafc219ed4ca7ebbd574710d36ef39362a47c76732"} Oct 30 18:59:40.131234 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:40.131173 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adc05f24-qlkw6" podStartSLOduration=2.131150921 podStartE2EDuration="2.131150921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:40.125445211 +0000 UTC m=+3677.215507553" watchObservedRunningTime="2026-10-30 18:59:40.131150921 +0000 UTC m=+3677.221213262" Oct 30 18:59:40.939820 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:40.939791 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" Oct 30 18:59:41.036888 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:41.036853 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fba1a472-f362-4900-a939-652f2354d12a-kube-api-access-gsn8g\" (UniqueName: \"kubernetes.io/projected/fba1a472-f362-4900-a939-652f2354d12a-kube-api-access-gsn8g\") pod \"fba1a472-f362-4900-a939-652f2354d12a\" (UID: \"fba1a472-f362-4900-a939-652f2354d12a\") " Oct 30 18:59:41.037097 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:41.036917 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fba1a472-f362-4900-a939-652f2354d12a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fba1a472-f362-4900-a939-652f2354d12a-istio-podinfo\") pod \"fba1a472-f362-4900-a939-652f2354d12a\" (UID: \"fba1a472-f362-4900-a939-652f2354d12a\") " Oct 30 18:59:41.037097 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:41.037009 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fba1a472-f362-4900-a939-652f2354d12a-istio-token\" (UniqueName: \"kubernetes.io/projected/fba1a472-f362-4900-a939-652f2354d12a-istio-token\") pod \"fba1a472-f362-4900-a939-652f2354d12a\" (UID: \"fba1a472-f362-4900-a939-652f2354d12a\") " Oct 30 18:59:41.037308 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:41.037101 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fba1a472-f362-4900-a939-652f2354d12a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba1a472-f362-4900-a939-652f2354d12a-workload-socket\") pod \"fba1a472-f362-4900-a939-652f2354d12a\" (UID: \"fba1a472-f362-4900-a939-652f2354d12a\") " Oct 30 18:59:41.037308 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:41.037169 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fba1a472-f362-4900-a939-652f2354d12a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba1a472-f362-4900-a939-652f2354d12a-credential-socket\") pod \"fba1a472-f362-4900-a939-652f2354d12a\" (UID: \"fba1a472-f362-4900-a939-652f2354d12a\") " Oct 30 18:59:41.037308 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:41.037202 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fba1a472-f362-4900-a939-652f2354d12a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fba1a472-f362-4900-a939-652f2354d12a-istio-data\") pod \"fba1a472-f362-4900-a939-652f2354d12a\" (UID: \"fba1a472-f362-4900-a939-652f2354d12a\") " Oct 30 18:59:41.037308 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:41.037232 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fba1a472-f362-4900-a939-652f2354d12a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fba1a472-f362-4900-a939-652f2354d12a-istiod-ca-cert\") pod \"fba1a472-f362-4900-a939-652f2354d12a\" (UID: \"fba1a472-f362-4900-a939-652f2354d12a\") " Oct 30 18:59:41.037618 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:41.037589 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba1a472-f362-4900-a939-652f2354d12a-credential-socket" pod "fba1a472-f362-4900-a939-652f2354d12a" (UID: "fba1a472-f362-4900-a939-652f2354d12a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:59:41.037618 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:41.037605 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba1a472-f362-4900-a939-652f2354d12a-istio-data" pod "fba1a472-f362-4900-a939-652f2354d12a" (UID: "fba1a472-f362-4900-a939-652f2354d12a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:59:41.037782 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:41.037672 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba1a472-f362-4900-a939-652f2354d12a-workload-socket" pod "fba1a472-f362-4900-a939-652f2354d12a" (UID: "fba1a472-f362-4900-a939-652f2354d12a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:59:41.038033 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:41.038005 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fba1a472-f362-4900-a939-652f2354d12a-istiod-ca-cert" pod "fba1a472-f362-4900-a939-652f2354d12a" (UID: "fba1a472-f362-4900-a939-652f2354d12a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:59:41.038415 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:41.038115 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fba1a472-f362-4900-a939-652f2354d12a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fba1a472-f362-4900-a939-652f2354d12a-workload-certs\") pod \"fba1a472-f362-4900-a939-652f2354d12a\" (UID: \"fba1a472-f362-4900-a939-652f2354d12a\") " Oct 30 18:59:41.038415 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:41.038230 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fba1a472-f362-4900-a939-652f2354d12a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fba1a472-f362-4900-a939-652f2354d12a-istio-envoy\") pod \"fba1a472-f362-4900-a939-652f2354d12a\" (UID: \"fba1a472-f362-4900-a939-652f2354d12a\") " Oct 30 18:59:41.038415 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:41.038376 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba1a472-f362-4900-a939-652f2354d12a-workload-certs" pod "fba1a472-f362-4900-a939-652f2354d12a" (UID: "fba1a472-f362-4900-a939-652f2354d12a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:59:41.040096 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:41.039460 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fba1a472-f362-4900-a939-652f2354d12a-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:59:41.040096 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:41.039488 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fba1a472-f362-4900-a939-652f2354d12a-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:59:41.040096 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:41.039503 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fba1a472-f362-4900-a939-652f2354d12a-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:59:41.040096 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:41.039520 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba1a472-f362-4900-a939-652f2354d12a-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:59:41.040096 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:41.039534 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba1a472-f362-4900-a939-652f2354d12a-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:59:41.040096 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:41.039850 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fba1a472-f362-4900-a939-652f2354d12a-kube-api-access-gsn8g" pod "fba1a472-f362-4900-a939-652f2354d12a" (UID: "fba1a472-f362-4900-a939-652f2354d12a"). InnerVolumeSpecName "kube-api-access-gsn8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:59:41.040096 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:41.040012 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fba1a472-f362-4900-a939-652f2354d12a-istio-token" pod "fba1a472-f362-4900-a939-652f2354d12a" (UID: "fba1a472-f362-4900-a939-652f2354d12a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:59:41.040584 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:41.040185 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fba1a472-f362-4900-a939-652f2354d12a-istio-podinfo" pod "fba1a472-f362-4900-a939-652f2354d12a" (UID: "fba1a472-f362-4900-a939-652f2354d12a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:59:41.040696 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:41.040676 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba1a472-f362-4900-a939-652f2354d12a-istio-envoy" pod "fba1a472-f362-4900-a939-652f2354d12a" (UID: "fba1a472-f362-4900-a939-652f2354d12a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:59:41.101144 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:41.101105 4442 generic.go:356] "Generic (PLEG): container finished" podID="fba1a472-f362-4900-a939-652f2354d12a" containerID="2d185dfc549dc350b24d608c838089c6cd0a11b95bbefb0adf0b101f7172c56d" exitCode=0 Oct 30 18:59:41.101666 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:41.101172 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" Oct 30 18:59:41.101666 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:41.101165 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" event={"ID":"fba1a472-f362-4900-a939-652f2354d12a","Type":"ContainerDied","Data":"2d185dfc549dc350b24d608c838089c6cd0a11b95bbefb0adf0b101f7172c56d"} Oct 30 18:59:41.101666 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:41.101297 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr" event={"ID":"fba1a472-f362-4900-a939-652f2354d12a","Type":"ContainerDied","Data":"960e406ce70843202f5ac909ec270faba3d1b27cbd05be023c283639a67d76d5"} Oct 30 18:59:41.101666 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:41.101326 4442 scope.go:122] "RemoveContainer" containerID="2d185dfc549dc350b24d608c838089c6cd0a11b95bbefb0adf0b101f7172c56d" Oct 30 18:59:41.104320 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:41.104284 4442 generic.go:356] "Generic (PLEG): container finished" podID="3a54f191-8b3f-4d88-87fb-c84c9e76efe8" containerID="63e206950b02387b2cceae2d0b8e44941be2f6a8b45932b0878d4ff8963d64dd" exitCode=0 Oct 30 18:59:41.104462 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:41.104358 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adc05f24-qlkw6" event={"ID":"3a54f191-8b3f-4d88-87fb-c84c9e76efe8","Type":"ContainerDied","Data":"63e206950b02387b2cceae2d0b8e44941be2f6a8b45932b0878d4ff8963d64dd"} Oct 30 18:59:41.114593 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:41.114559 4442 scope.go:122] "RemoveContainer" containerID="d1f0b4b942e6e0660a461c0a45ed03435ac5e54ff0bf6f952963231d7b0f5f17" Oct 30 18:59:41.130420 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:41.130394 4442 scope.go:122] "RemoveContainer" containerID="2d185dfc549dc350b24d608c838089c6cd0a11b95bbefb0adf0b101f7172c56d" Oct 30 18:59:41.130884 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:59:41.130856 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d185dfc549dc350b24d608c838089c6cd0a11b95bbefb0adf0b101f7172c56d\": container with ID starting with 2d185dfc549dc350b24d608c838089c6cd0a11b95bbefb0adf0b101f7172c56d not found: ID does not exist" containerID="2d185dfc549dc350b24d608c838089c6cd0a11b95bbefb0adf0b101f7172c56d" Oct 30 18:59:41.131008 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:41.130896 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d185dfc549dc350b24d608c838089c6cd0a11b95bbefb0adf0b101f7172c56d"} err="failed to get container status \"2d185dfc549dc350b24d608c838089c6cd0a11b95bbefb0adf0b101f7172c56d\": rpc error: code = NotFound desc = could not find container \"2d185dfc549dc350b24d608c838089c6cd0a11b95bbefb0adf0b101f7172c56d\": container with ID starting with 2d185dfc549dc350b24d608c838089c6cd0a11b95bbefb0adf0b101f7172c56d not found: ID does not exist" Oct 30 18:59:41.131008 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:41.130928 4442 scope.go:122] "RemoveContainer" containerID="d1f0b4b942e6e0660a461c0a45ed03435ac5e54ff0bf6f952963231d7b0f5f17" Oct 30 18:59:41.131353 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:59:41.131330 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1f0b4b942e6e0660a461c0a45ed03435ac5e54ff0bf6f952963231d7b0f5f17\": container with ID starting with d1f0b4b942e6e0660a461c0a45ed03435ac5e54ff0bf6f952963231d7b0f5f17 not found: ID does not exist" containerID="d1f0b4b942e6e0660a461c0a45ed03435ac5e54ff0bf6f952963231d7b0f5f17" Oct 30 18:59:41.131434 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:41.131364 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1f0b4b942e6e0660a461c0a45ed03435ac5e54ff0bf6f952963231d7b0f5f17"} err="failed to get container status \"d1f0b4b942e6e0660a461c0a45ed03435ac5e54ff0bf6f952963231d7b0f5f17\": rpc error: code = NotFound desc = could not find container \"d1f0b4b942e6e0660a461c0a45ed03435ac5e54ff0bf6f952963231d7b0f5f17\": container with ID starting with d1f0b4b942e6e0660a461c0a45ed03435ac5e54ff0bf6f952963231d7b0f5f17 not found: ID does not exist" Oct 30 18:59:41.141504 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:41.141426 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fba1a472-f362-4900-a939-652f2354d12a-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:59:41.141504 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:41.141449 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fba1a472-f362-4900-a939-652f2354d12a-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:59:41.141504 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:41.141459 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fba1a472-f362-4900-a939-652f2354d12a-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:59:41.141504 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:41.141468 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsn8g\" (UniqueName: \"kubernetes.io/projected/fba1a472-f362-4900-a939-652f2354d12a-kube-api-access-gsn8g\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:59:41.163847 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:41.163802 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr"] Oct 30 18:59:41.168864 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:41.168840 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-0cc6b1-openshift-default-7844696f7-7lfqr"] Oct 30 18:59:42.312300 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:42.312271 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adc05f24-qlkw6" Oct 30 18:59:42.463708 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:42.463618 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a54f191-8b3f-4d88-87fb-c84c9e76efe8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a54f191-8b3f-4d88-87fb-c84c9e76efe8-maas-api-xks-ca\") pod \"3a54f191-8b3f-4d88-87fb-c84c9e76efe8\" (UID: \"3a54f191-8b3f-4d88-87fb-c84c9e76efe8\") " Oct 30 18:59:42.463888 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:42.463749 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a54f191-8b3f-4d88-87fb-c84c9e76efe8-kube-api-access-vxggf\" (UniqueName: \"kubernetes.io/projected/3a54f191-8b3f-4d88-87fb-c84c9e76efe8-kube-api-access-vxggf\") pod \"3a54f191-8b3f-4d88-87fb-c84c9e76efe8\" (UID: \"3a54f191-8b3f-4d88-87fb-c84c9e76efe8\") " Oct 30 18:59:42.463973 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:42.463942 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a54f191-8b3f-4d88-87fb-c84c9e76efe8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a54f191-8b3f-4d88-87fb-c84c9e76efe8-maas-api-openshift-ca\") pod \"3a54f191-8b3f-4d88-87fb-c84c9e76efe8\" (UID: \"3a54f191-8b3f-4d88-87fb-c84c9e76efe8\") " Oct 30 18:59:42.464384 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:42.464352 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a54f191-8b3f-4d88-87fb-c84c9e76efe8-maas-api-openshift-ca" pod "3a54f191-8b3f-4d88-87fb-c84c9e76efe8" (UID: "3a54f191-8b3f-4d88-87fb-c84c9e76efe8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:59:42.464787 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:42.464751 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a54f191-8b3f-4d88-87fb-c84c9e76efe8-maas-api-openshift-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:59:42.466253 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:42.466195 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a54f191-8b3f-4d88-87fb-c84c9e76efe8-maas-api-xks-ca" pod "3a54f191-8b3f-4d88-87fb-c84c9e76efe8" (UID: "3a54f191-8b3f-4d88-87fb-c84c9e76efe8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:42.466547 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:42.466525 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a54f191-8b3f-4d88-87fb-c84c9e76efe8-kube-api-access-vxggf" pod "3a54f191-8b3f-4d88-87fb-c84c9e76efe8" (UID: "3a54f191-8b3f-4d88-87fb-c84c9e76efe8"). InnerVolumeSpecName "kube-api-access-vxggf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:59:42.567033 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:42.566994 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxggf\" (UniqueName: \"kubernetes.io/projected/3a54f191-8b3f-4d88-87fb-c84c9e76efe8-kube-api-access-vxggf\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:59:42.567033 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:42.567030 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a54f191-8b3f-4d88-87fb-c84c9e76efe8-maas-api-xks-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:59:43.071278 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:43.071243 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fba1a472-f362-4900-a939-652f2354d12a" path="/var/lib/kubelet/pods/fba1a472-f362-4900-a939-652f2354d12a/volumes" Oct 30 18:59:43.125895 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:43.125865 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adc05f24-qlkw6" Oct 30 18:59:43.126075 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:43.125895 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adc05f24-qlkw6" event={"ID":"3a54f191-8b3f-4d88-87fb-c84c9e76efe8","Type":"ContainerDied","Data":"bb5046c407354253ef77b2cafc219ed4ca7ebbd574710d36ef39362a47c76732"} Oct 30 18:59:43.126075 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:43.125949 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb5046c407354253ef77b2cafc219ed4ca7ebbd574710d36ef39362a47c76732" Oct 30 18:59:48.013114 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:59:48.013058 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 18:59:50.323752 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:50.323715 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7133e51d-zhbt6"] Oct 30 18:59:50.325019 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:50.324999 4442 cpu_manager.go:414] "containerMap: removing container" podUID="fba1a472-f362-4900-a939-652f2354d12a" containerName="istio-proxy" Oct 30 18:59:50.325081 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:50.325023 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fba1a472-f362-4900-a939-652f2354d12a" containerName="istio-proxy" Oct 30 18:59:50.325081 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:50.325040 4442 cpu_manager.go:414] "containerMap: removing container" podUID="fba1a472-f362-4900-a939-652f2354d12a" containerName="istio-proxy" Oct 30 18:59:50.325081 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:50.325052 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fba1a472-f362-4900-a939-652f2354d12a" containerName="istio-proxy" Oct 30 18:59:50.325205 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:50.325108 4442 cpu_manager.go:414] "containerMap: removing container" podUID="3a54f191-8b3f-4d88-87fb-c84c9e76efe8" containerName="revoke-keys" Oct 30 18:59:50.325205 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:50.325116 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a54f191-8b3f-4d88-87fb-c84c9e76efe8" containerName="revoke-keys" Oct 30 18:59:50.325344 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:50.325331 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a54f191-8b3f-4d88-87fb-c84c9e76efe8" containerName="revoke-keys" Oct 30 18:59:50.325380 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:50.325361 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="fba1a472-f362-4900-a939-652f2354d12a" containerName="istio-proxy" Oct 30 18:59:50.325380 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:50.325371 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="fba1a472-f362-4900-a939-652f2354d12a" containerName="istio-proxy" Oct 30 18:59:50.330069 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:50.330048 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7133e51d-zhbt6" Oct 30 18:59:50.348683 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:50.348652 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7133e51d-zhbt6"] Oct 30 18:59:50.352338 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:50.352312 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:59:50.352520 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:50.352401 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:59:50.407760 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:50.407723 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68876174-5cb8-4e63-85b2-a37cf5420e8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7133e51d-zhbt6\" (UID: \"68876174-5cb8-4e63-85b2-a37cf5420e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7133e51d-zhbt6" Oct 30 18:59:50.408125 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:50.408099 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfvrk\" (UniqueName: \"kubernetes.io/projected/68876174-5cb8-4e63-85b2-a37cf5420e8a-kube-api-access-xfvrk\") pod \"maas-api-revoke-sub-7133e51d-zhbt6\" (UID: \"68876174-5cb8-4e63-85b2-a37cf5420e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7133e51d-zhbt6" Oct 30 18:59:50.408550 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:50.408528 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68876174-5cb8-4e63-85b2-a37cf5420e8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7133e51d-zhbt6\" (UID: \"68876174-5cb8-4e63-85b2-a37cf5420e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7133e51d-zhbt6" Oct 30 18:59:50.510185 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:50.510149 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68876174-5cb8-4e63-85b2-a37cf5420e8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7133e51d-zhbt6\" (UID: \"68876174-5cb8-4e63-85b2-a37cf5420e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7133e51d-zhbt6" Oct 30 18:59:50.510371 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:50.510203 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfvrk\" (UniqueName: \"kubernetes.io/projected/68876174-5cb8-4e63-85b2-a37cf5420e8a-kube-api-access-xfvrk\") pod \"maas-api-revoke-sub-7133e51d-zhbt6\" (UID: \"68876174-5cb8-4e63-85b2-a37cf5420e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7133e51d-zhbt6" Oct 30 18:59:50.510417 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:50.510366 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68876174-5cb8-4e63-85b2-a37cf5420e8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7133e51d-zhbt6\" (UID: \"68876174-5cb8-4e63-85b2-a37cf5420e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7133e51d-zhbt6" Oct 30 18:59:50.511064 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:50.510981 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68876174-5cb8-4e63-85b2-a37cf5420e8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7133e51d-zhbt6\" (UID: \"68876174-5cb8-4e63-85b2-a37cf5420e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7133e51d-zhbt6" Oct 30 18:59:50.512722 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:50.512702 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68876174-5cb8-4e63-85b2-a37cf5420e8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7133e51d-zhbt6\" (UID: \"68876174-5cb8-4e63-85b2-a37cf5420e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7133e51d-zhbt6" Oct 30 18:59:50.530285 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:50.530258 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfvrk\" (UniqueName: \"kubernetes.io/projected/68876174-5cb8-4e63-85b2-a37cf5420e8a-kube-api-access-xfvrk\") pod \"maas-api-revoke-sub-7133e51d-zhbt6\" (UID: \"68876174-5cb8-4e63-85b2-a37cf5420e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7133e51d-zhbt6" Oct 30 18:59:50.652825 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:50.652787 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7133e51d-zhbt6" Oct 30 18:59:50.965324 ip-10-0-138-176 kubenswrapper[4442]: W1030 18:59:50.964509 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68876174_5cb8_4e63_85b2_a37cf5420e8a.slice/crio-d6306f3202b1341bd199085210b088a5401a98fee05e9272b7bc572f10d33dd1 WatchSource:0}: Error finding container d6306f3202b1341bd199085210b088a5401a98fee05e9272b7bc572f10d33dd1: Status 404 returned error can't find the container with id d6306f3202b1341bd199085210b088a5401a98fee05e9272b7bc572f10d33dd1 Oct 30 18:59:50.965570 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:50.965369 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7133e51d-zhbt6"] Oct 30 18:59:51.581114 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:51.581068 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7133e51d-zhbt6" event={"ID":"68876174-5cb8-4e63-85b2-a37cf5420e8a","Type":"ContainerStarted","Data":"7a4cb71c99c5c8ec3367c8db46813b7c9015a6dc8185e3dd57e85a93fdeab7d1"} Oct 30 18:59:51.581554 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:51.581191 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7133e51d-zhbt6" event={"ID":"68876174-5cb8-4e63-85b2-a37cf5420e8a","Type":"ContainerStarted","Data":"d6306f3202b1341bd199085210b088a5401a98fee05e9272b7bc572f10d33dd1"} Oct 30 18:59:51.621277 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:51.621225 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7133e51d-zhbt6" podStartSLOduration=1.621208154 podStartE2EDuration="1.621208154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:51.620129189 +0000 UTC m=+3688.710191534" watchObservedRunningTime="2026-10-30 18:59:51.621208154 +0000 UTC m=+3688.711270494" Oct 30 18:59:52.592267 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:52.592225 4442 generic.go:356] "Generic (PLEG): container finished" podID="68876174-5cb8-4e63-85b2-a37cf5420e8a" containerID="7a4cb71c99c5c8ec3367c8db46813b7c9015a6dc8185e3dd57e85a93fdeab7d1" exitCode=0 Oct 30 18:59:52.592778 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:52.592328 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7133e51d-zhbt6" event={"ID":"68876174-5cb8-4e63-85b2-a37cf5420e8a","Type":"ContainerDied","Data":"7a4cb71c99c5c8ec3367c8db46813b7c9015a6dc8185e3dd57e85a93fdeab7d1"} Oct 30 18:59:53.849160 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:53.848726 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466"] Oct 30 18:59:53.849691 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:53.849405 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" podUID="138a0f6a-a94a-4112-b13c-baf9877782c2" containerName="istio-proxy" containerID="cri-o://61c93894c3c8d6d5ce0fec5ec1b6d3f084f8bd763b2a9d47a62e13a06d1ea609" gracePeriod=30 Oct 30 18:59:54.119325 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:54.119295 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7133e51d-zhbt6" Oct 30 18:59:54.178119 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:54.178087 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68876174-5cb8-4e63-85b2-a37cf5420e8a-kube-api-access-xfvrk\" (UniqueName: \"kubernetes.io/projected/68876174-5cb8-4e63-85b2-a37cf5420e8a-kube-api-access-xfvrk\") pod \"68876174-5cb8-4e63-85b2-a37cf5420e8a\" (UID: \"68876174-5cb8-4e63-85b2-a37cf5420e8a\") " Oct 30 18:59:54.178348 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:54.178231 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68876174-5cb8-4e63-85b2-a37cf5420e8a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68876174-5cb8-4e63-85b2-a37cf5420e8a-maas-api-openshift-ca\") pod \"68876174-5cb8-4e63-85b2-a37cf5420e8a\" (UID: \"68876174-5cb8-4e63-85b2-a37cf5420e8a\") " Oct 30 18:59:54.178428 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:54.178368 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68876174-5cb8-4e63-85b2-a37cf5420e8a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68876174-5cb8-4e63-85b2-a37cf5420e8a-maas-api-xks-ca\") pod \"68876174-5cb8-4e63-85b2-a37cf5420e8a\" (UID: \"68876174-5cb8-4e63-85b2-a37cf5420e8a\") " Oct 30 18:59:54.178765 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:54.178734 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68876174-5cb8-4e63-85b2-a37cf5420e8a-maas-api-openshift-ca" pod "68876174-5cb8-4e63-85b2-a37cf5420e8a" (UID: "68876174-5cb8-4e63-85b2-a37cf5420e8a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:59:54.179350 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:54.179321 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68876174-5cb8-4e63-85b2-a37cf5420e8a-maas-api-openshift-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:59:54.180947 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:54.180908 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68876174-5cb8-4e63-85b2-a37cf5420e8a-kube-api-access-xfvrk" pod "68876174-5cb8-4e63-85b2-a37cf5420e8a" (UID: "68876174-5cb8-4e63-85b2-a37cf5420e8a"). InnerVolumeSpecName "kube-api-access-xfvrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:59:54.182576 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:54.182508 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68876174-5cb8-4e63-85b2-a37cf5420e8a-maas-api-xks-ca" pod "68876174-5cb8-4e63-85b2-a37cf5420e8a" (UID: "68876174-5cb8-4e63-85b2-a37cf5420e8a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:54.280706 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:54.280632 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68876174-5cb8-4e63-85b2-a37cf5420e8a-maas-api-xks-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:59:54.280706 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:54.280662 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfvrk\" (UniqueName: \"kubernetes.io/projected/68876174-5cb8-4e63-85b2-a37cf5420e8a-kube-api-access-xfvrk\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:59:54.640543 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:54.640518 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7133e51d-zhbt6" Oct 30 18:59:54.640740 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:54.640547 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7133e51d-zhbt6" event={"ID":"68876174-5cb8-4e63-85b2-a37cf5420e8a","Type":"ContainerDied","Data":"d6306f3202b1341bd199085210b088a5401a98fee05e9272b7bc572f10d33dd1"} Oct 30 18:59:54.640740 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:54.640603 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6306f3202b1341bd199085210b088a5401a98fee05e9272b7bc572f10d33dd1" Oct 30 18:59:57.799359 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:57.799321 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9"] Oct 30 18:59:57.799798 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:57.799546 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" podUID="418064ff-dabc-4b46-b2cf-133f7f20e9d7" containerName="istio-proxy" containerID="cri-o://befaf9497ac79c29b6005be15ed5ac671308122d37941c11afcee31e71d54214" gracePeriod=30 Oct 30 18:59:59.307382 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:59.307353 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" Oct 30 18:59:59.389452 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:59.389422 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/138a0f6a-a94a-4112-b13c-baf9877782c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138a0f6a-a94a-4112-b13c-baf9877782c2-istio-envoy\") pod \"138a0f6a-a94a-4112-b13c-baf9877782c2\" (UID: \"138a0f6a-a94a-4112-b13c-baf9877782c2\") " Oct 30 18:59:59.389635 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:59.389464 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/138a0f6a-a94a-4112-b13c-baf9877782c2-kube-api-access-r2xqs\" (UniqueName: \"kubernetes.io/projected/138a0f6a-a94a-4112-b13c-baf9877782c2-kube-api-access-r2xqs\") pod \"138a0f6a-a94a-4112-b13c-baf9877782c2\" (UID: \"138a0f6a-a94a-4112-b13c-baf9877782c2\") " Oct 30 18:59:59.389635 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:59.389535 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/138a0f6a-a94a-4112-b13c-baf9877782c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138a0f6a-a94a-4112-b13c-baf9877782c2-istiod-ca-cert\") pod \"138a0f6a-a94a-4112-b13c-baf9877782c2\" (UID: \"138a0f6a-a94a-4112-b13c-baf9877782c2\") " Oct 30 18:59:59.389635 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:59.389604 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/138a0f6a-a94a-4112-b13c-baf9877782c2-istio-token\" (UniqueName: \"kubernetes.io/projected/138a0f6a-a94a-4112-b13c-baf9877782c2-istio-token\") pod \"138a0f6a-a94a-4112-b13c-baf9877782c2\" (UID: \"138a0f6a-a94a-4112-b13c-baf9877782c2\") " Oct 30 18:59:59.389857 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:59.389678 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/138a0f6a-a94a-4112-b13c-baf9877782c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138a0f6a-a94a-4112-b13c-baf9877782c2-workload-socket\") pod \"138a0f6a-a94a-4112-b13c-baf9877782c2\" (UID: \"138a0f6a-a94a-4112-b13c-baf9877782c2\") " Oct 30 18:59:59.389857 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:59.389736 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/138a0f6a-a94a-4112-b13c-baf9877782c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138a0f6a-a94a-4112-b13c-baf9877782c2-credential-socket\") pod \"138a0f6a-a94a-4112-b13c-baf9877782c2\" (UID: \"138a0f6a-a94a-4112-b13c-baf9877782c2\") " Oct 30 18:59:59.389857 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:59.389775 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/138a0f6a-a94a-4112-b13c-baf9877782c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138a0f6a-a94a-4112-b13c-baf9877782c2-istio-podinfo\") pod \"138a0f6a-a94a-4112-b13c-baf9877782c2\" (UID: \"138a0f6a-a94a-4112-b13c-baf9877782c2\") " Oct 30 18:59:59.389857 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:59.389816 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/138a0f6a-a94a-4112-b13c-baf9877782c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138a0f6a-a94a-4112-b13c-baf9877782c2-istio-data\") pod \"138a0f6a-a94a-4112-b13c-baf9877782c2\" (UID: \"138a0f6a-a94a-4112-b13c-baf9877782c2\") " Oct 30 18:59:59.390119 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:59.389903 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/138a0f6a-a94a-4112-b13c-baf9877782c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138a0f6a-a94a-4112-b13c-baf9877782c2-workload-certs\") pod \"138a0f6a-a94a-4112-b13c-baf9877782c2\" (UID: \"138a0f6a-a94a-4112-b13c-baf9877782c2\") " Oct 30 18:59:59.390119 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:59.390068 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/138a0f6a-a94a-4112-b13c-baf9877782c2-credential-socket" pod "138a0f6a-a94a-4112-b13c-baf9877782c2" (UID: "138a0f6a-a94a-4112-b13c-baf9877782c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:59:59.390323 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:59.390302 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/138a0f6a-a94a-4112-b13c-baf9877782c2-istio-data" pod "138a0f6a-a94a-4112-b13c-baf9877782c2" (UID: "138a0f6a-a94a-4112-b13c-baf9877782c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:59:59.390396 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:59.390317 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/138a0f6a-a94a-4112-b13c-baf9877782c2-workload-socket" pod "138a0f6a-a94a-4112-b13c-baf9877782c2" (UID: "138a0f6a-a94a-4112-b13c-baf9877782c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:59:59.390566 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:59.390548 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/138a0f6a-a94a-4112-b13c-baf9877782c2-workload-certs" pod "138a0f6a-a94a-4112-b13c-baf9877782c2" (UID: "138a0f6a-a94a-4112-b13c-baf9877782c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:59:59.390950 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:59.390880 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/138a0f6a-a94a-4112-b13c-baf9877782c2-istiod-ca-cert" pod "138a0f6a-a94a-4112-b13c-baf9877782c2" (UID: "138a0f6a-a94a-4112-b13c-baf9877782c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:59:59.391054 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:59.390936 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138a0f6a-a94a-4112-b13c-baf9877782c2-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.391122 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:59.391067 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138a0f6a-a94a-4112-b13c-baf9877782c2-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.391122 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:59.391084 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138a0f6a-a94a-4112-b13c-baf9877782c2-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.391122 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:59.391098 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138a0f6a-a94a-4112-b13c-baf9877782c2-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.393828 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:59.393804 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/138a0f6a-a94a-4112-b13c-baf9877782c2-istio-token" pod "138a0f6a-a94a-4112-b13c-baf9877782c2" (UID: "138a0f6a-a94a-4112-b13c-baf9877782c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:59:59.393935 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:59.393889 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/138a0f6a-a94a-4112-b13c-baf9877782c2-istio-envoy" pod "138a0f6a-a94a-4112-b13c-baf9877782c2" (UID: "138a0f6a-a94a-4112-b13c-baf9877782c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:59:59.394086 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:59.394065 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/138a0f6a-a94a-4112-b13c-baf9877782c2-kube-api-access-r2xqs" pod "138a0f6a-a94a-4112-b13c-baf9877782c2" (UID: "138a0f6a-a94a-4112-b13c-baf9877782c2"). InnerVolumeSpecName "kube-api-access-r2xqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:59:59.395733 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:59.395705 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/138a0f6a-a94a-4112-b13c-baf9877782c2-istio-podinfo" pod "138a0f6a-a94a-4112-b13c-baf9877782c2" (UID: "138a0f6a-a94a-4112-b13c-baf9877782c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:59:59.493211 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:59.493119 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138a0f6a-a94a-4112-b13c-baf9877782c2-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.493211 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:59.493166 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2xqs\" (UniqueName: \"kubernetes.io/projected/138a0f6a-a94a-4112-b13c-baf9877782c2-kube-api-access-r2xqs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.493211 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:59.493182 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138a0f6a-a94a-4112-b13c-baf9877782c2-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.493211 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:59.493195 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/138a0f6a-a94a-4112-b13c-baf9877782c2-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.493211 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:59.493207 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138a0f6a-a94a-4112-b13c-baf9877782c2-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.699994 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:59.699956 4442 generic.go:356] "Generic (PLEG): container finished" podID="138a0f6a-a94a-4112-b13c-baf9877782c2" containerID="61c93894c3c8d6d5ce0fec5ec1b6d3f084f8bd763b2a9d47a62e13a06d1ea609" exitCode=0 Oct 30 18:59:59.700227 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:59.700012 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" Oct 30 18:59:59.700227 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:59.700033 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" event={"ID":"138a0f6a-a94a-4112-b13c-baf9877782c2","Type":"ContainerDied","Data":"61c93894c3c8d6d5ce0fec5ec1b6d3f084f8bd763b2a9d47a62e13a06d1ea609"} Oct 30 18:59:59.700227 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:59.700090 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466" event={"ID":"138a0f6a-a94a-4112-b13c-baf9877782c2","Type":"ContainerDied","Data":"02a81f84ce2a0d4eec3df8d36b6d1f3ce347ad8e4e44593dbdec2b38a7313f27"} Oct 30 18:59:59.700227 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:59.700123 4442 scope.go:122] "RemoveContainer" containerID="61c93894c3c8d6d5ce0fec5ec1b6d3f084f8bd763b2a9d47a62e13a06d1ea609" Oct 30 18:59:59.716236 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:59.716218 4442 scope.go:122] "RemoveContainer" containerID="61c93894c3c8d6d5ce0fec5ec1b6d3f084f8bd763b2a9d47a62e13a06d1ea609" Oct 30 18:59:59.716584 ip-10-0-138-176 kubenswrapper[4442]: E1030 18:59:59.716562 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61c93894c3c8d6d5ce0fec5ec1b6d3f084f8bd763b2a9d47a62e13a06d1ea609\": container with ID starting with 61c93894c3c8d6d5ce0fec5ec1b6d3f084f8bd763b2a9d47a62e13a06d1ea609 not found: ID does not exist" containerID="61c93894c3c8d6d5ce0fec5ec1b6d3f084f8bd763b2a9d47a62e13a06d1ea609" Oct 30 18:59:59.716633 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:59.716593 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61c93894c3c8d6d5ce0fec5ec1b6d3f084f8bd763b2a9d47a62e13a06d1ea609"} err="failed to get container status \"61c93894c3c8d6d5ce0fec5ec1b6d3f084f8bd763b2a9d47a62e13a06d1ea609\": rpc error: code = NotFound desc = could not find container \"61c93894c3c8d6d5ce0fec5ec1b6d3f084f8bd763b2a9d47a62e13a06d1ea609\": container with ID starting with 61c93894c3c8d6d5ce0fec5ec1b6d3f084f8bd763b2a9d47a62e13a06d1ea609 not found: ID does not exist" Oct 30 18:59:59.756852 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:59.756808 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466"] Oct 30 18:59:59.767673 ip-10-0-138-176 kubenswrapper[4442]: I1030 18:59:59.767645 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-rz466"] Oct 30 19:00:00.205156 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.201539 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889780-kp5fc"] Oct 30 19:00:00.205156 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.203082 4442 cpu_manager.go:414] "containerMap: removing container" podUID="138a0f6a-a94a-4112-b13c-baf9877782c2" containerName="istio-proxy" Oct 30 19:00:00.205156 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.203112 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="138a0f6a-a94a-4112-b13c-baf9877782c2" containerName="istio-proxy" Oct 30 19:00:00.205156 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.203151 4442 cpu_manager.go:414] "containerMap: removing container" podUID="68876174-5cb8-4e63-85b2-a37cf5420e8a" containerName="revoke-keys" Oct 30 19:00:00.205156 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.203159 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68876174-5cb8-4e63-85b2-a37cf5420e8a" containerName="revoke-keys" Oct 30 19:00:00.205156 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.203335 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="138a0f6a-a94a-4112-b13c-baf9877782c2" containerName="istio-proxy" Oct 30 19:00:00.205156 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.203350 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="68876174-5cb8-4e63-85b2-a37cf5420e8a" containerName="revoke-keys" Oct 30 19:00:00.240653 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.236366 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889780-kp5fc" Oct 30 19:00:00.253156 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.252837 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889780-jsvns"] Oct 30 19:00:00.270106 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.259804 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889780-jsvns" Oct 30 19:00:00.295155 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.286273 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889780-2hsw5"] Oct 30 19:00:00.295350 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.295251 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889780-kp5fc"] Oct 30 19:00:00.295442 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.295391 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889780-2hsw5" Oct 30 19:00:00.307379 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.300751 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889780-m94vs"] Oct 30 19:00:00.307379 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.306735 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889780-jsvns"] Oct 30 19:00:00.307379 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.306865 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-m94vs" Oct 30 19:00:00.315796 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.315761 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb85x\" (UniqueName: \"kubernetes.io/projected/2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81-kube-api-access-cb85x\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889780-kp5fc\" (UID: \"2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889780-kp5fc" Oct 30 19:00:00.316739 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.316084 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889780-kp5fc\" (UID: \"2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889780-kp5fc" Oct 30 19:00:00.318020 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.317994 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889780-2hsw5"] Oct 30 19:00:00.330780 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.330749 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889780-z79ct"] Oct 30 19:00:00.343369 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.342874 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:00:00.351721 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.351548 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889780-xvmhb"] Oct 30 19:00:00.351893 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.351779 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889780-z79ct" Oct 30 19:00:00.374615 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.374412 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889780-xvmhb" Oct 30 19:00:00.391860 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.391823 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2820d9-29889780-x8hc4"] Oct 30 19:00:00.413772 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.411441 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f37ca7-29889780-j2k5m"] Oct 30 19:00:00.413772 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.412608 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2820d9-29889780-x8hc4" Oct 30 19:00:00.429052 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.418173 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2ba9730-da34-4745-875f-cd941e32eb79-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-38e0db-29889780-2hsw5\" (UID: \"a2ba9730-da34-4745-875f-cd941e32eb79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889780-2hsw5" Oct 30 19:00:00.429052 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.418288 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gzkn\" (UniqueName: \"kubernetes.io/projected/525901b4-d3b5-4bd3-8458-b4f27ec0aef2-kube-api-access-7gzkn\") pod \"maas-api-key-cleanup-29889780-m94vs\" (UID: \"525901b4-d3b5-4bd3-8458-b4f27ec0aef2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-m94vs" Oct 30 19:00:00.429052 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.418327 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb85x\" (UniqueName: \"kubernetes.io/projected/2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81-kube-api-access-cb85x\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889780-kp5fc\" (UID: \"2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889780-kp5fc" Oct 30 19:00:00.429052 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.418359 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxffz\" (UniqueName: \"kubernetes.io/projected/ff29d9e8-03a0-4716-840d-a32e8a5aa628-kube-api-access-hxffz\") pod \"maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889780-z79ct\" (UID: \"ff29d9e8-03a0-4716-840d-a32e8a5aa628\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889780-z79ct" Oct 30 19:00:00.429052 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.418456 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889780-kp5fc\" (UID: \"2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889780-kp5fc" Oct 30 19:00:00.429052 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.419236 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889780-kp5fc\" (UID: \"2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889780-kp5fc" Oct 30 19:00:00.429052 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.419342 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5rr5\" (UniqueName: \"kubernetes.io/projected/a2ba9730-da34-4745-875f-cd941e32eb79-kube-api-access-z5rr5\") pod \"maas-api-key-cleanup-e2e-extmod-b-38e0db-29889780-2hsw5\" (UID: \"a2ba9730-da34-4745-875f-cd941e32eb79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889780-2hsw5" Oct 30 19:00:00.429052 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.419366 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/322ab2aa-d4f3-42ef-9820-9438ced76d6b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889780-jsvns\" (UID: \"322ab2aa-d4f3-42ef-9820-9438ced76d6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889780-jsvns" Oct 30 19:00:00.429052 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.419401 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2njt\" (UniqueName: \"kubernetes.io/projected/322ab2aa-d4f3-42ef-9820-9438ced76d6b-kube-api-access-g2njt\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889780-jsvns\" (UID: \"322ab2aa-d4f3-42ef-9820-9438ced76d6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889780-jsvns" Oct 30 19:00:00.429052 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.419450 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff29d9e8-03a0-4716-840d-a32e8a5aa628-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889780-z79ct\" (UID: \"ff29d9e8-03a0-4716-840d-a32e8a5aa628\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889780-z79ct" Oct 30 19:00:00.429052 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.419511 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/525901b4-d3b5-4bd3-8458-b4f27ec0aef2-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889780-m94vs\" (UID: \"525901b4-d3b5-4bd3-8458-b4f27ec0aef2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-m94vs" Oct 30 19:00:00.429052 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.422225 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9f6b7c-29889780-7dcw7"] Oct 30 19:00:00.429052 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.423612 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f37ca7-29889780-j2k5m" Oct 30 19:00:00.438184 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.435030 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889780-m94vs"] Oct 30 19:00:00.438184 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.435486 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9f6b7c-29889780-7dcw7" Oct 30 19:00:00.445249 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.445222 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889780-z79ct"] Oct 30 19:00:00.451262 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.448910 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb85x\" (UniqueName: \"kubernetes.io/projected/2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81-kube-api-access-cb85x\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889780-kp5fc\" (UID: \"2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889780-kp5fc" Oct 30 19:00:00.456290 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.456223 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889780-xvmhb"] Oct 30 19:00:00.467086 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.467059 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889780-h6xgl"] Oct 30 19:00:00.473546 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.472525 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889780-h6xgl" Oct 30 19:00:00.483965 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.483934 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f37ca7-29889780-j2k5m"] Oct 30 19:00:00.484092 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.484004 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9f6b7c-29889780-7dcw7"] Oct 30 19:00:00.490880 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.490852 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889780-vnmx9"] Oct 30 19:00:00.498082 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.498061 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889780-vnmx9" Oct 30 19:00:00.506788 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.506764 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889780-bpkmv"] Oct 30 19:00:00.512434 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.512414 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889780-bpkmv" Oct 30 19:00:00.512821 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.512807 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889780-h6xgl"] Oct 30 19:00:00.517878 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.517856 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889780-vnmx9"] Oct 30 19:00:00.520703 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.520680 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e7491f9-b178-456d-9a56-3eb6b5ee2e40-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-f37ca7-29889780-j2k5m\" (UID: \"1e7491f9-b178-456d-9a56-3eb6b5ee2e40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f37ca7-29889780-j2k5m" Oct 30 19:00:00.520805 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.520777 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2ba9730-da34-4745-875f-cd941e32eb79-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-38e0db-29889780-2hsw5\" (UID: \"a2ba9730-da34-4745-875f-cd941e32eb79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889780-2hsw5" Oct 30 19:00:00.520879 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.520861 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lms5n\" (UniqueName: \"kubernetes.io/projected/88daa438-101a-4adc-8307-727c4176c705-kube-api-access-lms5n\") pod \"maas-api-key-cleanup-e2e-inf-a-2820d9-29889780-x8hc4\" (UID: \"88daa438-101a-4adc-8307-727c4176c705\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2820d9-29889780-x8hc4" Oct 30 19:00:00.520958 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.520941 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gzkn\" (UniqueName: \"kubernetes.io/projected/525901b4-d3b5-4bd3-8458-b4f27ec0aef2-kube-api-access-7gzkn\") pod \"maas-api-key-cleanup-29889780-m94vs\" (UID: \"525901b4-d3b5-4bd3-8458-b4f27ec0aef2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-m94vs" Oct 30 19:00:00.521025 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.520980 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxffz\" (UniqueName: \"kubernetes.io/projected/ff29d9e8-03a0-4716-840d-a32e8a5aa628-kube-api-access-hxffz\") pod \"maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889780-z79ct\" (UID: \"ff29d9e8-03a0-4716-840d-a32e8a5aa628\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889780-z79ct" Oct 30 19:00:00.521082 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.521064 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97wqt\" (UniqueName: \"kubernetes.io/projected/1e7491f9-b178-456d-9a56-3eb6b5ee2e40-kube-api-access-97wqt\") pod \"maas-api-key-cleanup-e2e-inf-b-f37ca7-29889780-j2k5m\" (UID: \"1e7491f9-b178-456d-9a56-3eb6b5ee2e40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f37ca7-29889780-j2k5m" Oct 30 19:00:00.521196 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.521121 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889780-xvmhb\" (UID: \"9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889780-xvmhb" Oct 30 19:00:00.521275 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.521259 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvz6c\" (UniqueName: \"kubernetes.io/projected/4072bb06-edab-49c8-b213-a381823ec644-kube-api-access-hvz6c\") pod \"maas-api-key-cleanup-e2e-inf-b-9f6b7c-29889780-7dcw7\" (UID: \"4072bb06-edab-49c8-b213-a381823ec644\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9f6b7c-29889780-7dcw7" Oct 30 19:00:00.521336 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.521305 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5rr5\" (UniqueName: \"kubernetes.io/projected/a2ba9730-da34-4745-875f-cd941e32eb79-kube-api-access-z5rr5\") pod \"maas-api-key-cleanup-e2e-extmod-b-38e0db-29889780-2hsw5\" (UID: \"a2ba9730-da34-4745-875f-cd941e32eb79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889780-2hsw5" Oct 30 19:00:00.521391 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.521337 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/322ab2aa-d4f3-42ef-9820-9438ced76d6b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889780-jsvns\" (UID: \"322ab2aa-d4f3-42ef-9820-9438ced76d6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889780-jsvns" Oct 30 19:00:00.521529 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.521510 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4072bb06-edab-49c8-b213-a381823ec644-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9f6b7c-29889780-7dcw7\" (UID: \"4072bb06-edab-49c8-b213-a381823ec644\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9f6b7c-29889780-7dcw7" Oct 30 19:00:00.521603 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.521564 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2njt\" (UniqueName: \"kubernetes.io/projected/322ab2aa-d4f3-42ef-9820-9438ced76d6b-kube-api-access-g2njt\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889780-jsvns\" (UID: \"322ab2aa-d4f3-42ef-9820-9438ced76d6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889780-jsvns" Oct 30 19:00:00.521672 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.521615 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxl9w\" (UniqueName: \"kubernetes.io/projected/9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1-kube-api-access-bxl9w\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889780-xvmhb\" (UID: \"9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889780-xvmhb" Oct 30 19:00:00.521744 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.521699 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff29d9e8-03a0-4716-840d-a32e8a5aa628-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889780-z79ct\" (UID: \"ff29d9e8-03a0-4716-840d-a32e8a5aa628\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889780-z79ct" Oct 30 19:00:00.522318 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.522291 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/525901b4-d3b5-4bd3-8458-b4f27ec0aef2-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889780-m94vs\" (UID: \"525901b4-d3b5-4bd3-8458-b4f27ec0aef2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-m94vs" Oct 30 19:00:00.522447 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.522359 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88daa438-101a-4adc-8307-727c4176c705-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-2820d9-29889780-x8hc4\" (UID: \"88daa438-101a-4adc-8307-727c4176c705\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2820d9-29889780-x8hc4" Oct 30 19:00:00.522622 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.522601 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2ba9730-da34-4745-875f-cd941e32eb79-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-38e0db-29889780-2hsw5\" (UID: \"a2ba9730-da34-4745-875f-cd941e32eb79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889780-2hsw5" Oct 30 19:00:00.522845 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.522822 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/322ab2aa-d4f3-42ef-9820-9438ced76d6b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889780-jsvns\" (UID: \"322ab2aa-d4f3-42ef-9820-9438ced76d6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889780-jsvns" Oct 30 19:00:00.522926 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.522851 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff29d9e8-03a0-4716-840d-a32e8a5aa628-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889780-z79ct\" (UID: \"ff29d9e8-03a0-4716-840d-a32e8a5aa628\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889780-z79ct" Oct 30 19:00:00.523397 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.523288 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/525901b4-d3b5-4bd3-8458-b4f27ec0aef2-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889780-m94vs\" (UID: \"525901b4-d3b5-4bd3-8458-b4f27ec0aef2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-m94vs" Oct 30 19:00:00.525421 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.525371 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889780-qkctp"] Oct 30 19:00:00.535411 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.535385 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889780-bqnkh"] Oct 30 19:00:00.535785 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.535763 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889780-qkctp" Oct 30 19:00:00.545117 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.544704 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2njt\" (UniqueName: \"kubernetes.io/projected/322ab2aa-d4f3-42ef-9820-9438ced76d6b-kube-api-access-g2njt\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889780-jsvns\" (UID: \"322ab2aa-d4f3-42ef-9820-9438ced76d6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889780-jsvns" Oct 30 19:00:00.545697 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.545509 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2820d9-29889780-x8hc4"] Oct 30 19:00:00.545697 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.545695 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889780-bqnkh" Oct 30 19:00:00.550191 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.548078 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5rr5\" (UniqueName: \"kubernetes.io/projected/a2ba9730-da34-4745-875f-cd941e32eb79-kube-api-access-z5rr5\") pod \"maas-api-key-cleanup-e2e-extmod-b-38e0db-29889780-2hsw5\" (UID: \"a2ba9730-da34-4745-875f-cd941e32eb79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889780-2hsw5" Oct 30 19:00:00.550191 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.549815 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxffz\" (UniqueName: \"kubernetes.io/projected/ff29d9e8-03a0-4716-840d-a32e8a5aa628-kube-api-access-hxffz\") pod \"maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889780-z79ct\" (UID: \"ff29d9e8-03a0-4716-840d-a32e8a5aa628\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889780-z79ct" Oct 30 19:00:00.551651 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.551627 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gzkn\" (UniqueName: \"kubernetes.io/projected/525901b4-d3b5-4bd3-8458-b4f27ec0aef2-kube-api-access-7gzkn\") pod \"maas-api-key-cleanup-29889780-m94vs\" (UID: \"525901b4-d3b5-4bd3-8458-b4f27ec0aef2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-m94vs" Oct 30 19:00:00.557005 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.556981 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889780-7xcv7"] Oct 30 19:00:00.566558 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.566528 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889780-7xcv7" Oct 30 19:00:00.569226 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.569200 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889780-bpkmv"] Oct 30 19:00:00.572581 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.572546 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889780-kp5fc" Oct 30 19:00:00.573783 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.573760 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889780-qkctp"] Oct 30 19:00:00.580519 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.580499 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889780-xztsk"] Oct 30 19:00:00.587397 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.587377 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889780-xztsk" Oct 30 19:00:00.590515 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.590479 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1aa59a-29889780-bw98v"] Oct 30 19:00:00.595655 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.595636 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1aa59a-29889780-bw98v" Oct 30 19:00:00.595781 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.595768 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889780-7xcv7"] Oct 30 19:00:00.602959 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.602906 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889780-bqnkh"] Oct 30 19:00:00.612266 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.612241 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1aa59a-29889780-bw98v"] Oct 30 19:00:00.620665 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.620576 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889780-xztsk"] Oct 30 19:00:00.623721 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.623698 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e894ed2-7f33-42f4-aa19-9ba21268f296-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889780-bqnkh\" (UID: \"6e894ed2-7f33-42f4-aa19-9ba21268f296\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889780-bqnkh" Oct 30 19:00:00.623848 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.623748 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889780-xvmhb\" (UID: \"9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889780-xvmhb" Oct 30 19:00:00.623848 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.623775 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-380c08-29889780-bpkmv\" (UID: \"4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889780-bpkmv" Oct 30 19:00:00.623848 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.623796 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb187f07-08dd-40b1-bb89-7f4993d8494f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889780-qkctp\" (UID: \"fb187f07-08dd-40b1-bb89-7f4993d8494f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889780-qkctp" Oct 30 19:00:00.623986 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.623820 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp969\" (UniqueName: \"kubernetes.io/projected/2bb028e5-0be3-4923-9894-dc8f89889799-kube-api-access-cp969\") pod \"maas-api-key-cleanup-e2e-worker-w2-45dc34-29889780-h6xgl\" (UID: \"2bb028e5-0be3-4923-9894-dc8f89889799\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889780-h6xgl" Oct 30 19:00:00.623986 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.623924 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x65rh\" (UniqueName: \"kubernetes.io/projected/fb187f07-08dd-40b1-bb89-7f4993d8494f-kube-api-access-x65rh\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889780-qkctp\" (UID: \"fb187f07-08dd-40b1-bb89-7f4993d8494f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889780-qkctp" Oct 30 19:00:00.624118 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.623995 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bb028e5-0be3-4923-9894-dc8f89889799-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-45dc34-29889780-h6xgl\" (UID: \"2bb028e5-0be3-4923-9894-dc8f89889799\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889780-h6xgl" Oct 30 19:00:00.624118 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.624029 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvz6c\" (UniqueName: \"kubernetes.io/projected/4072bb06-edab-49c8-b213-a381823ec644-kube-api-access-hvz6c\") pod \"maas-api-key-cleanup-e2e-inf-b-9f6b7c-29889780-7dcw7\" (UID: \"4072bb06-edab-49c8-b213-a381823ec644\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9f6b7c-29889780-7dcw7" Oct 30 19:00:00.624118 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.624095 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmt2t\" (UniqueName: \"kubernetes.io/projected/6e894ed2-7f33-42f4-aa19-9ba21268f296-kube-api-access-qmt2t\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889780-bqnkh\" (UID: \"6e894ed2-7f33-42f4-aa19-9ba21268f296\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889780-bqnkh" Oct 30 19:00:00.624350 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.624128 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4072bb06-edab-49c8-b213-a381823ec644-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9f6b7c-29889780-7dcw7\" (UID: \"4072bb06-edab-49c8-b213-a381823ec644\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9f6b7c-29889780-7dcw7" Oct 30 19:00:00.624350 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.624184 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsltm\" (UniqueName: \"kubernetes.io/projected/9fcfbcab-a54b-47d4-957b-76d2cb2444f1-kube-api-access-rsltm\") pod \"maas-api-key-cleanup-e2e-worker-w0-921450-29889780-vnmx9\" (UID: \"9fcfbcab-a54b-47d4-957b-76d2cb2444f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889780-vnmx9" Oct 30 19:00:00.624350 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.624256 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxl9w\" (UniqueName: \"kubernetes.io/projected/9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1-kube-api-access-bxl9w\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889780-xvmhb\" (UID: \"9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889780-xvmhb" Oct 30 19:00:00.624350 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.624321 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsmdt\" (UniqueName: \"kubernetes.io/projected/4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2-kube-api-access-wsmdt\") pod \"maas-api-key-cleanup-e2e-worker-w1-380c08-29889780-bpkmv\" (UID: \"4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889780-bpkmv" Oct 30 19:00:00.624605 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.624407 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88daa438-101a-4adc-8307-727c4176c705-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-2820d9-29889780-x8hc4\" (UID: \"88daa438-101a-4adc-8307-727c4176c705\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2820d9-29889780-x8hc4" Oct 30 19:00:00.624605 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.624516 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e7491f9-b178-456d-9a56-3eb6b5ee2e40-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-f37ca7-29889780-j2k5m\" (UID: \"1e7491f9-b178-456d-9a56-3eb6b5ee2e40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f37ca7-29889780-j2k5m" Oct 30 19:00:00.625712 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.625690 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88daa438-101a-4adc-8307-727c4176c705-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-2820d9-29889780-x8hc4\" (UID: \"88daa438-101a-4adc-8307-727c4176c705\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2820d9-29889780-x8hc4" Oct 30 19:00:00.625877 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.625851 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e7491f9-b178-456d-9a56-3eb6b5ee2e40-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-f37ca7-29889780-j2k5m\" (UID: \"1e7491f9-b178-456d-9a56-3eb6b5ee2e40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f37ca7-29889780-j2k5m" Oct 30 19:00:00.626264 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.626244 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lms5n\" (UniqueName: \"kubernetes.io/projected/88daa438-101a-4adc-8307-727c4176c705-kube-api-access-lms5n\") pod \"maas-api-key-cleanup-e2e-inf-a-2820d9-29889780-x8hc4\" (UID: \"88daa438-101a-4adc-8307-727c4176c705\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2820d9-29889780-x8hc4" Oct 30 19:00:00.626350 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.626319 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fcfbcab-a54b-47d4-957b-76d2cb2444f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-921450-29889780-vnmx9\" (UID: \"9fcfbcab-a54b-47d4-957b-76d2cb2444f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889780-vnmx9" Oct 30 19:00:00.626403 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.626357 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97wqt\" (UniqueName: \"kubernetes.io/projected/1e7491f9-b178-456d-9a56-3eb6b5ee2e40-kube-api-access-97wqt\") pod \"maas-api-key-cleanup-e2e-inf-b-f37ca7-29889780-j2k5m\" (UID: \"1e7491f9-b178-456d-9a56-3eb6b5ee2e40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f37ca7-29889780-j2k5m" Oct 30 19:00:00.626445 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.626429 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889780-xvmhb\" (UID: \"9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889780-xvmhb" Oct 30 19:00:00.628636 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.628564 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4072bb06-edab-49c8-b213-a381823ec644-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9f6b7c-29889780-7dcw7\" (UID: \"4072bb06-edab-49c8-b213-a381823ec644\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9f6b7c-29889780-7dcw7" Oct 30 19:00:00.630326 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.630144 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889780-jsvns" Oct 30 19:00:00.655422 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.653794 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lms5n\" (UniqueName: \"kubernetes.io/projected/88daa438-101a-4adc-8307-727c4176c705-kube-api-access-lms5n\") pod \"maas-api-key-cleanup-e2e-inf-a-2820d9-29889780-x8hc4\" (UID: \"88daa438-101a-4adc-8307-727c4176c705\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2820d9-29889780-x8hc4" Oct 30 19:00:00.655422 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.654374 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889780-2hsw5" Oct 30 19:00:00.655422 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.655319 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxl9w\" (UniqueName: \"kubernetes.io/projected/9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1-kube-api-access-bxl9w\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889780-xvmhb\" (UID: \"9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889780-xvmhb" Oct 30 19:00:00.659200 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.657216 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvz6c\" (UniqueName: \"kubernetes.io/projected/4072bb06-edab-49c8-b213-a381823ec644-kube-api-access-hvz6c\") pod \"maas-api-key-cleanup-e2e-inf-b-9f6b7c-29889780-7dcw7\" (UID: \"4072bb06-edab-49c8-b213-a381823ec644\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9f6b7c-29889780-7dcw7" Oct 30 19:00:00.665750 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.665713 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97wqt\" (UniqueName: \"kubernetes.io/projected/1e7491f9-b178-456d-9a56-3eb6b5ee2e40-kube-api-access-97wqt\") pod \"maas-api-key-cleanup-e2e-inf-b-f37ca7-29889780-j2k5m\" (UID: \"1e7491f9-b178-456d-9a56-3eb6b5ee2e40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f37ca7-29889780-j2k5m" Oct 30 19:00:00.675362 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.675013 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-m94vs" Oct 30 19:00:00.691654 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.691437 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889780-z79ct" Oct 30 19:00:00.714732 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.714450 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889780-xvmhb" Oct 30 19:00:00.733878 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.733840 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bb028e5-0be3-4923-9894-dc8f89889799-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-45dc34-29889780-h6xgl\" (UID: \"2bb028e5-0be3-4923-9894-dc8f89889799\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889780-h6xgl" Oct 30 19:00:00.734024 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.733931 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmt2t\" (UniqueName: \"kubernetes.io/projected/6e894ed2-7f33-42f4-aa19-9ba21268f296-kube-api-access-qmt2t\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889780-bqnkh\" (UID: \"6e894ed2-7f33-42f4-aa19-9ba21268f296\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889780-bqnkh" Oct 30 19:00:00.734024 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.733975 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsltm\" (UniqueName: \"kubernetes.io/projected/9fcfbcab-a54b-47d4-957b-76d2cb2444f1-kube-api-access-rsltm\") pod \"maas-api-key-cleanup-e2e-worker-w0-921450-29889780-vnmx9\" (UID: \"9fcfbcab-a54b-47d4-957b-76d2cb2444f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889780-vnmx9" Oct 30 19:00:00.734186 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.734050 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llpgp\" (UniqueName: \"kubernetes.io/projected/5abb7ef5-cc99-469a-a165-9e9394370418-kube-api-access-llpgp\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889780-xztsk\" (UID: \"5abb7ef5-cc99-469a-a165-9e9394370418\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889780-xztsk" Oct 30 19:00:00.734186 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.734115 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pvr4\" (UniqueName: \"kubernetes.io/projected/00d6e25c-6e65-45de-a894-a6d1989bfd46-kube-api-access-2pvr4\") pod \"maas-api-key-cleanup-e2e-inf-a-1aa59a-29889780-bw98v\" (UID: \"00d6e25c-6e65-45de-a894-a6d1989bfd46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1aa59a-29889780-bw98v" Oct 30 19:00:00.734186 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.734178 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsmdt\" (UniqueName: \"kubernetes.io/projected/4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2-kube-api-access-wsmdt\") pod \"maas-api-key-cleanup-e2e-worker-w1-380c08-29889780-bpkmv\" (UID: \"4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889780-bpkmv" Oct 30 19:00:00.734384 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.734223 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2zr6\" (UniqueName: \"kubernetes.io/projected/39a6d924-7557-4f9b-b481-4398a402b1bd-kube-api-access-s2zr6\") pod \"maas-api-key-cleanup-e2e-extmod-b-39c99d-29889780-7xcv7\" (UID: \"39a6d924-7557-4f9b-b481-4398a402b1bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889780-7xcv7" Oct 30 19:00:00.735200 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.735171 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5abb7ef5-cc99-469a-a165-9e9394370418-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889780-xztsk\" (UID: \"5abb7ef5-cc99-469a-a165-9e9394370418\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889780-xztsk" Oct 30 19:00:00.735406 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.735384 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00d6e25c-6e65-45de-a894-a6d1989bfd46-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1aa59a-29889780-bw98v\" (UID: \"00d6e25c-6e65-45de-a894-a6d1989bfd46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1aa59a-29889780-bw98v" Oct 30 19:00:00.735487 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.735466 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39a6d924-7557-4f9b-b481-4398a402b1bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-39c99d-29889780-7xcv7\" (UID: \"39a6d924-7557-4f9b-b481-4398a402b1bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889780-7xcv7" Oct 30 19:00:00.735558 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.735515 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fcfbcab-a54b-47d4-957b-76d2cb2444f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-921450-29889780-vnmx9\" (UID: \"9fcfbcab-a54b-47d4-957b-76d2cb2444f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889780-vnmx9" Oct 30 19:00:00.735664 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.735644 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e894ed2-7f33-42f4-aa19-9ba21268f296-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889780-bqnkh\" (UID: \"6e894ed2-7f33-42f4-aa19-9ba21268f296\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889780-bqnkh" Oct 30 19:00:00.735769 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.735714 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-380c08-29889780-bpkmv\" (UID: \"4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889780-bpkmv" Oct 30 19:00:00.735769 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.735745 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb187f07-08dd-40b1-bb89-7f4993d8494f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889780-qkctp\" (UID: \"fb187f07-08dd-40b1-bb89-7f4993d8494f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889780-qkctp" Oct 30 19:00:00.737824 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.735788 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp969\" (UniqueName: \"kubernetes.io/projected/2bb028e5-0be3-4923-9894-dc8f89889799-kube-api-access-cp969\") pod \"maas-api-key-cleanup-e2e-worker-w2-45dc34-29889780-h6xgl\" (UID: \"2bb028e5-0be3-4923-9894-dc8f89889799\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889780-h6xgl" Oct 30 19:00:00.737824 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.737795 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x65rh\" (UniqueName: \"kubernetes.io/projected/fb187f07-08dd-40b1-bb89-7f4993d8494f-kube-api-access-x65rh\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889780-qkctp\" (UID: \"fb187f07-08dd-40b1-bb89-7f4993d8494f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889780-qkctp" Oct 30 19:00:00.739303 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.739273 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fcfbcab-a54b-47d4-957b-76d2cb2444f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-921450-29889780-vnmx9\" (UID: \"9fcfbcab-a54b-47d4-957b-76d2cb2444f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889780-vnmx9" Oct 30 19:00:00.739441 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.739418 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bb028e5-0be3-4923-9894-dc8f89889799-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-45dc34-29889780-h6xgl\" (UID: \"2bb028e5-0be3-4923-9894-dc8f89889799\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889780-h6xgl" Oct 30 19:00:00.739531 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.739430 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb187f07-08dd-40b1-bb89-7f4993d8494f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889780-qkctp\" (UID: \"fb187f07-08dd-40b1-bb89-7f4993d8494f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889780-qkctp" Oct 30 19:00:00.744379 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.740926 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-380c08-29889780-bpkmv\" (UID: \"4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889780-bpkmv" Oct 30 19:00:00.744379 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.743322 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e894ed2-7f33-42f4-aa19-9ba21268f296-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889780-bqnkh\" (UID: \"6e894ed2-7f33-42f4-aa19-9ba21268f296\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889780-bqnkh" Oct 30 19:00:00.756059 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.756034 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2820d9-29889780-x8hc4" Oct 30 19:00:00.770165 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.765541 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp969\" (UniqueName: \"kubernetes.io/projected/2bb028e5-0be3-4923-9894-dc8f89889799-kube-api-access-cp969\") pod \"maas-api-key-cleanup-e2e-worker-w2-45dc34-29889780-h6xgl\" (UID: \"2bb028e5-0be3-4923-9894-dc8f89889799\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889780-h6xgl" Oct 30 19:00:00.773380 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.771758 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x65rh\" (UniqueName: \"kubernetes.io/projected/fb187f07-08dd-40b1-bb89-7f4993d8494f-kube-api-access-x65rh\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889780-qkctp\" (UID: \"fb187f07-08dd-40b1-bb89-7f4993d8494f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889780-qkctp" Oct 30 19:00:00.795164 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.794927 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f37ca7-29889780-j2k5m" Oct 30 19:00:00.804516 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.804067 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmt2t\" (UniqueName: \"kubernetes.io/projected/6e894ed2-7f33-42f4-aa19-9ba21268f296-kube-api-access-qmt2t\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889780-bqnkh\" (UID: \"6e894ed2-7f33-42f4-aa19-9ba21268f296\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889780-bqnkh" Oct 30 19:00:00.808758 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.808018 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9f6b7c-29889780-7dcw7" Oct 30 19:00:00.824170 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.823927 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889780-h6xgl" Oct 30 19:00:00.844239 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.842490 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00d6e25c-6e65-45de-a894-a6d1989bfd46-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1aa59a-29889780-bw98v\" (UID: \"00d6e25c-6e65-45de-a894-a6d1989bfd46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1aa59a-29889780-bw98v" Oct 30 19:00:00.844239 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.842572 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39a6d924-7557-4f9b-b481-4398a402b1bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-39c99d-29889780-7xcv7\" (UID: \"39a6d924-7557-4f9b-b481-4398a402b1bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889780-7xcv7" Oct 30 19:00:00.844239 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.842869 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llpgp\" (UniqueName: \"kubernetes.io/projected/5abb7ef5-cc99-469a-a165-9e9394370418-kube-api-access-llpgp\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889780-xztsk\" (UID: \"5abb7ef5-cc99-469a-a165-9e9394370418\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889780-xztsk" Oct 30 19:00:00.844239 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.842951 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pvr4\" (UniqueName: \"kubernetes.io/projected/00d6e25c-6e65-45de-a894-a6d1989bfd46-kube-api-access-2pvr4\") pod \"maas-api-key-cleanup-e2e-inf-a-1aa59a-29889780-bw98v\" (UID: \"00d6e25c-6e65-45de-a894-a6d1989bfd46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1aa59a-29889780-bw98v" Oct 30 19:00:00.844239 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.843000 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2zr6\" (UniqueName: \"kubernetes.io/projected/39a6d924-7557-4f9b-b481-4398a402b1bd-kube-api-access-s2zr6\") pod \"maas-api-key-cleanup-e2e-extmod-b-39c99d-29889780-7xcv7\" (UID: \"39a6d924-7557-4f9b-b481-4398a402b1bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889780-7xcv7" Oct 30 19:00:00.844239 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.843115 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5abb7ef5-cc99-469a-a165-9e9394370418-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889780-xztsk\" (UID: \"5abb7ef5-cc99-469a-a165-9e9394370418\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889780-xztsk" Oct 30 19:00:00.844239 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.844088 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5abb7ef5-cc99-469a-a165-9e9394370418-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889780-xztsk\" (UID: \"5abb7ef5-cc99-469a-a165-9e9394370418\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889780-xztsk" Oct 30 19:00:00.844923 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.844897 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00d6e25c-6e65-45de-a894-a6d1989bfd46-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1aa59a-29889780-bw98v\" (UID: \"00d6e25c-6e65-45de-a894-a6d1989bfd46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1aa59a-29889780-bw98v" Oct 30 19:00:00.846209 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.846179 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39a6d924-7557-4f9b-b481-4398a402b1bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-39c99d-29889780-7xcv7\" (UID: \"39a6d924-7557-4f9b-b481-4398a402b1bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889780-7xcv7" Oct 30 19:00:00.848419 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.848388 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsmdt\" (UniqueName: \"kubernetes.io/projected/4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2-kube-api-access-wsmdt\") pod \"maas-api-key-cleanup-e2e-worker-w1-380c08-29889780-bpkmv\" (UID: \"4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889780-bpkmv" Oct 30 19:00:00.851790 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.851761 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsltm\" (UniqueName: \"kubernetes.io/projected/9fcfbcab-a54b-47d4-957b-76d2cb2444f1-kube-api-access-rsltm\") pod \"maas-api-key-cleanup-e2e-worker-w0-921450-29889780-vnmx9\" (UID: \"9fcfbcab-a54b-47d4-957b-76d2cb2444f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889780-vnmx9" Oct 30 19:00:00.852738 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.852628 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889780-bpkmv" Oct 30 19:00:00.878076 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.878044 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2zr6\" (UniqueName: \"kubernetes.io/projected/39a6d924-7557-4f9b-b481-4398a402b1bd-kube-api-access-s2zr6\") pod \"maas-api-key-cleanup-e2e-extmod-b-39c99d-29889780-7xcv7\" (UID: \"39a6d924-7557-4f9b-b481-4398a402b1bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889780-7xcv7" Oct 30 19:00:00.882436 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.882397 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pvr4\" (UniqueName: \"kubernetes.io/projected/00d6e25c-6e65-45de-a894-a6d1989bfd46-kube-api-access-2pvr4\") pod \"maas-api-key-cleanup-e2e-inf-a-1aa59a-29889780-bw98v\" (UID: \"00d6e25c-6e65-45de-a894-a6d1989bfd46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1aa59a-29889780-bw98v" Oct 30 19:00:00.889167 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.888490 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llpgp\" (UniqueName: \"kubernetes.io/projected/5abb7ef5-cc99-469a-a165-9e9394370418-kube-api-access-llpgp\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889780-xztsk\" (UID: \"5abb7ef5-cc99-469a-a165-9e9394370418\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889780-xztsk" Oct 30 19:00:00.947422 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.946912 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889780-qkctp" Oct 30 19:00:00.975748 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:00.962516 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889780-bqnkh" Oct 30 19:00:01.014937 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:01.014909 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889780-kp5fc"] Oct 30 19:00:01.065618 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:01.065407 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889780-7xcv7" Oct 30 19:00:01.091489 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:01.091460 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889780-xztsk" Oct 30 19:00:01.095085 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:01.095050 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1aa59a-29889780-bw98v" Oct 30 19:00:01.110887 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:01.110822 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="138a0f6a-a94a-4112-b13c-baf9877782c2" path="/var/lib/kubelet/pods/138a0f6a-a94a-4112-b13c-baf9877782c2/volumes" Oct 30 19:00:01.198110 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:01.196938 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889780-vnmx9" Oct 30 19:00:01.490646 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:01.490618 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889780-z79ct"] Oct 30 19:00:01.500558 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:01.500533 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889780-2hsw5"] Oct 30 19:00:01.647328 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:01.647302 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889780-jsvns"] Oct 30 19:00:01.661613 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:01.661583 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889780-xvmhb"] Oct 30 19:00:01.670686 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:01.670655 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2820d9-29889780-x8hc4"] Oct 30 19:00:01.733449 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:01.733367 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9f6b7c-29889780-7dcw7"] Oct 30 19:00:01.738952 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:01.738920 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f37ca7-29889780-j2k5m"] Oct 30 19:00:01.763188 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:01.763152 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889780-kp5fc" event={"ID":"2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81","Type":"ContainerStarted","Data":"6b84ea7f685f4173ebb5a869e7c08afa46bcc07ba728ccc9664ad2a4edd96849"} Oct 30 19:00:01.765062 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:01.765032 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889780-z79ct" event={"ID":"ff29d9e8-03a0-4716-840d-a32e8a5aa628","Type":"ContainerStarted","Data":"f72d8c7e12e7aac3c1999e133cfa2b4777021401466720bcdb81128d4a273853"} Oct 30 19:00:01.766772 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:01.766748 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889780-2hsw5" event={"ID":"a2ba9730-da34-4745-875f-cd941e32eb79","Type":"ContainerStarted","Data":"b3b5e5f1b9446bc089cbc2a5f43e54400f6a481980135e7372a65c77d4f7bce3"} Oct 30 19:00:01.767790 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:01.767652 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889780-m94vs"] Oct 30 19:00:01.847755 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:01.847706 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889780-bpkmv"] Oct 30 19:00:01.867858 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:01.867781 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1aa59a-29889780-bw98v"] Oct 30 19:00:01.874387 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:01.874349 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889780-vnmx9"] Oct 30 19:00:01.879816 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:01.879787 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889780-xztsk"] Oct 30 19:00:01.892207 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:01.892177 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889780-qkctp"] Oct 30 19:00:01.930191 ip-10-0-138-176 kubenswrapper[4442]: W1030 19:00:01.929965 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88daa438_101a_4adc_8307_727c4176c705.slice/crio-209a92d135a6f973fcac4d44b8478333177e947ae85406a746205175de23df5a WatchSource:0}: Error finding container 209a92d135a6f973fcac4d44b8478333177e947ae85406a746205175de23df5a: Status 404 returned error can't find the container with id 209a92d135a6f973fcac4d44b8478333177e947ae85406a746205175de23df5a Oct 30 19:00:01.932759 ip-10-0-138-176 kubenswrapper[4442]: W1030 19:00:01.932728 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b5613a0_fe8a_4ade_9f7d_9ce7485b85d1.slice/crio-d55ce94070dfce30947f5552869a72291b76899312072e2dd2927b2ca9288f23 WatchSource:0}: Error finding container d55ce94070dfce30947f5552869a72291b76899312072e2dd2927b2ca9288f23: Status 404 returned error can't find the container with id d55ce94070dfce30947f5552869a72291b76899312072e2dd2927b2ca9288f23 Oct 30 19:00:01.959473 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:01.958291 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889780-bqnkh"] Oct 30 19:00:01.966717 ip-10-0-138-176 kubenswrapper[4442]: W1030 19:00:01.965937 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5abb7ef5_cc99_469a_a165_9e9394370418.slice/crio-9abb0c82ca6de0fe59ef6fa6981772e0b121e96b37ae8de7c06e149c42c629c9 WatchSource:0}: Error finding container 9abb0c82ca6de0fe59ef6fa6981772e0b121e96b37ae8de7c06e149c42c629c9: Status 404 returned error can't find the container with id 9abb0c82ca6de0fe59ef6fa6981772e0b121e96b37ae8de7c06e149c42c629c9 Oct 30 19:00:02.038810 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:02.034214 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889780-7xcv7"] Oct 30 19:00:02.043583 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:02.042895 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889780-h6xgl"] Oct 30 19:00:02.421780 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:02.421739 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll"] Oct 30 19:00:02.422115 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:02.422091 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" podUID="04780470-66fb-45c6-b86e-8befadb35217" containerName="istio-proxy" containerID="cri-o://ad3911fabe4c2c631c1add2c9c8231cb461f13a2b61b3e851896815f13aa0829" gracePeriod=30 Oct 30 19:00:02.780589 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:02.780553 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889780-xvmhb" event={"ID":"9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1","Type":"ContainerStarted","Data":"d55ce94070dfce30947f5552869a72291b76899312072e2dd2927b2ca9288f23"} Oct 30 19:00:02.782069 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:02.782039 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889780-bpkmv" event={"ID":"4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2","Type":"ContainerStarted","Data":"1cfe6587840938d23e8880be9c7a98943e75d4e243ff580538408179dfb2c766"} Oct 30 19:00:02.783648 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:02.783620 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f37ca7-29889780-j2k5m" event={"ID":"1e7491f9-b178-456d-9a56-3eb6b5ee2e40","Type":"ContainerStarted","Data":"9b410e2c97ef0f9507a69d6d2254d4aaca52e657c1f5bef3dea9919f22384533"} Oct 30 19:00:02.785702 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:02.785675 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889780-qkctp" event={"ID":"fb187f07-08dd-40b1-bb89-7f4993d8494f","Type":"ContainerStarted","Data":"a8cfc77cedafdd7bcbc3b2e17a64f94b2c5c7c041e094aa5b8c351b7595a14e0"} Oct 30 19:00:02.785852 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:02.785729 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889780-qkctp" event={"ID":"fb187f07-08dd-40b1-bb89-7f4993d8494f","Type":"ContainerStarted","Data":"035ab7b23ec1e3ee6a2442ac4719c12042504914cdfaad7ec8d805a06eb4bd0d"} Oct 30 19:00:02.787644 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:02.787619 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889780-h6xgl" event={"ID":"2bb028e5-0be3-4923-9894-dc8f89889799","Type":"ContainerStarted","Data":"9ff790f08dd1e910993c719a6adaa5555844ebfa2c3ed5db0bc79d0577a45c45"} Oct 30 19:00:02.789031 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:02.789010 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889780-bqnkh" event={"ID":"6e894ed2-7f33-42f4-aa19-9ba21268f296","Type":"ContainerStarted","Data":"d4d36fc7fbcf75774f79b42561560887510c741ff420b9d0b2cbbd62ba1e1c54"} Oct 30 19:00:02.790485 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:02.790455 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2820d9-29889780-x8hc4" event={"ID":"88daa438-101a-4adc-8307-727c4176c705","Type":"ContainerStarted","Data":"209a92d135a6f973fcac4d44b8478333177e947ae85406a746205175de23df5a"} Oct 30 19:00:02.791901 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:02.791879 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9f6b7c-29889780-7dcw7" event={"ID":"4072bb06-edab-49c8-b213-a381823ec644","Type":"ContainerStarted","Data":"27ab561b798f39a5082b62634f54f4a189b033e801565a9a4312732e32cb781a"} Oct 30 19:00:02.793707 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:02.793683 4442 generic.go:356] "Generic (PLEG): container finished" podID="2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81" containerID="bfbe09e315456c4763aeecb5b5f9d344a35cafca48df6e78cf3952145d717793" exitCode=60 Oct 30 19:00:02.793798 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:02.793752 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889780-kp5fc" event={"ID":"2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81","Type":"ContainerDied","Data":"bfbe09e315456c4763aeecb5b5f9d344a35cafca48df6e78cf3952145d717793"} Oct 30 19:00:02.794201 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:02.794118 4442 scope.go:122] "RemoveContainer" containerID="bfbe09e315456c4763aeecb5b5f9d344a35cafca48df6e78cf3952145d717793" Oct 30 19:00:02.795369 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:02.795337 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889780-vnmx9" event={"ID":"9fcfbcab-a54b-47d4-957b-76d2cb2444f1","Type":"ContainerStarted","Data":"2ff17a72cd2ad36112e7c8c0cb1d43a2442a2dfdeff5c5375f656b6989b61ec8"} Oct 30 19:00:02.798389 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:02.798358 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889780-jsvns" event={"ID":"322ab2aa-d4f3-42ef-9820-9438ced76d6b","Type":"ContainerStarted","Data":"ef56fe24750db9c5f1676fe6ef9ded5a0afbd7da2a1b87a9d81e1ecb1045c417"} Oct 30 19:00:02.799818 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:02.799795 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1aa59a-29889780-bw98v" event={"ID":"00d6e25c-6e65-45de-a894-a6d1989bfd46","Type":"ContainerStarted","Data":"e320981fb539ad0027273e788b548690b9bbb4a8812a3ee85b7ae40509f248ec"} Oct 30 19:00:02.801290 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:02.801268 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-m94vs" event={"ID":"525901b4-d3b5-4bd3-8458-b4f27ec0aef2","Type":"ContainerStarted","Data":"e06b83314f88ea0bd06ccdeaccaaa13be24f8e426983faaada0b9d0fee794f14"} Oct 30 19:00:02.803566 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:02.803538 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889780-2hsw5" event={"ID":"a2ba9730-da34-4745-875f-cd941e32eb79","Type":"ContainerStarted","Data":"9f5ef389355d262f3e7ef646408d257619247aea96ecc6b16b6f2d87a0c23e9f"} Oct 30 19:00:02.805250 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:02.805227 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889780-xztsk" event={"ID":"5abb7ef5-cc99-469a-a165-9e9394370418","Type":"ContainerStarted","Data":"9abb0c82ca6de0fe59ef6fa6981772e0b121e96b37ae8de7c06e149c42c629c9"} Oct 30 19:00:02.807047 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:02.807003 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889780-7xcv7" event={"ID":"39a6d924-7557-4f9b-b481-4398a402b1bd","Type":"ContainerStarted","Data":"d4d98dccaa8581b683ab56189c150d3e2eec04428938415806a92a5ae3e17dc9"} Oct 30 19:00:02.834075 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:02.834014 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889780-qkctp" podStartSLOduration=2.8339954990000003 podStartE2EDuration="2.833995499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:02.825763686 +0000 UTC m=+3699.915826026" watchObservedRunningTime="2026-10-30 19:00:02.833995499 +0000 UTC m=+3699.924057844" Oct 30 19:00:02.930082 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:02.923951 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889780-2hsw5" podStartSLOduration=2.9239321350000003 podStartE2EDuration="2.923932135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:02.872768273 +0000 UTC m=+3699.962830613" watchObservedRunningTime="2026-10-30 19:00:02.923932135 +0000 UTC m=+3700.013994475" Oct 30 19:00:03.382437 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.382406 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" Oct 30 19:00:03.451234 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.444915 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/418064ff-dabc-4b46-b2cf-133f7f20e9d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/418064ff-dabc-4b46-b2cf-133f7f20e9d7-istiod-ca-cert\") pod \"418064ff-dabc-4b46-b2cf-133f7f20e9d7\" (UID: \"418064ff-dabc-4b46-b2cf-133f7f20e9d7\") " Oct 30 19:00:03.451234 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.444972 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/418064ff-dabc-4b46-b2cf-133f7f20e9d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/418064ff-dabc-4b46-b2cf-133f7f20e9d7-workload-socket\") pod \"418064ff-dabc-4b46-b2cf-133f7f20e9d7\" (UID: \"418064ff-dabc-4b46-b2cf-133f7f20e9d7\") " Oct 30 19:00:03.451234 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.445080 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/418064ff-dabc-4b46-b2cf-133f7f20e9d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/418064ff-dabc-4b46-b2cf-133f7f20e9d7-credential-socket\") pod \"418064ff-dabc-4b46-b2cf-133f7f20e9d7\" (UID: \"418064ff-dabc-4b46-b2cf-133f7f20e9d7\") " Oct 30 19:00:03.451234 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.445175 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/418064ff-dabc-4b46-b2cf-133f7f20e9d7-istio-token\" (UniqueName: \"kubernetes.io/projected/418064ff-dabc-4b46-b2cf-133f7f20e9d7-istio-token\") pod \"418064ff-dabc-4b46-b2cf-133f7f20e9d7\" (UID: \"418064ff-dabc-4b46-b2cf-133f7f20e9d7\") " Oct 30 19:00:03.451234 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.445262 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/418064ff-dabc-4b46-b2cf-133f7f20e9d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/418064ff-dabc-4b46-b2cf-133f7f20e9d7-istio-podinfo\") pod \"418064ff-dabc-4b46-b2cf-133f7f20e9d7\" (UID: \"418064ff-dabc-4b46-b2cf-133f7f20e9d7\") " Oct 30 19:00:03.451234 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.445323 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/418064ff-dabc-4b46-b2cf-133f7f20e9d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/418064ff-dabc-4b46-b2cf-133f7f20e9d7-istio-data\") pod \"418064ff-dabc-4b46-b2cf-133f7f20e9d7\" (UID: \"418064ff-dabc-4b46-b2cf-133f7f20e9d7\") " Oct 30 19:00:03.451234 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.445486 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/418064ff-dabc-4b46-b2cf-133f7f20e9d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/418064ff-dabc-4b46-b2cf-133f7f20e9d7-istio-envoy\") pod \"418064ff-dabc-4b46-b2cf-133f7f20e9d7\" (UID: \"418064ff-dabc-4b46-b2cf-133f7f20e9d7\") " Oct 30 19:00:03.451234 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.445572 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/418064ff-dabc-4b46-b2cf-133f7f20e9d7-kube-api-access-9k4sb\" (UniqueName: \"kubernetes.io/projected/418064ff-dabc-4b46-b2cf-133f7f20e9d7-kube-api-access-9k4sb\") pod \"418064ff-dabc-4b46-b2cf-133f7f20e9d7\" (UID: \"418064ff-dabc-4b46-b2cf-133f7f20e9d7\") " Oct 30 19:00:03.451234 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.445632 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/418064ff-dabc-4b46-b2cf-133f7f20e9d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/418064ff-dabc-4b46-b2cf-133f7f20e9d7-workload-certs\") pod \"418064ff-dabc-4b46-b2cf-133f7f20e9d7\" (UID: \"418064ff-dabc-4b46-b2cf-133f7f20e9d7\") " Oct 30 19:00:03.452032 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.451580 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/418064ff-dabc-4b46-b2cf-133f7f20e9d7-workload-certs" pod "418064ff-dabc-4b46-b2cf-133f7f20e9d7" (UID: "418064ff-dabc-4b46-b2cf-133f7f20e9d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:03.455527 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.455230 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/418064ff-dabc-4b46-b2cf-133f7f20e9d7-workload-socket" pod "418064ff-dabc-4b46-b2cf-133f7f20e9d7" (UID: "418064ff-dabc-4b46-b2cf-133f7f20e9d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:03.456087 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.456062 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/418064ff-dabc-4b46-b2cf-133f7f20e9d7-credential-socket" pod "418064ff-dabc-4b46-b2cf-133f7f20e9d7" (UID: "418064ff-dabc-4b46-b2cf-133f7f20e9d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:03.459344 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.458459 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/418064ff-dabc-4b46-b2cf-133f7f20e9d7-istiod-ca-cert" pod "418064ff-dabc-4b46-b2cf-133f7f20e9d7" (UID: "418064ff-dabc-4b46-b2cf-133f7f20e9d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:03.473361 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.473316 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/418064ff-dabc-4b46-b2cf-133f7f20e9d7-istio-data" pod "418064ff-dabc-4b46-b2cf-133f7f20e9d7" (UID: "418064ff-dabc-4b46-b2cf-133f7f20e9d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:03.493207 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.492700 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/418064ff-dabc-4b46-b2cf-133f7f20e9d7-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:03.493207 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.492728 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/418064ff-dabc-4b46-b2cf-133f7f20e9d7-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:03.493207 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.492744 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/418064ff-dabc-4b46-b2cf-133f7f20e9d7-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:03.493207 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.492762 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/418064ff-dabc-4b46-b2cf-133f7f20e9d7-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:03.493207 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.492779 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/418064ff-dabc-4b46-b2cf-133f7f20e9d7-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:03.626436 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.625082 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/418064ff-dabc-4b46-b2cf-133f7f20e9d7-istio-podinfo" pod "418064ff-dabc-4b46-b2cf-133f7f20e9d7" (UID: "418064ff-dabc-4b46-b2cf-133f7f20e9d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:00:03.626436 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.625270 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/418064ff-dabc-4b46-b2cf-133f7f20e9d7-istio-token" pod "418064ff-dabc-4b46-b2cf-133f7f20e9d7" (UID: "418064ff-dabc-4b46-b2cf-133f7f20e9d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:03.628966 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.627553 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/418064ff-dabc-4b46-b2cf-133f7f20e9d7-istio-envoy" pod "418064ff-dabc-4b46-b2cf-133f7f20e9d7" (UID: "418064ff-dabc-4b46-b2cf-133f7f20e9d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:03.637563 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.637530 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/418064ff-dabc-4b46-b2cf-133f7f20e9d7-kube-api-access-9k4sb" pod "418064ff-dabc-4b46-b2cf-133f7f20e9d7" (UID: "418064ff-dabc-4b46-b2cf-133f7f20e9d7"). InnerVolumeSpecName "kube-api-access-9k4sb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:03.698114 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.698081 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/418064ff-dabc-4b46-b2cf-133f7f20e9d7-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:03.698114 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.698113 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k4sb\" (UniqueName: \"kubernetes.io/projected/418064ff-dabc-4b46-b2cf-133f7f20e9d7-kube-api-access-9k4sb\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:03.698330 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.698125 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/418064ff-dabc-4b46-b2cf-133f7f20e9d7-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:03.698330 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.698153 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/418064ff-dabc-4b46-b2cf-133f7f20e9d7-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:03.823320 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.823279 4442 generic.go:356] "Generic (PLEG): container finished" podID="418064ff-dabc-4b46-b2cf-133f7f20e9d7" containerID="befaf9497ac79c29b6005be15ed5ac671308122d37941c11afcee31e71d54214" exitCode=0 Oct 30 19:00:03.823853 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.823395 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" event={"ID":"418064ff-dabc-4b46-b2cf-133f7f20e9d7","Type":"ContainerDied","Data":"befaf9497ac79c29b6005be15ed5ac671308122d37941c11afcee31e71d54214"} Oct 30 19:00:03.823853 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.823445 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" event={"ID":"418064ff-dabc-4b46-b2cf-133f7f20e9d7","Type":"ContainerDied","Data":"6e85e0085165d640b722690d32767a610e4371852a5ee415c3127d3cf7d3bb0d"} Oct 30 19:00:03.823853 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.823473 4442 scope.go:122] "RemoveContainer" containerID="befaf9497ac79c29b6005be15ed5ac671308122d37941c11afcee31e71d54214" Oct 30 19:00:03.824194 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.824114 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9" Oct 30 19:00:03.835087 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.835057 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889780-bpkmv" event={"ID":"4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2","Type":"ContainerStarted","Data":"549fd59797e7104fa04e3b03f49b0bb69712dfeed22249625c97602ccd4b707d"} Oct 30 19:00:03.837476 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.837447 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f37ca7-29889780-j2k5m" event={"ID":"1e7491f9-b178-456d-9a56-3eb6b5ee2e40","Type":"ContainerStarted","Data":"611c1df0460515aea951048d5141f8139261e058fdf28fd56ca8c0255f241dd9"} Oct 30 19:00:03.839643 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.839613 4442 generic.go:356] "Generic (PLEG): container finished" podID="fb187f07-08dd-40b1-bb89-7f4993d8494f" containerID="a8cfc77cedafdd7bcbc3b2e17a64f94b2c5c7c041e094aa5b8c351b7595a14e0" exitCode=60 Oct 30 19:00:03.839792 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.839702 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889780-qkctp" event={"ID":"fb187f07-08dd-40b1-bb89-7f4993d8494f","Type":"ContainerDied","Data":"a8cfc77cedafdd7bcbc3b2e17a64f94b2c5c7c041e094aa5b8c351b7595a14e0"} Oct 30 19:00:03.840003 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.839978 4442 scope.go:122] "RemoveContainer" containerID="a8cfc77cedafdd7bcbc3b2e17a64f94b2c5c7c041e094aa5b8c351b7595a14e0" Oct 30 19:00:03.841825 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.841795 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889780-h6xgl" event={"ID":"2bb028e5-0be3-4923-9894-dc8f89889799","Type":"ContainerStarted","Data":"b429fe9c4221d54f43d5142e643abb11ea30bc8e39f8d7cf319eb497fcb104b6"} Oct 30 19:00:03.845487 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.844326 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889780-bqnkh" event={"ID":"6e894ed2-7f33-42f4-aa19-9ba21268f296","Type":"ContainerStarted","Data":"9c29d31d40f934f0f286b60e930b17b99f22a2f4e4d5187a49f7dbb6fa58efc8"} Oct 30 19:00:03.847059 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.847031 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2820d9-29889780-x8hc4" event={"ID":"88daa438-101a-4adc-8307-727c4176c705","Type":"ContainerStarted","Data":"cb203d532efa3611ad31a66d87f20e0f3519efa72e2aba11f4b693639a742f9f"} Oct 30 19:00:03.850653 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.850629 4442 generic.go:356] "Generic (PLEG): container finished" podID="4072bb06-edab-49c8-b213-a381823ec644" containerID="4371c4bb73d393fc7da67924ecea76e8b01d98a5120b33ef4a41ad9ba17810e3" exitCode=60 Oct 30 19:00:03.850795 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.850726 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9f6b7c-29889780-7dcw7" event={"ID":"4072bb06-edab-49c8-b213-a381823ec644","Type":"ContainerDied","Data":"4371c4bb73d393fc7da67924ecea76e8b01d98a5120b33ef4a41ad9ba17810e3"} Oct 30 19:00:03.851048 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.851028 4442 scope.go:122] "RemoveContainer" containerID="4371c4bb73d393fc7da67924ecea76e8b01d98a5120b33ef4a41ad9ba17810e3" Oct 30 19:00:03.853592 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.853567 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889780-kp5fc" event={"ID":"2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81","Type":"ContainerStarted","Data":"bed6c23990b5fbc950e083dc49cf78746bcf8609f2b2d06ebfac7934c328083f"} Oct 30 19:00:03.855710 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.855653 4442 generic.go:356] "Generic (PLEG): container finished" podID="9fcfbcab-a54b-47d4-957b-76d2cb2444f1" containerID="14e8c9bffdbba5b16efe0d00da19d272d6e218dd12a661b086388637758f8af7" exitCode=60 Oct 30 19:00:03.855887 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.855755 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889780-vnmx9" event={"ID":"9fcfbcab-a54b-47d4-957b-76d2cb2444f1","Type":"ContainerDied","Data":"14e8c9bffdbba5b16efe0d00da19d272d6e218dd12a661b086388637758f8af7"} Oct 30 19:00:03.856054 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.856015 4442 scope.go:122] "RemoveContainer" containerID="14e8c9bffdbba5b16efe0d00da19d272d6e218dd12a661b086388637758f8af7" Oct 30 19:00:03.859291 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.859251 4442 generic.go:356] "Generic (PLEG): container finished" podID="322ab2aa-d4f3-42ef-9820-9438ced76d6b" containerID="3cc41a06c0eb569b636da088cb2653697fd1d6f1e70aa97604ce1d5026176135" exitCode=60 Oct 30 19:00:03.859442 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.859350 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889780-jsvns" event={"ID":"322ab2aa-d4f3-42ef-9820-9438ced76d6b","Type":"ContainerDied","Data":"3cc41a06c0eb569b636da088cb2653697fd1d6f1e70aa97604ce1d5026176135"} Oct 30 19:00:03.859673 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.859652 4442 scope.go:122] "RemoveContainer" containerID="3cc41a06c0eb569b636da088cb2653697fd1d6f1e70aa97604ce1d5026176135" Oct 30 19:00:03.864525 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.864500 4442 generic.go:356] "Generic (PLEG): container finished" podID="ff29d9e8-03a0-4716-840d-a32e8a5aa628" containerID="eb814add1c94107343d202cd84bdd6636d1e254356a1a2722cb7a9785b70275e" exitCode=60 Oct 30 19:00:03.864608 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.864575 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889780-z79ct" event={"ID":"ff29d9e8-03a0-4716-840d-a32e8a5aa628","Type":"ContainerDied","Data":"eb814add1c94107343d202cd84bdd6636d1e254356a1a2722cb7a9785b70275e"} Oct 30 19:00:03.864897 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.864883 4442 scope.go:122] "RemoveContainer" containerID="eb814add1c94107343d202cd84bdd6636d1e254356a1a2722cb7a9785b70275e" Oct 30 19:00:03.866832 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.866804 4442 generic.go:356] "Generic (PLEG): container finished" podID="00d6e25c-6e65-45de-a894-a6d1989bfd46" containerID="fb926ac2c27a96c07d5374869e83fbc7306d680a5224b0f762cac7836141f8d5" exitCode=60 Oct 30 19:00:03.867059 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.867034 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1aa59a-29889780-bw98v" event={"ID":"00d6e25c-6e65-45de-a894-a6d1989bfd46","Type":"ContainerDied","Data":"fb926ac2c27a96c07d5374869e83fbc7306d680a5224b0f762cac7836141f8d5"} Oct 30 19:00:03.870868 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.867534 4442 scope.go:122] "RemoveContainer" containerID="fb926ac2c27a96c07d5374869e83fbc7306d680a5224b0f762cac7836141f8d5" Oct 30 19:00:03.870868 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.869474 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-m94vs" event={"ID":"525901b4-d3b5-4bd3-8458-b4f27ec0aef2","Type":"ContainerStarted","Data":"7f7d5e73dbdc9ee291422ae569752e401840e2d1f1870f0d2b09f472c56770a8"} Oct 30 19:00:03.872243 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.872217 4442 generic.go:356] "Generic (PLEG): container finished" podID="a2ba9730-da34-4745-875f-cd941e32eb79" containerID="9f5ef389355d262f3e7ef646408d257619247aea96ecc6b16b6f2d87a0c23e9f" exitCode=60 Oct 30 19:00:03.872356 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.872300 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889780-2hsw5" event={"ID":"a2ba9730-da34-4745-875f-cd941e32eb79","Type":"ContainerDied","Data":"9f5ef389355d262f3e7ef646408d257619247aea96ecc6b16b6f2d87a0c23e9f"} Oct 30 19:00:03.872547 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.872530 4442 scope.go:122] "RemoveContainer" containerID="9f5ef389355d262f3e7ef646408d257619247aea96ecc6b16b6f2d87a0c23e9f" Oct 30 19:00:03.874324 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.874297 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889780-xztsk" event={"ID":"5abb7ef5-cc99-469a-a165-9e9394370418","Type":"ContainerStarted","Data":"8e381723db3630ee2b7c63cbef32702850bbdf543adcd012f5c6803c134e1db1"} Oct 30 19:00:03.879389 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.879363 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889780-7xcv7" event={"ID":"39a6d924-7557-4f9b-b481-4398a402b1bd","Type":"ContainerStarted","Data":"423e03aa0125e80503edac7d5dd353721764a72c68cce872b2add5bea522d6bc"} Oct 30 19:00:03.887513 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.885681 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889780-xvmhb" event={"ID":"9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1","Type":"ContainerStarted","Data":"077003e3458872d0b80936e82f79459a3b11717c618ac5bb59f7be9995011435"} Oct 30 19:00:03.934318 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.931583 4442 scope.go:122] "RemoveContainer" containerID="befaf9497ac79c29b6005be15ed5ac671308122d37941c11afcee31e71d54214" Oct 30 19:00:03.934574 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:03.934329 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"befaf9497ac79c29b6005be15ed5ac671308122d37941c11afcee31e71d54214\": container with ID starting with befaf9497ac79c29b6005be15ed5ac671308122d37941c11afcee31e71d54214 not found: ID does not exist" containerID="befaf9497ac79c29b6005be15ed5ac671308122d37941c11afcee31e71d54214" Oct 30 19:00:03.934574 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.934372 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"befaf9497ac79c29b6005be15ed5ac671308122d37941c11afcee31e71d54214"} err="failed to get container status \"befaf9497ac79c29b6005be15ed5ac671308122d37941c11afcee31e71d54214\": rpc error: code = NotFound desc = could not find container \"befaf9497ac79c29b6005be15ed5ac671308122d37941c11afcee31e71d54214\": container with ID starting with befaf9497ac79c29b6005be15ed5ac671308122d37941c11afcee31e71d54214 not found: ID does not exist" Oct 30 19:00:03.940339 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.940288 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f37ca7-29889780-j2k5m" podStartSLOduration=3.940268864 podStartE2EDuration="3.940268864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:03.931389976 +0000 UTC m=+3701.021452315" watchObservedRunningTime="2026-10-30 19:00:03.940268864 +0000 UTC m=+3701.030331205" Oct 30 19:00:03.974972 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:03.974906 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889780-bpkmv" podStartSLOduration=3.97488406 podStartE2EDuration="3.97488406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:03.970981707 +0000 UTC m=+3701.061044045" watchObservedRunningTime="2026-10-30 19:00:03.97488406 +0000 UTC m=+3701.064946395" Oct 30 19:00:04.110158 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:04.105863 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889780-bqnkh" podStartSLOduration=4.105845155 podStartE2EDuration="4.105845155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:04.06531747 +0000 UTC m=+3701.155379833" watchObservedRunningTime="2026-10-30 19:00:04.105845155 +0000 UTC m=+3701.195907494" Oct 30 19:00:04.147164 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:04.146031 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889780-h6xgl" podStartSLOduration=4.146006323 podStartE2EDuration="4.146006323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:04.105222271 +0000 UTC m=+3701.195284611" watchObservedRunningTime="2026-10-30 19:00:04.146006323 +0000 UTC m=+3701.236068663" Oct 30 19:00:04.284165 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:04.237051 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889780-xztsk" podStartSLOduration=4.237029284 podStartE2EDuration="4.237029284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:04.16252609 +0000 UTC m=+3701.252588430" watchObservedRunningTime="2026-10-30 19:00:04.237029284 +0000 UTC m=+3701.327091625" Oct 30 19:00:04.468889 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:04.468838 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2820d9-29889780-x8hc4" podStartSLOduration=4.468823677 podStartE2EDuration="4.468823677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:04.353552446 +0000 UTC m=+3701.443614826" watchObservedRunningTime="2026-10-30 19:00:04.468823677 +0000 UTC m=+3701.558886016" Oct 30 19:00:04.480778 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:04.480720 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889780-kp5fc" podStartSLOduration=4.48069904 podStartE2EDuration="4.48069904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:04.393815761 +0000 UTC m=+3701.483878101" watchObservedRunningTime="2026-10-30 19:00:04.48069904 +0000 UTC m=+3701.570761381" Oct 30 19:00:04.496010 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:04.495947 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889780-7xcv7" podStartSLOduration=4.495926148 podStartE2EDuration="4.495926148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:04.466770018 +0000 UTC m=+3701.556832355" watchObservedRunningTime="2026-10-30 19:00:04.495926148 +0000 UTC m=+3701.585988486" Oct 30 19:00:04.557972 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:04.557911 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-m94vs" podStartSLOduration=4.557890209 podStartE2EDuration="4.557890209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:04.54479072 +0000 UTC m=+3701.634853063" watchObservedRunningTime="2026-10-30 19:00:04.557890209 +0000 UTC m=+3701.647952551" Oct 30 19:00:04.593787 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:04.593729 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889780-xvmhb" podStartSLOduration=4.59370845 podStartE2EDuration="4.59370845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:04.592458772 +0000 UTC m=+3701.682521108" watchObservedRunningTime="2026-10-30 19:00:04.59370845 +0000 UTC m=+3701.683770789" Oct 30 19:00:04.713122 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:04.713091 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9"] Oct 30 19:00:04.862049 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:04.861314 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-2pbp9"] Oct 30 19:00:04.964698 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:04.963106 4442 generic.go:356] "Generic (PLEG): container finished" podID="2bb028e5-0be3-4923-9894-dc8f89889799" containerID="b429fe9c4221d54f43d5142e643abb11ea30bc8e39f8d7cf319eb497fcb104b6" exitCode=60 Oct 30 19:00:04.964698 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:04.963237 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889780-h6xgl" event={"ID":"2bb028e5-0be3-4923-9894-dc8f89889799","Type":"ContainerDied","Data":"b429fe9c4221d54f43d5142e643abb11ea30bc8e39f8d7cf319eb497fcb104b6"} Oct 30 19:00:04.964698 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:04.963775 4442 scope.go:122] "RemoveContainer" containerID="b429fe9c4221d54f43d5142e643abb11ea30bc8e39f8d7cf319eb497fcb104b6" Oct 30 19:00:04.992200 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:04.987655 4442 generic.go:356] "Generic (PLEG): container finished" podID="6e894ed2-7f33-42f4-aa19-9ba21268f296" containerID="9c29d31d40f934f0f286b60e930b17b99f22a2f4e4d5187a49f7dbb6fa58efc8" exitCode=60 Oct 30 19:00:04.992200 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:04.987769 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889780-bqnkh" event={"ID":"6e894ed2-7f33-42f4-aa19-9ba21268f296","Type":"ContainerDied","Data":"9c29d31d40f934f0f286b60e930b17b99f22a2f4e4d5187a49f7dbb6fa58efc8"} Oct 30 19:00:04.992200 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:04.988341 4442 scope.go:122] "RemoveContainer" containerID="9c29d31d40f934f0f286b60e930b17b99f22a2f4e4d5187a49f7dbb6fa58efc8" Oct 30 19:00:05.122677 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:05.093593 4442 generic.go:356] "Generic (PLEG): container finished" podID="88daa438-101a-4adc-8307-727c4176c705" containerID="cb203d532efa3611ad31a66d87f20e0f3519efa72e2aba11f4b693639a742f9f" exitCode=60 Oct 30 19:00:05.122677 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:05.100564 4442 scope.go:122] "RemoveContainer" containerID="cb203d532efa3611ad31a66d87f20e0f3519efa72e2aba11f4b693639a742f9f" Oct 30 19:00:05.128437 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:05.128112 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="418064ff-dabc-4b46-b2cf-133f7f20e9d7" path="/var/lib/kubelet/pods/418064ff-dabc-4b46-b2cf-133f7f20e9d7/volumes" Oct 30 19:00:05.128947 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:05.128913 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2820d9-29889780-x8hc4" event={"ID":"88daa438-101a-4adc-8307-727c4176c705","Type":"ContainerDied","Data":"cb203d532efa3611ad31a66d87f20e0f3519efa72e2aba11f4b693639a742f9f"} Oct 30 19:00:05.165954 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:05.165921 4442 generic.go:356] "Generic (PLEG): container finished" podID="2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81" containerID="bed6c23990b5fbc950e083dc49cf78746bcf8609f2b2d06ebfac7934c328083f" exitCode=60 Oct 30 19:00:05.166303 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:05.165926 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889780-kp5fc" event={"ID":"2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81","Type":"ContainerDied","Data":"bed6c23990b5fbc950e083dc49cf78746bcf8609f2b2d06ebfac7934c328083f"} Oct 30 19:00:05.166688 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:05.166669 4442 scope.go:122] "RemoveContainer" containerID="bfbe09e315456c4763aeecb5b5f9d344a35cafca48df6e78cf3952145d717793" Oct 30 19:00:05.167074 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:05.167050 4442 scope.go:122] "RemoveContainer" containerID="bed6c23990b5fbc950e083dc49cf78746bcf8609f2b2d06ebfac7934c328083f" Oct 30 19:00:05.168576 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:05.168547 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889780-kp5fc_odh-ai-gateway-infra(2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889780-kp5fc" podUID="2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81" Oct 30 19:00:05.187281 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:05.187237 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889780-vnmx9" event={"ID":"9fcfbcab-a54b-47d4-957b-76d2cb2444f1","Type":"ContainerStarted","Data":"7d9647608b1d512f3eed071f0504e94b91fa41051cdc81bb3664360c5404baeb"} Oct 30 19:00:05.192143 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:05.192097 4442 generic.go:356] "Generic (PLEG): container finished" podID="525901b4-d3b5-4bd3-8458-b4f27ec0aef2" containerID="7f7d5e73dbdc9ee291422ae569752e401840e2d1f1870f0d2b09f472c56770a8" exitCode=60 Oct 30 19:00:05.192278 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:05.192167 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-m94vs" event={"ID":"525901b4-d3b5-4bd3-8458-b4f27ec0aef2","Type":"ContainerDied","Data":"7f7d5e73dbdc9ee291422ae569752e401840e2d1f1870f0d2b09f472c56770a8"} Oct 30 19:00:05.192590 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:05.192570 4442 scope.go:122] "RemoveContainer" containerID="7f7d5e73dbdc9ee291422ae569752e401840e2d1f1870f0d2b09f472c56770a8" Oct 30 19:00:05.196330 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:05.196300 4442 generic.go:356] "Generic (PLEG): container finished" podID="5abb7ef5-cc99-469a-a165-9e9394370418" containerID="8e381723db3630ee2b7c63cbef32702850bbdf543adcd012f5c6803c134e1db1" exitCode=60 Oct 30 19:00:05.196444 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:05.196412 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889780-xztsk" event={"ID":"5abb7ef5-cc99-469a-a165-9e9394370418","Type":"ContainerDied","Data":"8e381723db3630ee2b7c63cbef32702850bbdf543adcd012f5c6803c134e1db1"} Oct 30 19:00:05.196810 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:05.196687 4442 scope.go:122] "RemoveContainer" containerID="8e381723db3630ee2b7c63cbef32702850bbdf543adcd012f5c6803c134e1db1" Oct 30 19:00:05.199741 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:05.199716 4442 generic.go:356] "Generic (PLEG): container finished" podID="39a6d924-7557-4f9b-b481-4398a402b1bd" containerID="423e03aa0125e80503edac7d5dd353721764a72c68cce872b2add5bea522d6bc" exitCode=60 Oct 30 19:00:05.199928 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:05.199911 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889780-7xcv7" event={"ID":"39a6d924-7557-4f9b-b481-4398a402b1bd","Type":"ContainerDied","Data":"423e03aa0125e80503edac7d5dd353721764a72c68cce872b2add5bea522d6bc"} Oct 30 19:00:05.201379 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:05.201363 4442 scope.go:122] "RemoveContainer" containerID="423e03aa0125e80503edac7d5dd353721764a72c68cce872b2add5bea522d6bc" Oct 30 19:00:05.207265 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:05.207229 4442 generic.go:356] "Generic (PLEG): container finished" podID="9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1" containerID="077003e3458872d0b80936e82f79459a3b11717c618ac5bb59f7be9995011435" exitCode=60 Oct 30 19:00:05.207378 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:05.207321 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889780-xvmhb" event={"ID":"9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1","Type":"ContainerDied","Data":"077003e3458872d0b80936e82f79459a3b11717c618ac5bb59f7be9995011435"} Oct 30 19:00:05.207666 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:05.207627 4442 scope.go:122] "RemoveContainer" containerID="077003e3458872d0b80936e82f79459a3b11717c618ac5bb59f7be9995011435" Oct 30 19:00:05.213258 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:05.212372 4442 generic.go:356] "Generic (PLEG): container finished" podID="4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2" containerID="549fd59797e7104fa04e3b03f49b0bb69712dfeed22249625c97602ccd4b707d" exitCode=60 Oct 30 19:00:05.213258 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:05.212470 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889780-bpkmv" event={"ID":"4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2","Type":"ContainerDied","Data":"549fd59797e7104fa04e3b03f49b0bb69712dfeed22249625c97602ccd4b707d"} Oct 30 19:00:05.213258 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:05.212800 4442 scope.go:122] "RemoveContainer" containerID="549fd59797e7104fa04e3b03f49b0bb69712dfeed22249625c97602ccd4b707d" Oct 30 19:00:05.215940 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:05.215909 4442 generic.go:356] "Generic (PLEG): container finished" podID="1e7491f9-b178-456d-9a56-3eb6b5ee2e40" containerID="611c1df0460515aea951048d5141f8139261e058fdf28fd56ca8c0255f241dd9" exitCode=60 Oct 30 19:00:05.216018 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:05.215950 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f37ca7-29889780-j2k5m" event={"ID":"1e7491f9-b178-456d-9a56-3eb6b5ee2e40","Type":"ContainerDied","Data":"611c1df0460515aea951048d5141f8139261e058fdf28fd56ca8c0255f241dd9"} Oct 30 19:00:05.216691 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:05.216674 4442 scope.go:122] "RemoveContainer" containerID="611c1df0460515aea951048d5141f8139261e058fdf28fd56ca8c0255f241dd9" Oct 30 19:00:05.222244 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:05.222217 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889780-qkctp" event={"ID":"fb187f07-08dd-40b1-bb89-7f4993d8494f","Type":"ContainerStarted","Data":"c8c4bbf0374dbf36b3b0a51afb54f9a7189dc34742c46de60df34f56b199acd4"} Oct 30 19:00:05.283057 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:05.283004 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889780-vnmx9" podStartSLOduration=5.282983843 podStartE2EDuration="5.282983843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:05.282282228 +0000 UTC m=+3702.372344568" watchObservedRunningTime="2026-10-30 19:00:05.282983843 +0000 UTC m=+3702.373046189" Oct 30 19:00:06.242623 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:06.242588 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2820d9-29889780-x8hc4" event={"ID":"88daa438-101a-4adc-8307-727c4176c705","Type":"ContainerStarted","Data":"c83e53807de77746da4bd90f979125e4e54ac2188de143bdafa02c48dc17f624"} Oct 30 19:00:06.243295 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:06.243270 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:00:06.243435 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:06.243318 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" podUID="04780470-66fb-45c6-b86e-8befadb35217" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:00:06.245175 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:06.245148 4442 generic.go:356] "Generic (PLEG): container finished" podID="4072bb06-edab-49c8-b213-a381823ec644" containerID="9fcc9137488fd59b8a93b804d1632f1fd8ddc0c8b0222a248a0fef1bffea3d94" exitCode=60 Oct 30 19:00:06.245282 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:06.245229 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9f6b7c-29889780-7dcw7" event={"ID":"4072bb06-edab-49c8-b213-a381823ec644","Type":"ContainerDied","Data":"9fcc9137488fd59b8a93b804d1632f1fd8ddc0c8b0222a248a0fef1bffea3d94"} Oct 30 19:00:06.245346 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:06.245289 4442 scope.go:122] "RemoveContainer" containerID="4371c4bb73d393fc7da67924ecea76e8b01d98a5120b33ef4a41ad9ba17810e3" Oct 30 19:00:06.245603 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:06.245587 4442 scope.go:122] "RemoveContainer" containerID="9fcc9137488fd59b8a93b804d1632f1fd8ddc0c8b0222a248a0fef1bffea3d94" Oct 30 19:00:06.245995 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:06.245972 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9f6b7c-29889780-7dcw7_odh-ai-gateway-infra(4072bb06-edab-49c8-b213-a381823ec644)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9f6b7c-29889780-7dcw7" podUID="4072bb06-edab-49c8-b213-a381823ec644" Oct 30 19:00:06.249583 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:06.249554 4442 scope.go:122] "RemoveContainer" containerID="bed6c23990b5fbc950e083dc49cf78746bcf8609f2b2d06ebfac7934c328083f" Oct 30 19:00:06.250021 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:06.249990 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889780-kp5fc_odh-ai-gateway-infra(2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889780-kp5fc" podUID="2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81" Oct 30 19:00:06.250902 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:06.250880 4442 generic.go:356] "Generic (PLEG): container finished" podID="9fcfbcab-a54b-47d4-957b-76d2cb2444f1" containerID="7d9647608b1d512f3eed071f0504e94b91fa41051cdc81bb3664360c5404baeb" exitCode=60 Oct 30 19:00:06.250989 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:06.250970 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889780-vnmx9" event={"ID":"9fcfbcab-a54b-47d4-957b-76d2cb2444f1","Type":"ContainerDied","Data":"7d9647608b1d512f3eed071f0504e94b91fa41051cdc81bb3664360c5404baeb"} Oct 30 19:00:06.251590 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:06.251432 4442 scope.go:122] "RemoveContainer" containerID="7d9647608b1d512f3eed071f0504e94b91fa41051cdc81bb3664360c5404baeb" Oct 30 19:00:06.251947 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:06.251874 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-921450-29889780-vnmx9_odh-ai-gateway-infra(9fcfbcab-a54b-47d4-957b-76d2cb2444f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889780-vnmx9" podUID="9fcfbcab-a54b-47d4-957b-76d2cb2444f1" Oct 30 19:00:06.258095 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:06.257285 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889780-jsvns" event={"ID":"322ab2aa-d4f3-42ef-9820-9438ced76d6b","Type":"ContainerStarted","Data":"c4b00b6eecaa267a1cd1a860d910fa883ccd9713feb6570d0038debab4031fba"} Oct 30 19:00:06.259999 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:06.259974 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889780-z79ct" event={"ID":"ff29d9e8-03a0-4716-840d-a32e8a5aa628","Type":"ContainerStarted","Data":"53d61493663c21940b45f5bd949cda482c0f0c2c7578366d10c72c72fd989f2b"} Oct 30 19:00:06.263295 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:06.263271 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1aa59a-29889780-bw98v" event={"ID":"00d6e25c-6e65-45de-a894-a6d1989bfd46","Type":"ContainerStarted","Data":"02a8c9cae50e24ef176091696fcb817d7b257fa0aa610e2e13e20407e045b07e"} Oct 30 19:00:06.266263 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:06.266233 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889780-2hsw5" event={"ID":"a2ba9730-da34-4745-875f-cd941e32eb79","Type":"ContainerStarted","Data":"efa9992046cd918168968a92b81dfd8a8db688c6e01d2408adda15c16b1e4802"} Oct 30 19:00:06.267427 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:06.267403 4442 scope.go:122] "RemoveContainer" containerID="14e8c9bffdbba5b16efe0d00da19d272d6e218dd12a661b086388637758f8af7" Oct 30 19:00:06.268848 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:06.268823 4442 generic.go:356] "Generic (PLEG): container finished" podID="fb187f07-08dd-40b1-bb89-7f4993d8494f" containerID="c8c4bbf0374dbf36b3b0a51afb54f9a7189dc34742c46de60df34f56b199acd4" exitCode=60 Oct 30 19:00:06.268973 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:06.268951 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889780-qkctp" event={"ID":"fb187f07-08dd-40b1-bb89-7f4993d8494f","Type":"ContainerDied","Data":"c8c4bbf0374dbf36b3b0a51afb54f9a7189dc34742c46de60df34f56b199acd4"} Oct 30 19:00:06.269243 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:06.269221 4442 scope.go:122] "RemoveContainer" containerID="c8c4bbf0374dbf36b3b0a51afb54f9a7189dc34742c46de60df34f56b199acd4" Oct 30 19:00:06.269576 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:06.269555 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889780-qkctp_odh-ai-gateway-infra(fb187f07-08dd-40b1-bb89-7f4993d8494f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889780-qkctp" podUID="fb187f07-08dd-40b1-bb89-7f4993d8494f" Oct 30 19:00:06.271189 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:06.271163 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889780-h6xgl" event={"ID":"2bb028e5-0be3-4923-9894-dc8f89889799","Type":"ContainerStarted","Data":"762c977776ca5bddcad3b1ec3c9fa5a975452d72a71c7c677f54340c5066371b"} Oct 30 19:00:06.283973 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:06.283954 4442 scope.go:122] "RemoveContainer" containerID="a8cfc77cedafdd7bcbc3b2e17a64f94b2c5c7c041e094aa5b8c351b7595a14e0" Oct 30 19:00:06.474673 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:06.474616 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889780-jsvns" podStartSLOduration=6.474601894 podStartE2EDuration="6.474601894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:06.470536174 +0000 UTC m=+3703.560598514" watchObservedRunningTime="2026-10-30 19:00:06.474601894 +0000 UTC m=+3703.564664232" Oct 30 19:00:06.513444 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:06.513328 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1aa59a-29889780-bw98v" podStartSLOduration=6.51331055 podStartE2EDuration="6.51331055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:06.50722112 +0000 UTC m=+3703.597283461" watchObservedRunningTime="2026-10-30 19:00:06.51331055 +0000 UTC m=+3703.603372889" Oct 30 19:00:06.632842 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:06.632779 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889780-z79ct" podStartSLOduration=6.632759034 podStartE2EDuration="6.632759034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:06.629624968 +0000 UTC m=+3703.719687308" watchObservedRunningTime="2026-10-30 19:00:06.632759034 +0000 UTC m=+3703.722821374" Oct 30 19:00:07.292181 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.292149 4442 generic.go:356] "Generic (PLEG): container finished" podID="88daa438-101a-4adc-8307-727c4176c705" containerID="c83e53807de77746da4bd90f979125e4e54ac2188de143bdafa02c48dc17f624" exitCode=60 Oct 30 19:00:07.292181 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.292166 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2820d9-29889780-x8hc4" event={"ID":"88daa438-101a-4adc-8307-727c4176c705","Type":"ContainerDied","Data":"c83e53807de77746da4bd90f979125e4e54ac2188de143bdafa02c48dc17f624"} Oct 30 19:00:07.292717 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.292229 4442 scope.go:122] "RemoveContainer" containerID="cb203d532efa3611ad31a66d87f20e0f3519efa72e2aba11f4b693639a742f9f" Oct 30 19:00:07.292717 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.292617 4442 scope.go:122] "RemoveContainer" containerID="c83e53807de77746da4bd90f979125e4e54ac2188de143bdafa02c48dc17f624" Oct 30 19:00:07.293111 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:07.293087 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-2820d9-29889780-x8hc4_odh-ai-gateway-infra(88daa438-101a-4adc-8307-727c4176c705)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2820d9-29889780-x8hc4" podUID="88daa438-101a-4adc-8307-727c4176c705" Oct 30 19:00:07.296938 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.296917 4442 scope.go:122] "RemoveContainer" containerID="9fcc9137488fd59b8a93b804d1632f1fd8ddc0c8b0222a248a0fef1bffea3d94" Oct 30 19:00:07.297386 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:07.297362 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9f6b7c-29889780-7dcw7_odh-ai-gateway-infra(4072bb06-edab-49c8-b213-a381823ec644)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9f6b7c-29889780-7dcw7" podUID="4072bb06-edab-49c8-b213-a381823ec644" Oct 30 19:00:07.299419 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.299395 4442 scope.go:122] "RemoveContainer" containerID="7d9647608b1d512f3eed071f0504e94b91fa41051cdc81bb3664360c5404baeb" Oct 30 19:00:07.299854 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:07.299790 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-921450-29889780-vnmx9_odh-ai-gateway-infra(9fcfbcab-a54b-47d4-957b-76d2cb2444f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889780-vnmx9" podUID="9fcfbcab-a54b-47d4-957b-76d2cb2444f1" Oct 30 19:00:07.301294 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.301267 4442 generic.go:356] "Generic (PLEG): container finished" podID="322ab2aa-d4f3-42ef-9820-9438ced76d6b" containerID="c4b00b6eecaa267a1cd1a860d910fa883ccd9713feb6570d0038debab4031fba" exitCode=60 Oct 30 19:00:07.301412 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.301389 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889780-jsvns" event={"ID":"322ab2aa-d4f3-42ef-9820-9438ced76d6b","Type":"ContainerDied","Data":"c4b00b6eecaa267a1cd1a860d910fa883ccd9713feb6570d0038debab4031fba"} Oct 30 19:00:07.301748 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.301728 4442 scope.go:122] "RemoveContainer" containerID="c4b00b6eecaa267a1cd1a860d910fa883ccd9713feb6570d0038debab4031fba" Oct 30 19:00:07.302257 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:07.302231 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889780-jsvns_odh-ai-gateway-infra(322ab2aa-d4f3-42ef-9820-9438ced76d6b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889780-jsvns" podUID="322ab2aa-d4f3-42ef-9820-9438ced76d6b" Oct 30 19:00:07.304084 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.304061 4442 generic.go:356] "Generic (PLEG): container finished" podID="ff29d9e8-03a0-4716-840d-a32e8a5aa628" containerID="53d61493663c21940b45f5bd949cda482c0f0c2c7578366d10c72c72fd989f2b" exitCode=60 Oct 30 19:00:07.304239 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.304155 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889780-z79ct" event={"ID":"ff29d9e8-03a0-4716-840d-a32e8a5aa628","Type":"ContainerDied","Data":"53d61493663c21940b45f5bd949cda482c0f0c2c7578366d10c72c72fd989f2b"} Oct 30 19:00:07.304510 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.304497 4442 scope.go:122] "RemoveContainer" containerID="53d61493663c21940b45f5bd949cda482c0f0c2c7578366d10c72c72fd989f2b" Oct 30 19:00:07.304909 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:07.304885 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889780-z79ct_odh-ai-gateway-infra(ff29d9e8-03a0-4716-840d-a32e8a5aa628)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889780-z79ct" podUID="ff29d9e8-03a0-4716-840d-a32e8a5aa628" Oct 30 19:00:07.306538 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.306517 4442 generic.go:356] "Generic (PLEG): container finished" podID="00d6e25c-6e65-45de-a894-a6d1989bfd46" containerID="02a8c9cae50e24ef176091696fcb817d7b257fa0aa610e2e13e20407e045b07e" exitCode=60 Oct 30 19:00:07.306645 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.306585 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1aa59a-29889780-bw98v" event={"ID":"00d6e25c-6e65-45de-a894-a6d1989bfd46","Type":"ContainerDied","Data":"02a8c9cae50e24ef176091696fcb817d7b257fa0aa610e2e13e20407e045b07e"} Oct 30 19:00:07.306967 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.306945 4442 scope.go:122] "RemoveContainer" containerID="02a8c9cae50e24ef176091696fcb817d7b257fa0aa610e2e13e20407e045b07e" Oct 30 19:00:07.307415 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:07.307390 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-1aa59a-29889780-bw98v_odh-ai-gateway-infra(00d6e25c-6e65-45de-a894-a6d1989bfd46)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1aa59a-29889780-bw98v" podUID="00d6e25c-6e65-45de-a894-a6d1989bfd46" Oct 30 19:00:07.309010 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.308989 4442 generic.go:356] "Generic (PLEG): container finished" podID="525901b4-d3b5-4bd3-8458-b4f27ec0aef2" containerID="884e8e2d0639c60f86eb19b9321784376270cab9f2f3554b1c7e478da4df04e8" exitCode=60 Oct 30 19:00:07.309116 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.309068 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-m94vs" event={"ID":"525901b4-d3b5-4bd3-8458-b4f27ec0aef2","Type":"ContainerDied","Data":"884e8e2d0639c60f86eb19b9321784376270cab9f2f3554b1c7e478da4df04e8"} Oct 30 19:00:07.309383 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.309368 4442 scope.go:122] "RemoveContainer" containerID="884e8e2d0639c60f86eb19b9321784376270cab9f2f3554b1c7e478da4df04e8" Oct 30 19:00:07.309631 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:07.309607 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889780-m94vs_odh-ai-gateway-infra(525901b4-d3b5-4bd3-8458-b4f27ec0aef2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-m94vs" podUID="525901b4-d3b5-4bd3-8458-b4f27ec0aef2" Oct 30 19:00:07.311489 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.311467 4442 generic.go:356] "Generic (PLEG): container finished" podID="a2ba9730-da34-4745-875f-cd941e32eb79" containerID="efa9992046cd918168968a92b81dfd8a8db688c6e01d2408adda15c16b1e4802" exitCode=60 Oct 30 19:00:07.311591 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.311549 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889780-2hsw5" event={"ID":"a2ba9730-da34-4745-875f-cd941e32eb79","Type":"ContainerDied","Data":"efa9992046cd918168968a92b81dfd8a8db688c6e01d2408adda15c16b1e4802"} Oct 30 19:00:07.311875 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.311858 4442 scope.go:122] "RemoveContainer" containerID="efa9992046cd918168968a92b81dfd8a8db688c6e01d2408adda15c16b1e4802" Oct 30 19:00:07.312251 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:07.312223 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-38e0db-29889780-2hsw5_odh-ai-gateway-infra(a2ba9730-da34-4745-875f-cd941e32eb79)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889780-2hsw5" podUID="a2ba9730-da34-4745-875f-cd941e32eb79" Oct 30 19:00:07.313994 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.313973 4442 generic.go:356] "Generic (PLEG): container finished" podID="5abb7ef5-cc99-469a-a165-9e9394370418" containerID="56036f5498cdd52d2b5a261c9b72bd9406acfd052381e44ec3c5a80337a49758" exitCode=60 Oct 30 19:00:07.314092 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.314000 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889780-xztsk" event={"ID":"5abb7ef5-cc99-469a-a165-9e9394370418","Type":"ContainerDied","Data":"56036f5498cdd52d2b5a261c9b72bd9406acfd052381e44ec3c5a80337a49758"} Oct 30 19:00:07.314383 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.314367 4442 scope.go:122] "RemoveContainer" containerID="56036f5498cdd52d2b5a261c9b72bd9406acfd052381e44ec3c5a80337a49758" Oct 30 19:00:07.314690 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:07.314669 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889780-xztsk_odh-ai-gateway-infra(5abb7ef5-cc99-469a-a165-9e9394370418)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889780-xztsk" podUID="5abb7ef5-cc99-469a-a165-9e9394370418" Oct 30 19:00:07.316457 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.316434 4442 generic.go:356] "Generic (PLEG): container finished" podID="39a6d924-7557-4f9b-b481-4398a402b1bd" containerID="bd091ebf501fa386172839cd120584d9cf257de6d0b7b9ed01a60abbfb03e594" exitCode=60 Oct 30 19:00:07.316549 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.316512 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889780-7xcv7" event={"ID":"39a6d924-7557-4f9b-b481-4398a402b1bd","Type":"ContainerDied","Data":"bd091ebf501fa386172839cd120584d9cf257de6d0b7b9ed01a60abbfb03e594"} Oct 30 19:00:07.316755 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.316742 4442 scope.go:122] "RemoveContainer" containerID="bd091ebf501fa386172839cd120584d9cf257de6d0b7b9ed01a60abbfb03e594" Oct 30 19:00:07.317034 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:07.317011 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-39c99d-29889780-7xcv7_odh-ai-gateway-infra(39a6d924-7557-4f9b-b481-4398a402b1bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889780-7xcv7" podUID="39a6d924-7557-4f9b-b481-4398a402b1bd" Oct 30 19:00:07.319451 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.319430 4442 generic.go:356] "Generic (PLEG): container finished" podID="9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1" containerID="df03fc3017b1b19ae972815bbade48c2f6fe3d23eccac93f34aa48ac602f7164" exitCode=60 Oct 30 19:00:07.319555 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.319532 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889780-xvmhb" event={"ID":"9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1","Type":"ContainerDied","Data":"df03fc3017b1b19ae972815bbade48c2f6fe3d23eccac93f34aa48ac602f7164"} Oct 30 19:00:07.319859 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.319844 4442 scope.go:122] "RemoveContainer" containerID="df03fc3017b1b19ae972815bbade48c2f6fe3d23eccac93f34aa48ac602f7164" Oct 30 19:00:07.320161 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:07.320108 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889780-xvmhb_odh-ai-gateway-infra(9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889780-xvmhb" podUID="9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1" Oct 30 19:00:07.322042 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.322019 4442 generic.go:356] "Generic (PLEG): container finished" podID="4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2" containerID="981ec931ea42bd0b0439c791c052545c585ba28be5907d69078fcae12e6da488" exitCode=60 Oct 30 19:00:07.322240 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.322072 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889780-bpkmv" event={"ID":"4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2","Type":"ContainerDied","Data":"981ec931ea42bd0b0439c791c052545c585ba28be5907d69078fcae12e6da488"} Oct 30 19:00:07.322407 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.322387 4442 scope.go:122] "RemoveContainer" containerID="981ec931ea42bd0b0439c791c052545c585ba28be5907d69078fcae12e6da488" Oct 30 19:00:07.322700 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:07.322680 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-380c08-29889780-bpkmv_odh-ai-gateway-infra(4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889780-bpkmv" podUID="4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2" Oct 30 19:00:07.327918 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.327899 4442 generic.go:356] "Generic (PLEG): container finished" podID="1e7491f9-b178-456d-9a56-3eb6b5ee2e40" containerID="2155792d1bff789bcf3694382ee657c4df4ba1758c7743d78d67b611523474f8" exitCode=60 Oct 30 19:00:07.328020 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.327990 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f37ca7-29889780-j2k5m" event={"ID":"1e7491f9-b178-456d-9a56-3eb6b5ee2e40","Type":"ContainerDied","Data":"2155792d1bff789bcf3694382ee657c4df4ba1758c7743d78d67b611523474f8"} Oct 30 19:00:07.328286 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.328267 4442 scope.go:122] "RemoveContainer" containerID="2155792d1bff789bcf3694382ee657c4df4ba1758c7743d78d67b611523474f8" Oct 30 19:00:07.328616 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:07.328589 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-f37ca7-29889780-j2k5m_odh-ai-gateway-infra(1e7491f9-b178-456d-9a56-3eb6b5ee2e40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f37ca7-29889780-j2k5m" podUID="1e7491f9-b178-456d-9a56-3eb6b5ee2e40" Oct 30 19:00:07.330046 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.330026 4442 scope.go:122] "RemoveContainer" containerID="c8c4bbf0374dbf36b3b0a51afb54f9a7189dc34742c46de60df34f56b199acd4" Oct 30 19:00:07.330360 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:07.330338 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889780-qkctp_odh-ai-gateway-infra(fb187f07-08dd-40b1-bb89-7f4993d8494f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889780-qkctp" podUID="fb187f07-08dd-40b1-bb89-7f4993d8494f" Oct 30 19:00:07.330980 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.330962 4442 scope.go:122] "RemoveContainer" containerID="3cc41a06c0eb569b636da088cb2653697fd1d6f1e70aa97604ce1d5026176135" Oct 30 19:00:07.331810 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.331789 4442 generic.go:356] "Generic (PLEG): container finished" podID="2bb028e5-0be3-4923-9894-dc8f89889799" containerID="762c977776ca5bddcad3b1ec3c9fa5a975452d72a71c7c677f54340c5066371b" exitCode=60 Oct 30 19:00:07.331928 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.331881 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889780-h6xgl" event={"ID":"2bb028e5-0be3-4923-9894-dc8f89889799","Type":"ContainerDied","Data":"762c977776ca5bddcad3b1ec3c9fa5a975452d72a71c7c677f54340c5066371b"} Oct 30 19:00:07.332099 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.332081 4442 scope.go:122] "RemoveContainer" containerID="762c977776ca5bddcad3b1ec3c9fa5a975452d72a71c7c677f54340c5066371b" Oct 30 19:00:07.332410 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:07.332388 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-45dc34-29889780-h6xgl_odh-ai-gateway-infra(2bb028e5-0be3-4923-9894-dc8f89889799)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889780-h6xgl" podUID="2bb028e5-0be3-4923-9894-dc8f89889799" Oct 30 19:00:07.333943 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.333923 4442 generic.go:356] "Generic (PLEG): container finished" podID="6e894ed2-7f33-42f4-aa19-9ba21268f296" containerID="cf4e9770f01d04d12c1fcfa31983c6bd9add848982c43ab026926afbf12c89da" exitCode=60 Oct 30 19:00:07.334013 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.333977 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889780-bqnkh" event={"ID":"6e894ed2-7f33-42f4-aa19-9ba21268f296","Type":"ContainerDied","Data":"cf4e9770f01d04d12c1fcfa31983c6bd9add848982c43ab026926afbf12c89da"} Oct 30 19:00:07.334266 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.334250 4442 scope.go:122] "RemoveContainer" containerID="cf4e9770f01d04d12c1fcfa31983c6bd9add848982c43ab026926afbf12c89da" Oct 30 19:00:07.334528 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:07.334511 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889780-bqnkh_odh-ai-gateway-infra(6e894ed2-7f33-42f4-aa19-9ba21268f296)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889780-bqnkh" podUID="6e894ed2-7f33-42f4-aa19-9ba21268f296" Oct 30 19:00:07.416795 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.416769 4442 scope.go:122] "RemoveContainer" containerID="eb814add1c94107343d202cd84bdd6636d1e254356a1a2722cb7a9785b70275e" Oct 30 19:00:07.433856 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.433831 4442 scope.go:122] "RemoveContainer" containerID="fb926ac2c27a96c07d5374869e83fbc7306d680a5224b0f762cac7836141f8d5" Oct 30 19:00:07.449848 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.449826 4442 scope.go:122] "RemoveContainer" containerID="7f7d5e73dbdc9ee291422ae569752e401840e2d1f1870f0d2b09f472c56770a8" Oct 30 19:00:07.465411 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.465387 4442 scope.go:122] "RemoveContainer" containerID="9f5ef389355d262f3e7ef646408d257619247aea96ecc6b16b6f2d87a0c23e9f" Oct 30 19:00:07.484476 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.484448 4442 scope.go:122] "RemoveContainer" containerID="8e381723db3630ee2b7c63cbef32702850bbdf543adcd012f5c6803c134e1db1" Oct 30 19:00:07.591760 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.591729 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w"] Oct 30 19:00:07.592040 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.592000 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" podUID="c7e7ff65-c137-48d2-a189-eb7f64bb1ff2" containerName="istio-proxy" containerID="cri-o://91593f9a61c1578c25f2243d3802cdf9a82469710a7eb79d449bc50d34981656" gracePeriod=30 Oct 30 19:00:07.598002 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.597977 4442 scope.go:122] "RemoveContainer" containerID="423e03aa0125e80503edac7d5dd353721764a72c68cce872b2add5bea522d6bc" Oct 30 19:00:07.621777 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.621745 4442 scope.go:122] "RemoveContainer" containerID="077003e3458872d0b80936e82f79459a3b11717c618ac5bb59f7be9995011435" Oct 30 19:00:07.659670 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.658646 4442 scope.go:122] "RemoveContainer" containerID="549fd59797e7104fa04e3b03f49b0bb69712dfeed22249625c97602ccd4b707d" Oct 30 19:00:07.908550 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:07.892350 4442 scope.go:122] "RemoveContainer" containerID="611c1df0460515aea951048d5141f8139261e058fdf28fd56ca8c0255f241dd9" Oct 30 19:00:08.086799 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.086768 4442 scope.go:122] "RemoveContainer" containerID="b429fe9c4221d54f43d5142e643abb11ea30bc8e39f8d7cf319eb497fcb104b6" Oct 30 19:00:08.138454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.137611 4442 scope.go:122] "RemoveContainer" containerID="9c29d31d40f934f0f286b60e930b17b99f22a2f4e4d5187a49f7dbb6fa58efc8" Oct 30 19:00:08.240911 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.240886 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" Oct 30 19:00:08.350449 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.350420 4442 scope.go:122] "RemoveContainer" containerID="c83e53807de77746da4bd90f979125e4e54ac2188de143bdafa02c48dc17f624" Oct 30 19:00:08.352098 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:08.352065 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-2820d9-29889780-x8hc4_odh-ai-gateway-infra(88daa438-101a-4adc-8307-727c4176c705)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2820d9-29889780-x8hc4" podUID="88daa438-101a-4adc-8307-727c4176c705" Oct 30 19:00:08.353486 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.353466 4442 scope.go:122] "RemoveContainer" containerID="c4b00b6eecaa267a1cd1a860d910fa883ccd9713feb6570d0038debab4031fba" Oct 30 19:00:08.353998 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:08.353973 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889780-jsvns_odh-ai-gateway-infra(322ab2aa-d4f3-42ef-9820-9438ced76d6b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889780-jsvns" podUID="322ab2aa-d4f3-42ef-9820-9438ced76d6b" Oct 30 19:00:08.355802 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.355781 4442 scope.go:122] "RemoveContainer" containerID="53d61493663c21940b45f5bd949cda482c0f0c2c7578366d10c72c72fd989f2b" Oct 30 19:00:08.356236 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:08.356209 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889780-z79ct_odh-ai-gateway-infra(ff29d9e8-03a0-4716-840d-a32e8a5aa628)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889780-z79ct" podUID="ff29d9e8-03a0-4716-840d-a32e8a5aa628" Oct 30 19:00:08.357580 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.357562 4442 scope.go:122] "RemoveContainer" containerID="02a8c9cae50e24ef176091696fcb817d7b257fa0aa610e2e13e20407e045b07e" Oct 30 19:00:08.357854 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:08.357833 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-1aa59a-29889780-bw98v_odh-ai-gateway-infra(00d6e25c-6e65-45de-a894-a6d1989bfd46)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1aa59a-29889780-bw98v" podUID="00d6e25c-6e65-45de-a894-a6d1989bfd46" Oct 30 19:00:08.360537 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.360520 4442 scope.go:122] "RemoveContainer" containerID="884e8e2d0639c60f86eb19b9321784376270cab9f2f3554b1c7e478da4df04e8" Oct 30 19:00:08.360792 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:08.360771 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889780-m94vs_odh-ai-gateway-infra(525901b4-d3b5-4bd3-8458-b4f27ec0aef2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-m94vs" podUID="525901b4-d3b5-4bd3-8458-b4f27ec0aef2" Oct 30 19:00:08.362693 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.362673 4442 scope.go:122] "RemoveContainer" containerID="efa9992046cd918168968a92b81dfd8a8db688c6e01d2408adda15c16b1e4802" Oct 30 19:00:08.362935 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:08.362913 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-38e0db-29889780-2hsw5_odh-ai-gateway-infra(a2ba9730-da34-4745-875f-cd941e32eb79)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889780-2hsw5" podUID="a2ba9730-da34-4745-875f-cd941e32eb79" Oct 30 19:00:08.365328 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.365309 4442 scope.go:122] "RemoveContainer" containerID="56036f5498cdd52d2b5a261c9b72bd9406acfd052381e44ec3c5a80337a49758" Oct 30 19:00:08.365566 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:08.365548 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889780-xztsk_odh-ai-gateway-infra(5abb7ef5-cc99-469a-a165-9e9394370418)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889780-xztsk" podUID="5abb7ef5-cc99-469a-a165-9e9394370418" Oct 30 19:00:08.367834 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.367817 4442 scope.go:122] "RemoveContainer" containerID="bd091ebf501fa386172839cd120584d9cf257de6d0b7b9ed01a60abbfb03e594" Oct 30 19:00:08.369443 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:08.369422 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-39c99d-29889780-7xcv7_odh-ai-gateway-infra(39a6d924-7557-4f9b-b481-4398a402b1bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889780-7xcv7" podUID="39a6d924-7557-4f9b-b481-4398a402b1bd" Oct 30 19:00:08.370990 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.370960 4442 scope.go:122] "RemoveContainer" containerID="df03fc3017b1b19ae972815bbade48c2f6fe3d23eccac93f34aa48ac602f7164" Oct 30 19:00:08.372484 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:08.372456 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889780-xvmhb_odh-ai-gateway-infra(9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889780-xvmhb" podUID="9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1" Oct 30 19:00:08.374046 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.374028 4442 scope.go:122] "RemoveContainer" containerID="981ec931ea42bd0b0439c791c052545c585ba28be5907d69078fcae12e6da488" Oct 30 19:00:08.374473 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:08.374308 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-380c08-29889780-bpkmv_odh-ai-gateway-infra(4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889780-bpkmv" podUID="4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2" Oct 30 19:00:08.376612 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.376591 4442 scope.go:122] "RemoveContainer" containerID="2155792d1bff789bcf3694382ee657c4df4ba1758c7743d78d67b611523474f8" Oct 30 19:00:08.376904 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:08.376875 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-f37ca7-29889780-j2k5m_odh-ai-gateway-infra(1e7491f9-b178-456d-9a56-3eb6b5ee2e40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f37ca7-29889780-j2k5m" podUID="1e7491f9-b178-456d-9a56-3eb6b5ee2e40" Oct 30 19:00:08.378604 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.378588 4442 scope.go:122] "RemoveContainer" containerID="762c977776ca5bddcad3b1ec3c9fa5a975452d72a71c7c677f54340c5066371b" Oct 30 19:00:08.378868 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:08.378834 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-45dc34-29889780-h6xgl_odh-ai-gateway-infra(2bb028e5-0be3-4923-9894-dc8f89889799)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889780-h6xgl" podUID="2bb028e5-0be3-4923-9894-dc8f89889799" Oct 30 19:00:08.380600 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.380581 4442 scope.go:122] "RemoveContainer" containerID="cf4e9770f01d04d12c1fcfa31983c6bd9add848982c43ab026926afbf12c89da" Oct 30 19:00:08.380860 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:08.380836 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889780-bqnkh_odh-ai-gateway-infra(6e894ed2-7f33-42f4-aa19-9ba21268f296)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889780-bqnkh" podUID="6e894ed2-7f33-42f4-aa19-9ba21268f296" Oct 30 19:00:08.381920 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.381896 4442 generic.go:356] "Generic (PLEG): container finished" podID="04780470-66fb-45c6-b86e-8befadb35217" containerID="ad3911fabe4c2c631c1add2c9c8231cb461f13a2b61b3e851896815f13aa0829" exitCode=0 Oct 30 19:00:08.382038 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.381943 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" Oct 30 19:00:08.382038 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.381948 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" event={"ID":"04780470-66fb-45c6-b86e-8befadb35217","Type":"ContainerDied","Data":"ad3911fabe4c2c631c1add2c9c8231cb461f13a2b61b3e851896815f13aa0829"} Oct 30 19:00:08.382169 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.382038 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll" event={"ID":"04780470-66fb-45c6-b86e-8befadb35217","Type":"ContainerDied","Data":"daa90ee13b402c3fa44c55776cf7eaba845aca9bae3de73569642575a7f897bb"} Oct 30 19:00:08.382169 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.382069 4442 scope.go:122] "RemoveContainer" containerID="ad3911fabe4c2c631c1add2c9c8231cb461f13a2b61b3e851896815f13aa0829" Oct 30 19:00:08.397890 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.397868 4442 scope.go:122] "RemoveContainer" containerID="ad3911fabe4c2c631c1add2c9c8231cb461f13a2b61b3e851896815f13aa0829" Oct 30 19:00:08.398283 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:08.398261 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad3911fabe4c2c631c1add2c9c8231cb461f13a2b61b3e851896815f13aa0829\": container with ID starting with ad3911fabe4c2c631c1add2c9c8231cb461f13a2b61b3e851896815f13aa0829 not found: ID does not exist" containerID="ad3911fabe4c2c631c1add2c9c8231cb461f13a2b61b3e851896815f13aa0829" Oct 30 19:00:08.398375 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.398290 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad3911fabe4c2c631c1add2c9c8231cb461f13a2b61b3e851896815f13aa0829"} err="failed to get container status \"ad3911fabe4c2c631c1add2c9c8231cb461f13a2b61b3e851896815f13aa0829\": rpc error: code = NotFound desc = could not find container \"ad3911fabe4c2c631c1add2c9c8231cb461f13a2b61b3e851896815f13aa0829\": container with ID starting with ad3911fabe4c2c631c1add2c9c8231cb461f13a2b61b3e851896815f13aa0829 not found: ID does not exist" Oct 30 19:00:08.401937 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.401910 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04780470-66fb-45c6-b86e-8befadb35217-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04780470-66fb-45c6-b86e-8befadb35217-workload-socket\") pod \"04780470-66fb-45c6-b86e-8befadb35217\" (UID: \"04780470-66fb-45c6-b86e-8befadb35217\") " Oct 30 19:00:08.402055 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.401961 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04780470-66fb-45c6-b86e-8befadb35217-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04780470-66fb-45c6-b86e-8befadb35217-istiod-ca-cert\") pod \"04780470-66fb-45c6-b86e-8befadb35217\" (UID: \"04780470-66fb-45c6-b86e-8befadb35217\") " Oct 30 19:00:08.402167 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.402057 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04780470-66fb-45c6-b86e-8befadb35217-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04780470-66fb-45c6-b86e-8befadb35217-istio-envoy\") pod \"04780470-66fb-45c6-b86e-8befadb35217\" (UID: \"04780470-66fb-45c6-b86e-8befadb35217\") " Oct 30 19:00:08.402167 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.402104 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04780470-66fb-45c6-b86e-8befadb35217-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04780470-66fb-45c6-b86e-8befadb35217-istio-data\") pod \"04780470-66fb-45c6-b86e-8befadb35217\" (UID: \"04780470-66fb-45c6-b86e-8befadb35217\") " Oct 30 19:00:08.402291 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.402200 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04780470-66fb-45c6-b86e-8befadb35217-workload-socket" pod "04780470-66fb-45c6-b86e-8befadb35217" (UID: "04780470-66fb-45c6-b86e-8befadb35217"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:08.402291 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.402225 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04780470-66fb-45c6-b86e-8befadb35217-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04780470-66fb-45c6-b86e-8befadb35217-credential-socket\") pod \"04780470-66fb-45c6-b86e-8befadb35217\" (UID: \"04780470-66fb-45c6-b86e-8befadb35217\") " Oct 30 19:00:08.402414 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.402258 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04780470-66fb-45c6-b86e-8befadb35217-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04780470-66fb-45c6-b86e-8befadb35217-istio-podinfo\") pod \"04780470-66fb-45c6-b86e-8befadb35217\" (UID: \"04780470-66fb-45c6-b86e-8befadb35217\") " Oct 30 19:00:08.402414 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.402343 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04780470-66fb-45c6-b86e-8befadb35217-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04780470-66fb-45c6-b86e-8befadb35217-workload-certs\") pod \"04780470-66fb-45c6-b86e-8befadb35217\" (UID: \"04780470-66fb-45c6-b86e-8befadb35217\") " Oct 30 19:00:08.402414 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.402361 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04780470-66fb-45c6-b86e-8befadb35217-istio-data" pod "04780470-66fb-45c6-b86e-8befadb35217" (UID: "04780470-66fb-45c6-b86e-8befadb35217"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:08.402605 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.402423 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04780470-66fb-45c6-b86e-8befadb35217-credential-socket" pod "04780470-66fb-45c6-b86e-8befadb35217" (UID: "04780470-66fb-45c6-b86e-8befadb35217"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:08.402605 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.402450 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04780470-66fb-45c6-b86e-8befadb35217-istiod-ca-cert" pod "04780470-66fb-45c6-b86e-8befadb35217" (UID: "04780470-66fb-45c6-b86e-8befadb35217"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:08.402605 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.402591 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04780470-66fb-45c6-b86e-8befadb35217-kube-api-access-zrrz6\" (UniqueName: \"kubernetes.io/projected/04780470-66fb-45c6-b86e-8befadb35217-kube-api-access-zrrz6\") pod \"04780470-66fb-45c6-b86e-8befadb35217\" (UID: \"04780470-66fb-45c6-b86e-8befadb35217\") " Oct 30 19:00:08.402779 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.402688 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04780470-66fb-45c6-b86e-8befadb35217-workload-certs" pod "04780470-66fb-45c6-b86e-8befadb35217" (UID: "04780470-66fb-45c6-b86e-8befadb35217"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:08.402779 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.402721 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04780470-66fb-45c6-b86e-8befadb35217-istio-token\" (UniqueName: \"kubernetes.io/projected/04780470-66fb-45c6-b86e-8befadb35217-istio-token\") pod \"04780470-66fb-45c6-b86e-8befadb35217\" (UID: \"04780470-66fb-45c6-b86e-8befadb35217\") " Oct 30 19:00:08.403733 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.403704 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04780470-66fb-45c6-b86e-8befadb35217-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:08.403876 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.403738 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04780470-66fb-45c6-b86e-8befadb35217-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:08.403876 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.403753 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04780470-66fb-45c6-b86e-8befadb35217-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:08.403876 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.403768 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04780470-66fb-45c6-b86e-8befadb35217-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:08.403876 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.403783 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04780470-66fb-45c6-b86e-8befadb35217-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:08.405034 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.405001 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04780470-66fb-45c6-b86e-8befadb35217-istio-token" pod "04780470-66fb-45c6-b86e-8befadb35217" (UID: "04780470-66fb-45c6-b86e-8befadb35217"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:08.405126 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.405080 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04780470-66fb-45c6-b86e-8befadb35217-kube-api-access-zrrz6" pod "04780470-66fb-45c6-b86e-8befadb35217" (UID: "04780470-66fb-45c6-b86e-8befadb35217"). InnerVolumeSpecName "kube-api-access-zrrz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:08.405126 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.405102 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04780470-66fb-45c6-b86e-8befadb35217-istio-podinfo" pod "04780470-66fb-45c6-b86e-8befadb35217" (UID: "04780470-66fb-45c6-b86e-8befadb35217"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:00:08.405608 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.405593 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04780470-66fb-45c6-b86e-8befadb35217-istio-envoy" pod "04780470-66fb-45c6-b86e-8befadb35217" (UID: "04780470-66fb-45c6-b86e-8befadb35217"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:08.472983 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.472952 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m"] Oct 30 19:00:08.473234 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.473206 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" podUID="aac37dcc-c395-40e0-8d20-ec7733106b01" containerName="istio-proxy" containerID="cri-o://e9ab725a3dd442f3c283138f96df0107b9eaa46818815f533a687c8b8dfb2068" gracePeriod=30 Oct 30 19:00:08.506656 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.506258 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04780470-66fb-45c6-b86e-8befadb35217-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:08.506656 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.506406 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04780470-66fb-45c6-b86e-8befadb35217-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:08.506656 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.506480 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrrz6\" (UniqueName: \"kubernetes.io/projected/04780470-66fb-45c6-b86e-8befadb35217-kube-api-access-zrrz6\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:08.506656 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:08.506500 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04780470-66fb-45c6-b86e-8befadb35217-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:09.015750 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:09.015635 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll"] Oct 30 19:00:09.022802 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:09.022777 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-2d5ll"] Oct 30 19:00:09.119161 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:09.119106 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04780470-66fb-45c6-b86e-8befadb35217" path="/var/lib/kubelet/pods/04780470-66fb-45c6-b86e-8befadb35217/volumes" Oct 30 19:00:11.503072 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:11.503030 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78763cdc-7vznh"] Oct 30 19:00:11.504113 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:11.504095 4442 cpu_manager.go:414] "containerMap: removing container" podUID="04780470-66fb-45c6-b86e-8befadb35217" containerName="istio-proxy" Oct 30 19:00:11.504194 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:11.504115 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04780470-66fb-45c6-b86e-8befadb35217" containerName="istio-proxy" Oct 30 19:00:11.504194 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:11.504177 4442 cpu_manager.go:414] "containerMap: removing container" podUID="418064ff-dabc-4b46-b2cf-133f7f20e9d7" containerName="istio-proxy" Oct 30 19:00:11.504194 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:11.504183 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="418064ff-dabc-4b46-b2cf-133f7f20e9d7" containerName="istio-proxy" Oct 30 19:00:11.504328 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:11.504317 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="04780470-66fb-45c6-b86e-8befadb35217" containerName="istio-proxy" Oct 30 19:00:11.504361 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:11.504340 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="418064ff-dabc-4b46-b2cf-133f7f20e9d7" containerName="istio-proxy" Oct 30 19:00:11.510027 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:11.510006 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78763cdc-7vznh" Oct 30 19:00:11.528351 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:11.528314 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78763cdc-7vznh"] Oct 30 19:00:11.531688 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:11.531111 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:00:11.566547 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:11.566512 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/422f6f0d-871f-44d2-8fba-2dc65d684ced-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78763cdc-7vznh\" (UID: \"422f6f0d-871f-44d2-8fba-2dc65d684ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78763cdc-7vznh" Oct 30 19:00:11.566765 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:11.566688 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmgqp\" (UniqueName: \"kubernetes.io/projected/422f6f0d-871f-44d2-8fba-2dc65d684ced-kube-api-access-jmgqp\") pod \"maas-api-revoke-sub-78763cdc-7vznh\" (UID: \"422f6f0d-871f-44d2-8fba-2dc65d684ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78763cdc-7vznh" Oct 30 19:00:11.566874 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:11.566854 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/422f6f0d-871f-44d2-8fba-2dc65d684ced-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78763cdc-7vznh\" (UID: \"422f6f0d-871f-44d2-8fba-2dc65d684ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78763cdc-7vznh" Oct 30 19:00:11.668091 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:11.668052 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/422f6f0d-871f-44d2-8fba-2dc65d684ced-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78763cdc-7vznh\" (UID: \"422f6f0d-871f-44d2-8fba-2dc65d684ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78763cdc-7vznh" Oct 30 19:00:11.668299 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:11.668195 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmgqp\" (UniqueName: \"kubernetes.io/projected/422f6f0d-871f-44d2-8fba-2dc65d684ced-kube-api-access-jmgqp\") pod \"maas-api-revoke-sub-78763cdc-7vznh\" (UID: \"422f6f0d-871f-44d2-8fba-2dc65d684ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78763cdc-7vznh" Oct 30 19:00:11.668366 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:11.668297 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/422f6f0d-871f-44d2-8fba-2dc65d684ced-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78763cdc-7vznh\" (UID: \"422f6f0d-871f-44d2-8fba-2dc65d684ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78763cdc-7vznh" Oct 30 19:00:11.668724 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:11.668703 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/422f6f0d-871f-44d2-8fba-2dc65d684ced-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78763cdc-7vznh\" (UID: \"422f6f0d-871f-44d2-8fba-2dc65d684ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78763cdc-7vznh" Oct 30 19:00:11.671031 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:11.671007 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/422f6f0d-871f-44d2-8fba-2dc65d684ced-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78763cdc-7vznh\" (UID: \"422f6f0d-871f-44d2-8fba-2dc65d684ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78763cdc-7vznh" Oct 30 19:00:11.690574 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:11.690547 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmgqp\" (UniqueName: \"kubernetes.io/projected/422f6f0d-871f-44d2-8fba-2dc65d684ced-kube-api-access-jmgqp\") pod \"maas-api-revoke-sub-78763cdc-7vznh\" (UID: \"422f6f0d-871f-44d2-8fba-2dc65d684ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78763cdc-7vznh" Oct 30 19:00:11.830154 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:11.830059 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78763cdc-7vznh" Oct 30 19:00:12.144588 ip-10-0-138-176 kubenswrapper[4442]: W1030 19:00:12.144552 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod422f6f0d_871f_44d2_8fba_2dc65d684ced.slice/crio-a6c2157330e1b5d783f29be62d6f9b5d8b81762c9cec8cf068309bdc52f0e445 WatchSource:0}: Error finding container a6c2157330e1b5d783f29be62d6f9b5d8b81762c9cec8cf068309bdc52f0e445: Status 404 returned error can't find the container with id a6c2157330e1b5d783f29be62d6f9b5d8b81762c9cec8cf068309bdc52f0e445 Oct 30 19:00:12.146179 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:12.146127 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78763cdc-7vznh"] Oct 30 19:00:12.251911 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:12.251721 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-551d0b/test-model-551d0b-kserve-65db7ffb5-b828f"] Oct 30 19:00:12.593641 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:12.593605 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78763cdc-7vznh" event={"ID":"422f6f0d-871f-44d2-8fba-2dc65d684ced","Type":"ContainerStarted","Data":"a6c2157330e1b5d783f29be62d6f9b5d8b81762c9cec8cf068309bdc52f0e445"} Oct 30 19:00:12.943636 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:12.943599 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" Oct 30 19:00:13.003807 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.003765 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-workload-certs\") pod \"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2\" (UID: \"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2\") " Oct 30 19:00:13.003980 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.003954 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-kube-api-access-5v5ht\" (UniqueName: \"kubernetes.io/projected/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-kube-api-access-5v5ht\") pod \"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2\" (UID: \"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2\") " Oct 30 19:00:13.004071 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.004043 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-workload-certs" pod "c7e7ff65-c137-48d2-a189-eb7f64bb1ff2" (UID: "c7e7ff65-c137-48d2-a189-eb7f64bb1ff2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:13.004569 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.004543 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-istio-token\" (UniqueName: \"kubernetes.io/projected/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-istio-token\") pod \"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2\" (UID: \"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2\") " Oct 30 19:00:13.005013 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.004996 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-istio-data\") pod \"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2\" (UID: \"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2\") " Oct 30 19:00:13.005689 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.005632 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-istio-data" pod "c7e7ff65-c137-48d2-a189-eb7f64bb1ff2" (UID: "c7e7ff65-c137-48d2-a189-eb7f64bb1ff2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:13.006042 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.005248 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-istio-podinfo\") pod \"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2\" (UID: \"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2\") " Oct 30 19:00:13.006212 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.006194 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-istiod-ca-cert\") pod \"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2\" (UID: \"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2\") " Oct 30 19:00:13.006478 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.006459 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-istio-envoy\") pod \"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2\" (UID: \"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2\") " Oct 30 19:00:13.006586 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.006529 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-credential-socket\") pod \"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2\" (UID: \"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2\") " Oct 30 19:00:13.006586 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.006565 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-workload-socket\") pod \"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2\" (UID: \"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2\") " Oct 30 19:00:13.007293 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.007271 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:13.007397 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.007297 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:13.007522 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.007504 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-workload-socket" pod "c7e7ff65-c137-48d2-a189-eb7f64bb1ff2" (UID: "c7e7ff65-c137-48d2-a189-eb7f64bb1ff2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:13.007833 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.007814 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-istiod-ca-cert" pod "c7e7ff65-c137-48d2-a189-eb7f64bb1ff2" (UID: "c7e7ff65-c137-48d2-a189-eb7f64bb1ff2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:13.008297 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.008237 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-credential-socket" pod "c7e7ff65-c137-48d2-a189-eb7f64bb1ff2" (UID: "c7e7ff65-c137-48d2-a189-eb7f64bb1ff2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:13.010544 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.010505 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-kube-api-access-5v5ht" pod "c7e7ff65-c137-48d2-a189-eb7f64bb1ff2" (UID: "c7e7ff65-c137-48d2-a189-eb7f64bb1ff2"). InnerVolumeSpecName "kube-api-access-5v5ht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:13.013663 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.011346 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-istio-token" pod "c7e7ff65-c137-48d2-a189-eb7f64bb1ff2" (UID: "c7e7ff65-c137-48d2-a189-eb7f64bb1ff2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:13.014543 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.014512 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-istio-podinfo" pod "c7e7ff65-c137-48d2-a189-eb7f64bb1ff2" (UID: "c7e7ff65-c137-48d2-a189-eb7f64bb1ff2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:00:13.014997 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.014969 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-istio-envoy" pod "c7e7ff65-c137-48d2-a189-eb7f64bb1ff2" (UID: "c7e7ff65-c137-48d2-a189-eb7f64bb1ff2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:13.111847 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.111761 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:13.112021 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.111968 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:13.112021 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.111993 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:13.112021 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.112011 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v5ht\" (UniqueName: \"kubernetes.io/projected/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-kube-api-access-5v5ht\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:13.112206 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.112027 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:13.112206 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.112042 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:13.112206 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.112058 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:13.607707 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.607671 4442 generic.go:356] "Generic (PLEG): container finished" podID="c7e7ff65-c137-48d2-a189-eb7f64bb1ff2" containerID="91593f9a61c1578c25f2243d3802cdf9a82469710a7eb79d449bc50d34981656" exitCode=0 Oct 30 19:00:13.608197 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.607741 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" Oct 30 19:00:13.608197 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.607750 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" event={"ID":"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2","Type":"ContainerDied","Data":"91593f9a61c1578c25f2243d3802cdf9a82469710a7eb79d449bc50d34981656"} Oct 30 19:00:13.608197 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.607814 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w" event={"ID":"c7e7ff65-c137-48d2-a189-eb7f64bb1ff2","Type":"ContainerDied","Data":"58add8e7335419c60bd614a71d6c517213eb0fd129b359db412f5179984937bb"} Oct 30 19:00:13.608197 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.607843 4442 scope.go:122] "RemoveContainer" containerID="91593f9a61c1578c25f2243d3802cdf9a82469710a7eb79d449bc50d34981656" Oct 30 19:00:13.609981 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.609954 4442 generic.go:356] "Generic (PLEG): container finished" podID="422f6f0d-871f-44d2-8fba-2dc65d684ced" containerID="c00c75efa7a49f95d797d6838c588bb6a83e3e266586f75834fca09c9d8a2832" exitCode=0 Oct 30 19:00:13.610110 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.610049 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78763cdc-7vznh" event={"ID":"422f6f0d-871f-44d2-8fba-2dc65d684ced","Type":"ContainerDied","Data":"c00c75efa7a49f95d797d6838c588bb6a83e3e266586f75834fca09c9d8a2832"} Oct 30 19:00:13.612750 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.612726 4442 generic.go:356] "Generic (PLEG): container finished" podID="aac37dcc-c395-40e0-8d20-ec7733106b01" containerID="e9ab725a3dd442f3c283138f96df0107b9eaa46818815f533a687c8b8dfb2068" exitCode=0 Oct 30 19:00:13.612849 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.612802 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" event={"ID":"aac37dcc-c395-40e0-8d20-ec7733106b01","Type":"ContainerDied","Data":"e9ab725a3dd442f3c283138f96df0107b9eaa46818815f533a687c8b8dfb2068"} Oct 30 19:00:13.626681 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.626539 4442 scope.go:122] "RemoveContainer" containerID="9ece5b1efeaf907e8e1f56ccd2da07a49a6b8207b3c212b1043c78ca79b7558d" Oct 30 19:00:13.642590 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.642565 4442 scope.go:122] "RemoveContainer" containerID="91593f9a61c1578c25f2243d3802cdf9a82469710a7eb79d449bc50d34981656" Oct 30 19:00:13.642961 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:13.642935 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91593f9a61c1578c25f2243d3802cdf9a82469710a7eb79d449bc50d34981656\": container with ID starting with 91593f9a61c1578c25f2243d3802cdf9a82469710a7eb79d449bc50d34981656 not found: ID does not exist" containerID="91593f9a61c1578c25f2243d3802cdf9a82469710a7eb79d449bc50d34981656" Oct 30 19:00:13.643092 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.642974 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91593f9a61c1578c25f2243d3802cdf9a82469710a7eb79d449bc50d34981656"} err="failed to get container status \"91593f9a61c1578c25f2243d3802cdf9a82469710a7eb79d449bc50d34981656\": rpc error: code = NotFound desc = could not find container \"91593f9a61c1578c25f2243d3802cdf9a82469710a7eb79d449bc50d34981656\": container with ID starting with 91593f9a61c1578c25f2243d3802cdf9a82469710a7eb79d449bc50d34981656 not found: ID does not exist" Oct 30 19:00:13.643092 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.642999 4442 scope.go:122] "RemoveContainer" containerID="9ece5b1efeaf907e8e1f56ccd2da07a49a6b8207b3c212b1043c78ca79b7558d" Oct 30 19:00:13.643537 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:13.643370 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ece5b1efeaf907e8e1f56ccd2da07a49a6b8207b3c212b1043c78ca79b7558d\": container with ID starting with 9ece5b1efeaf907e8e1f56ccd2da07a49a6b8207b3c212b1043c78ca79b7558d not found: ID does not exist" containerID="9ece5b1efeaf907e8e1f56ccd2da07a49a6b8207b3c212b1043c78ca79b7558d" Oct 30 19:00:13.643537 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.643398 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ece5b1efeaf907e8e1f56ccd2da07a49a6b8207b3c212b1043c78ca79b7558d"} err="failed to get container status \"9ece5b1efeaf907e8e1f56ccd2da07a49a6b8207b3c212b1043c78ca79b7558d\": rpc error: code = NotFound desc = could not find container \"9ece5b1efeaf907e8e1f56ccd2da07a49a6b8207b3c212b1043c78ca79b7558d\": container with ID starting with 9ece5b1efeaf907e8e1f56ccd2da07a49a6b8207b3c212b1043c78ca79b7558d not found: ID does not exist" Oct 30 19:00:13.691233 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.691176 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w"] Oct 30 19:00:13.695868 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.695841 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c52ec7-openshift-default-78bf6c4c47-mj78w"] Oct 30 19:00:13.816325 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.816293 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" Oct 30 19:00:13.936645 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.936607 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aac37dcc-c395-40e0-8d20-ec7733106b01-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aac37dcc-c395-40e0-8d20-ec7733106b01-istio-data\") pod \"aac37dcc-c395-40e0-8d20-ec7733106b01\" (UID: \"aac37dcc-c395-40e0-8d20-ec7733106b01\") " Oct 30 19:00:13.936841 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.936679 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aac37dcc-c395-40e0-8d20-ec7733106b01-istio-token\" (UniqueName: \"kubernetes.io/projected/aac37dcc-c395-40e0-8d20-ec7733106b01-istio-token\") pod \"aac37dcc-c395-40e0-8d20-ec7733106b01\" (UID: \"aac37dcc-c395-40e0-8d20-ec7733106b01\") " Oct 30 19:00:13.936841 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.936755 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aac37dcc-c395-40e0-8d20-ec7733106b01-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aac37dcc-c395-40e0-8d20-ec7733106b01-istio-podinfo\") pod \"aac37dcc-c395-40e0-8d20-ec7733106b01\" (UID: \"aac37dcc-c395-40e0-8d20-ec7733106b01\") " Oct 30 19:00:13.936841 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.936793 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aac37dcc-c395-40e0-8d20-ec7733106b01-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac37dcc-c395-40e0-8d20-ec7733106b01-workload-socket\") pod \"aac37dcc-c395-40e0-8d20-ec7733106b01\" (UID: \"aac37dcc-c395-40e0-8d20-ec7733106b01\") " Oct 30 19:00:13.937035 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.936976 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aac37dcc-c395-40e0-8d20-ec7733106b01-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aac37dcc-c395-40e0-8d20-ec7733106b01-workload-certs\") pod \"aac37dcc-c395-40e0-8d20-ec7733106b01\" (UID: \"aac37dcc-c395-40e0-8d20-ec7733106b01\") " Oct 30 19:00:13.937035 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.937017 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aac37dcc-c395-40e0-8d20-ec7733106b01-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aac37dcc-c395-40e0-8d20-ec7733106b01-istiod-ca-cert\") pod \"aac37dcc-c395-40e0-8d20-ec7733106b01\" (UID: \"aac37dcc-c395-40e0-8d20-ec7733106b01\") " Oct 30 19:00:13.937206 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.937127 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aac37dcc-c395-40e0-8d20-ec7733106b01-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac37dcc-c395-40e0-8d20-ec7733106b01-credential-socket\") pod \"aac37dcc-c395-40e0-8d20-ec7733106b01\" (UID: \"aac37dcc-c395-40e0-8d20-ec7733106b01\") " Oct 30 19:00:13.937306 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.937286 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aac37dcc-c395-40e0-8d20-ec7733106b01-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aac37dcc-c395-40e0-8d20-ec7733106b01-istio-envoy\") pod \"aac37dcc-c395-40e0-8d20-ec7733106b01\" (UID: \"aac37dcc-c395-40e0-8d20-ec7733106b01\") " Oct 30 19:00:13.937366 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.937331 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aac37dcc-c395-40e0-8d20-ec7733106b01-kube-api-access-w8xk8\" (UniqueName: \"kubernetes.io/projected/aac37dcc-c395-40e0-8d20-ec7733106b01-kube-api-access-w8xk8\") pod \"aac37dcc-c395-40e0-8d20-ec7733106b01\" (UID: \"aac37dcc-c395-40e0-8d20-ec7733106b01\") " Oct 30 19:00:13.937428 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.937403 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac37dcc-c395-40e0-8d20-ec7733106b01-workload-certs" pod "aac37dcc-c395-40e0-8d20-ec7733106b01" (UID: "aac37dcc-c395-40e0-8d20-ec7733106b01"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:13.937608 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.937588 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac37dcc-c395-40e0-8d20-ec7733106b01-credential-socket" pod "aac37dcc-c395-40e0-8d20-ec7733106b01" (UID: "aac37dcc-c395-40e0-8d20-ec7733106b01"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:13.938208 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.938175 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aac37dcc-c395-40e0-8d20-ec7733106b01-istiod-ca-cert" pod "aac37dcc-c395-40e0-8d20-ec7733106b01" (UID: "aac37dcc-c395-40e0-8d20-ec7733106b01"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:13.938613 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.938552 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aac37dcc-c395-40e0-8d20-ec7733106b01-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:13.938613 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.938581 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac37dcc-c395-40e0-8d20-ec7733106b01-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:13.939964 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.939940 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac37dcc-c395-40e0-8d20-ec7733106b01-workload-socket" pod "aac37dcc-c395-40e0-8d20-ec7733106b01" (UID: "aac37dcc-c395-40e0-8d20-ec7733106b01"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:13.940304 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.940285 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac37dcc-c395-40e0-8d20-ec7733106b01-istio-data" pod "aac37dcc-c395-40e0-8d20-ec7733106b01" (UID: "aac37dcc-c395-40e0-8d20-ec7733106b01"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:13.943032 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.942968 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aac37dcc-c395-40e0-8d20-ec7733106b01-kube-api-access-w8xk8" pod "aac37dcc-c395-40e0-8d20-ec7733106b01" (UID: "aac37dcc-c395-40e0-8d20-ec7733106b01"). InnerVolumeSpecName "kube-api-access-w8xk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:13.946180 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.946146 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aac37dcc-c395-40e0-8d20-ec7733106b01-istio-token" pod "aac37dcc-c395-40e0-8d20-ec7733106b01" (UID: "aac37dcc-c395-40e0-8d20-ec7733106b01"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:13.950618 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.950588 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac37dcc-c395-40e0-8d20-ec7733106b01-istio-envoy" pod "aac37dcc-c395-40e0-8d20-ec7733106b01" (UID: "aac37dcc-c395-40e0-8d20-ec7733106b01"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:13.951031 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:13.950969 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aac37dcc-c395-40e0-8d20-ec7733106b01-istio-podinfo" pod "aac37dcc-c395-40e0-8d20-ec7733106b01" (UID: "aac37dcc-c395-40e0-8d20-ec7733106b01"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:00:14.039779 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:14.039695 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aac37dcc-c395-40e0-8d20-ec7733106b01-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:14.039779 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:14.039727 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8xk8\" (UniqueName: \"kubernetes.io/projected/aac37dcc-c395-40e0-8d20-ec7733106b01-kube-api-access-w8xk8\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:14.039779 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:14.039737 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aac37dcc-c395-40e0-8d20-ec7733106b01-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:14.039779 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:14.039749 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aac37dcc-c395-40e0-8d20-ec7733106b01-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:14.039779 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:14.039757 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aac37dcc-c395-40e0-8d20-ec7733106b01-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:14.039779 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:14.039766 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac37dcc-c395-40e0-8d20-ec7733106b01-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:14.039779 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:14.039775 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aac37dcc-c395-40e0-8d20-ec7733106b01-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:14.627055 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:14.627027 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" Oct 30 19:00:14.628803 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:14.628310 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m" event={"ID":"aac37dcc-c395-40e0-8d20-ec7733106b01","Type":"ContainerDied","Data":"3103d5cdef52f30815e3ca8dafbcc7140703869aa51c68eba4c9c502ac1f6061"} Oct 30 19:00:14.628803 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:14.628582 4442 scope.go:122] "RemoveContainer" containerID="e9ab725a3dd442f3c283138f96df0107b9eaa46818815f533a687c8b8dfb2068" Oct 30 19:00:14.684563 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:14.684483 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m"] Oct 30 19:00:14.695514 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:14.695482 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-7jp6m"] Oct 30 19:00:14.846972 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:14.846927 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78763cdc-7vznh" Oct 30 19:00:14.962286 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:14.962161 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/422f6f0d-871f-44d2-8fba-2dc65d684ced-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/422f6f0d-871f-44d2-8fba-2dc65d684ced-maas-api-openshift-ca\") pod \"422f6f0d-871f-44d2-8fba-2dc65d684ced\" (UID: \"422f6f0d-871f-44d2-8fba-2dc65d684ced\") " Oct 30 19:00:14.962531 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:14.962318 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/422f6f0d-871f-44d2-8fba-2dc65d684ced-kube-api-access-jmgqp\" (UniqueName: \"kubernetes.io/projected/422f6f0d-871f-44d2-8fba-2dc65d684ced-kube-api-access-jmgqp\") pod \"422f6f0d-871f-44d2-8fba-2dc65d684ced\" (UID: \"422f6f0d-871f-44d2-8fba-2dc65d684ced\") " Oct 30 19:00:14.962643 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:14.962573 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/422f6f0d-871f-44d2-8fba-2dc65d684ced-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/422f6f0d-871f-44d2-8fba-2dc65d684ced-maas-api-xks-ca\") pod \"422f6f0d-871f-44d2-8fba-2dc65d684ced\" (UID: \"422f6f0d-871f-44d2-8fba-2dc65d684ced\") " Oct 30 19:00:14.963617 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:14.963540 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/422f6f0d-871f-44d2-8fba-2dc65d684ced-maas-api-openshift-ca" pod "422f6f0d-871f-44d2-8fba-2dc65d684ced" (UID: "422f6f0d-871f-44d2-8fba-2dc65d684ced"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:14.966187 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:14.966100 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/422f6f0d-871f-44d2-8fba-2dc65d684ced-maas-api-xks-ca" pod "422f6f0d-871f-44d2-8fba-2dc65d684ced" (UID: "422f6f0d-871f-44d2-8fba-2dc65d684ced"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:00:14.966325 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:14.966294 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/422f6f0d-871f-44d2-8fba-2dc65d684ced-kube-api-access-jmgqp" pod "422f6f0d-871f-44d2-8fba-2dc65d684ced" (UID: "422f6f0d-871f-44d2-8fba-2dc65d684ced"). InnerVolumeSpecName "kube-api-access-jmgqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:15.065672 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:15.065635 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmgqp\" (UniqueName: \"kubernetes.io/projected/422f6f0d-871f-44d2-8fba-2dc65d684ced-kube-api-access-jmgqp\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:15.065672 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:15.065670 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/422f6f0d-871f-44d2-8fba-2dc65d684ced-maas-api-xks-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:15.065932 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:15.065684 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/422f6f0d-871f-44d2-8fba-2dc65d684ced-maas-api-openshift-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:15.071228 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:15.071185 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aac37dcc-c395-40e0-8d20-ec7733106b01" path="/var/lib/kubelet/pods/aac37dcc-c395-40e0-8d20-ec7733106b01/volumes" Oct 30 19:00:15.071906 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:15.071886 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7e7ff65-c137-48d2-a189-eb7f64bb1ff2" path="/var/lib/kubelet/pods/c7e7ff65-c137-48d2-a189-eb7f64bb1ff2/volumes" Oct 30 19:00:15.640392 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:15.640366 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78763cdc-7vznh" Oct 30 19:00:15.640871 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:15.640361 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78763cdc-7vznh" event={"ID":"422f6f0d-871f-44d2-8fba-2dc65d684ced","Type":"ContainerDied","Data":"a6c2157330e1b5d783f29be62d6f9b5d8b81762c9cec8cf068309bdc52f0e445"} Oct 30 19:00:15.640871 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:15.640516 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6c2157330e1b5d783f29be62d6f9b5d8b81762c9cec8cf068309bdc52f0e445" Oct 30 19:00:16.058903 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:16.058809 4442 scope.go:122] "RemoveContainer" containerID="c8c4bbf0374dbf36b3b0a51afb54f9a7189dc34742c46de60df34f56b199acd4" Oct 30 19:00:16.655211 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:16.655168 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889780-qkctp" event={"ID":"fb187f07-08dd-40b1-bb89-7f4993d8494f","Type":"ContainerStarted","Data":"2e75dbc5e2b8e79dae323da489dc6dde7a6c15271882c1631d51cc2fef733b99"} Oct 30 19:00:17.057695 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:17.057665 4442 scope.go:122] "RemoveContainer" containerID="efa9992046cd918168968a92b81dfd8a8db688c6e01d2408adda15c16b1e4802" Oct 30 19:00:17.058212 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:17.057916 4442 scope.go:122] "RemoveContainer" containerID="762c977776ca5bddcad3b1ec3c9fa5a975452d72a71c7c677f54340c5066371b" Oct 30 19:00:17.565588 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:17.565554 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 19:00:17.669440 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:17.669402 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889780-2hsw5" event={"ID":"a2ba9730-da34-4745-875f-cd941e32eb79","Type":"ContainerStarted","Data":"4fd9ce2c01adebe9ef897f77cf0ab8a99b2cb1c78d7132121729edb0bedfb4f1"} Oct 30 19:00:17.671773 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:17.671746 4442 generic.go:356] "Generic (PLEG): container finished" podID="fb187f07-08dd-40b1-bb89-7f4993d8494f" containerID="2e75dbc5e2b8e79dae323da489dc6dde7a6c15271882c1631d51cc2fef733b99" exitCode=60 Oct 30 19:00:17.671916 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:17.671820 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889780-qkctp" event={"ID":"fb187f07-08dd-40b1-bb89-7f4993d8494f","Type":"ContainerDied","Data":"2e75dbc5e2b8e79dae323da489dc6dde7a6c15271882c1631d51cc2fef733b99"} Oct 30 19:00:17.671916 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:17.671877 4442 scope.go:122] "RemoveContainer" containerID="c8c4bbf0374dbf36b3b0a51afb54f9a7189dc34742c46de60df34f56b199acd4" Oct 30 19:00:17.672340 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:17.672321 4442 scope.go:122] "RemoveContainer" containerID="2e75dbc5e2b8e79dae323da489dc6dde7a6c15271882c1631d51cc2fef733b99" Oct 30 19:00:17.672750 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:17.672721 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889780-qkctp_odh-ai-gateway-infra(fb187f07-08dd-40b1-bb89-7f4993d8494f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889780-qkctp" podUID="fb187f07-08dd-40b1-bb89-7f4993d8494f" Oct 30 19:00:17.674178 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:17.674155 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889780-h6xgl" event={"ID":"2bb028e5-0be3-4923-9894-dc8f89889799","Type":"ContainerStarted","Data":"d603451329e208c4e5a782fa813d3b63310044c242503006d5764dcffafd6922"} Oct 30 19:00:17.706541 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:17.706514 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889780-qkctp"] Oct 30 19:00:18.057979 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:18.057891 4442 scope.go:122] "RemoveContainer" containerID="c4b00b6eecaa267a1cd1a860d910fa883ccd9713feb6570d0038debab4031fba" Oct 30 19:00:18.058632 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:18.058608 4442 scope.go:122] "RemoveContainer" containerID="7d9647608b1d512f3eed071f0504e94b91fa41051cdc81bb3664360c5404baeb" Oct 30 19:00:18.058942 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:18.058919 4442 scope.go:122] "RemoveContainer" containerID="981ec931ea42bd0b0439c791c052545c585ba28be5907d69078fcae12e6da488" Oct 30 19:00:18.059264 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:18.059249 4442 scope.go:122] "RemoveContainer" containerID="02a8c9cae50e24ef176091696fcb817d7b257fa0aa610e2e13e20407e045b07e" Oct 30 19:00:18.059527 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:18.059508 4442 scope.go:122] "RemoveContainer" containerID="9fcc9137488fd59b8a93b804d1632f1fd8ddc0c8b0222a248a0fef1bffea3d94" Oct 30 19:00:18.059759 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:18.059743 4442 scope.go:122] "RemoveContainer" containerID="53d61493663c21940b45f5bd949cda482c0f0c2c7578366d10c72c72fd989f2b" Oct 30 19:00:18.696265 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:18.696233 4442 generic.go:356] "Generic (PLEG): container finished" podID="2bb028e5-0be3-4923-9894-dc8f89889799" containerID="d603451329e208c4e5a782fa813d3b63310044c242503006d5764dcffafd6922" exitCode=60 Oct 30 19:00:18.696694 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:18.696313 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889780-h6xgl" event={"ID":"2bb028e5-0be3-4923-9894-dc8f89889799","Type":"ContainerDied","Data":"d603451329e208c4e5a782fa813d3b63310044c242503006d5764dcffafd6922"} Oct 30 19:00:18.696694 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:18.696417 4442 scope.go:122] "RemoveContainer" containerID="762c977776ca5bddcad3b1ec3c9fa5a975452d72a71c7c677f54340c5066371b" Oct 30 19:00:18.696882 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:18.696866 4442 scope.go:122] "RemoveContainer" containerID="d603451329e208c4e5a782fa813d3b63310044c242503006d5764dcffafd6922" Oct 30 19:00:18.697375 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:18.697350 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-45dc34-29889780-h6xgl_odh-ai-gateway-infra(2bb028e5-0be3-4923-9894-dc8f89889799)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889780-h6xgl" podUID="2bb028e5-0be3-4923-9894-dc8f89889799" Oct 30 19:00:18.698935 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:18.698910 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889780-vnmx9" event={"ID":"9fcfbcab-a54b-47d4-957b-76d2cb2444f1","Type":"ContainerStarted","Data":"aea98d3f54c4367eb203d1c0e0f7391f96610e5b89fc2d35a0050462cc2851e1"} Oct 30 19:00:18.701090 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:18.701069 4442 generic.go:356] "Generic (PLEG): container finished" podID="a2ba9730-da34-4745-875f-cd941e32eb79" containerID="4fd9ce2c01adebe9ef897f77cf0ab8a99b2cb1c78d7132121729edb0bedfb4f1" exitCode=60 Oct 30 19:00:18.701210 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:18.701152 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889780-2hsw5" event={"ID":"a2ba9730-da34-4745-875f-cd941e32eb79","Type":"ContainerDied","Data":"4fd9ce2c01adebe9ef897f77cf0ab8a99b2cb1c78d7132121729edb0bedfb4f1"} Oct 30 19:00:18.701506 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:18.701487 4442 scope.go:122] "RemoveContainer" containerID="4fd9ce2c01adebe9ef897f77cf0ab8a99b2cb1c78d7132121729edb0bedfb4f1" Oct 30 19:00:18.701834 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:18.701812 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-38e0db-29889780-2hsw5_odh-ai-gateway-infra(a2ba9730-da34-4745-875f-cd941e32eb79)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889780-2hsw5" podUID="a2ba9730-da34-4745-875f-cd941e32eb79" Oct 30 19:00:18.712167 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:18.712122 4442 scope.go:122] "RemoveContainer" containerID="efa9992046cd918168968a92b81dfd8a8db688c6e01d2408adda15c16b1e4802" Oct 30 19:00:18.736951 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:18.733957 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889780-2hsw5"] Oct 30 19:00:18.767558 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:18.767527 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889780-h6xgl"] Oct 30 19:00:19.091881 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:19.090767 4442 scope.go:122] "RemoveContainer" containerID="c83e53807de77746da4bd90f979125e4e54ac2188de143bdafa02c48dc17f624" Oct 30 19:00:19.093977 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:19.092230 4442 scope.go:122] "RemoveContainer" containerID="bed6c23990b5fbc950e083dc49cf78746bcf8609f2b2d06ebfac7934c328083f" Oct 30 19:00:19.118114 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:19.111363 4442 scope.go:122] "RemoveContainer" containerID="56036f5498cdd52d2b5a261c9b72bd9406acfd052381e44ec3c5a80337a49758" Oct 30 19:00:19.219417 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:19.219390 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889780-qkctp" Oct 30 19:00:19.400103 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:19.400012 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb187f07-08dd-40b1-bb89-7f4993d8494f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb187f07-08dd-40b1-bb89-7f4993d8494f-maas-api-service-ca\") pod \"fb187f07-08dd-40b1-bb89-7f4993d8494f\" (UID: \"fb187f07-08dd-40b1-bb89-7f4993d8494f\") " Oct 30 19:00:19.400406 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:19.400385 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb187f07-08dd-40b1-bb89-7f4993d8494f-kube-api-access-x65rh\" (UniqueName: \"kubernetes.io/projected/fb187f07-08dd-40b1-bb89-7f4993d8494f-kube-api-access-x65rh\") pod \"fb187f07-08dd-40b1-bb89-7f4993d8494f\" (UID: \"fb187f07-08dd-40b1-bb89-7f4993d8494f\") " Oct 30 19:00:19.400472 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:19.400454 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb187f07-08dd-40b1-bb89-7f4993d8494f-maas-api-service-ca" pod "fb187f07-08dd-40b1-bb89-7f4993d8494f" (UID: "fb187f07-08dd-40b1-bb89-7f4993d8494f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:19.401081 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:19.401055 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb187f07-08dd-40b1-bb89-7f4993d8494f-maas-api-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:19.433534 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:19.433249 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb187f07-08dd-40b1-bb89-7f4993d8494f-kube-api-access-x65rh" pod "fb187f07-08dd-40b1-bb89-7f4993d8494f" (UID: "fb187f07-08dd-40b1-bb89-7f4993d8494f"). InnerVolumeSpecName "kube-api-access-x65rh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:19.503105 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:19.503064 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x65rh\" (UniqueName: \"kubernetes.io/projected/fb187f07-08dd-40b1-bb89-7f4993d8494f-kube-api-access-x65rh\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:19.717308 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:19.717268 4442 generic.go:356] "Generic (PLEG): container finished" podID="4072bb06-edab-49c8-b213-a381823ec644" containerID="e95ff8f2d2455013e2d16225a765dec79e351b762c1c824dab2186236e40a00f" exitCode=60 Oct 30 19:00:19.717688 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:19.717428 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9f6b7c-29889780-7dcw7" event={"ID":"4072bb06-edab-49c8-b213-a381823ec644","Type":"ContainerDied","Data":"e95ff8f2d2455013e2d16225a765dec79e351b762c1c824dab2186236e40a00f"} Oct 30 19:00:19.717688 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:19.717482 4442 scope.go:122] "RemoveContainer" containerID="9fcc9137488fd59b8a93b804d1632f1fd8ddc0c8b0222a248a0fef1bffea3d94" Oct 30 19:00:19.717933 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:19.717920 4442 scope.go:122] "RemoveContainer" containerID="e95ff8f2d2455013e2d16225a765dec79e351b762c1c824dab2186236e40a00f" Oct 30 19:00:19.718322 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:19.718301 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9f6b7c-29889780-7dcw7_odh-ai-gateway-infra(4072bb06-edab-49c8-b213-a381823ec644)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9f6b7c-29889780-7dcw7" podUID="4072bb06-edab-49c8-b213-a381823ec644" Oct 30 19:00:19.720824 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:19.720800 4442 generic.go:356] "Generic (PLEG): container finished" podID="9fcfbcab-a54b-47d4-957b-76d2cb2444f1" containerID="aea98d3f54c4367eb203d1c0e0f7391f96610e5b89fc2d35a0050462cc2851e1" exitCode=60 Oct 30 19:00:19.720921 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:19.720868 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889780-vnmx9" event={"ID":"9fcfbcab-a54b-47d4-957b-76d2cb2444f1","Type":"ContainerDied","Data":"aea98d3f54c4367eb203d1c0e0f7391f96610e5b89fc2d35a0050462cc2851e1"} Oct 30 19:00:19.721288 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:19.721270 4442 scope.go:122] "RemoveContainer" containerID="aea98d3f54c4367eb203d1c0e0f7391f96610e5b89fc2d35a0050462cc2851e1" Oct 30 19:00:19.721586 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:19.721563 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-921450-29889780-vnmx9_odh-ai-gateway-infra(9fcfbcab-a54b-47d4-957b-76d2cb2444f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889780-vnmx9" podUID="9fcfbcab-a54b-47d4-957b-76d2cb2444f1" Oct 30 19:00:19.723010 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:19.722974 4442 generic.go:356] "Generic (PLEG): container finished" podID="322ab2aa-d4f3-42ef-9820-9438ced76d6b" containerID="bbf390fea71b273e8d6c9e7a9dc336e420a00fa15ed53b89166f3043d61acb44" exitCode=60 Oct 30 19:00:19.723010 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:19.723001 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889780-jsvns" event={"ID":"322ab2aa-d4f3-42ef-9820-9438ced76d6b","Type":"ContainerDied","Data":"bbf390fea71b273e8d6c9e7a9dc336e420a00fa15ed53b89166f3043d61acb44"} Oct 30 19:00:19.723490 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:19.723466 4442 scope.go:122] "RemoveContainer" containerID="bbf390fea71b273e8d6c9e7a9dc336e420a00fa15ed53b89166f3043d61acb44" Oct 30 19:00:19.723833 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:19.723805 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889780-jsvns_odh-ai-gateway-infra(322ab2aa-d4f3-42ef-9820-9438ced76d6b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889780-jsvns" podUID="322ab2aa-d4f3-42ef-9820-9438ced76d6b" Oct 30 19:00:19.725732 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:19.725709 4442 generic.go:356] "Generic (PLEG): container finished" podID="ff29d9e8-03a0-4716-840d-a32e8a5aa628" containerID="5c3641646a5513898dd658f5f5982aa0f5d10fa339285ae26bbe345df16b2657" exitCode=60 Oct 30 19:00:19.725840 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:19.725783 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889780-z79ct" event={"ID":"ff29d9e8-03a0-4716-840d-a32e8a5aa628","Type":"ContainerDied","Data":"5c3641646a5513898dd658f5f5982aa0f5d10fa339285ae26bbe345df16b2657"} Oct 30 19:00:19.726121 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:19.726102 4442 scope.go:122] "RemoveContainer" containerID="5c3641646a5513898dd658f5f5982aa0f5d10fa339285ae26bbe345df16b2657" Oct 30 19:00:19.726419 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:19.726397 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889780-z79ct_odh-ai-gateway-infra(ff29d9e8-03a0-4716-840d-a32e8a5aa628)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889780-z79ct" podUID="ff29d9e8-03a0-4716-840d-a32e8a5aa628" Oct 30 19:00:19.728733 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:19.728714 4442 generic.go:356] "Generic (PLEG): container finished" podID="00d6e25c-6e65-45de-a894-a6d1989bfd46" containerID="25d6c35d3c77abdd5d1318310c4443c17bfea69f32de7e86c26ba4ae01925b10" exitCode=60 Oct 30 19:00:19.728840 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:19.728783 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1aa59a-29889780-bw98v" event={"ID":"00d6e25c-6e65-45de-a894-a6d1989bfd46","Type":"ContainerDied","Data":"25d6c35d3c77abdd5d1318310c4443c17bfea69f32de7e86c26ba4ae01925b10"} Oct 30 19:00:19.729211 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:19.729190 4442 scope.go:122] "RemoveContainer" containerID="25d6c35d3c77abdd5d1318310c4443c17bfea69f32de7e86c26ba4ae01925b10" Oct 30 19:00:19.729525 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:19.729501 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-1aa59a-29889780-bw98v_odh-ai-gateway-infra(00d6e25c-6e65-45de-a894-a6d1989bfd46)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1aa59a-29889780-bw98v" podUID="00d6e25c-6e65-45de-a894-a6d1989bfd46" Oct 30 19:00:19.733114 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:19.733082 4442 generic.go:356] "Generic (PLEG): container finished" podID="4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2" containerID="1b37fe81ab2064434fa5118348f93e1abcbb0220baaf2bd7f9c984fb238728a7" exitCode=60 Oct 30 19:00:19.733241 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:19.733183 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889780-bpkmv" event={"ID":"4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2","Type":"ContainerDied","Data":"1b37fe81ab2064434fa5118348f93e1abcbb0220baaf2bd7f9c984fb238728a7"} Oct 30 19:00:19.733556 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:19.733538 4442 scope.go:122] "RemoveContainer" containerID="1b37fe81ab2064434fa5118348f93e1abcbb0220baaf2bd7f9c984fb238728a7" Oct 30 19:00:19.733811 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:19.733791 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-380c08-29889780-bpkmv_odh-ai-gateway-infra(4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889780-bpkmv" podUID="4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2" Oct 30 19:00:19.735555 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:19.735533 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889780-qkctp" event={"ID":"fb187f07-08dd-40b1-bb89-7f4993d8494f","Type":"ContainerDied","Data":"035ab7b23ec1e3ee6a2442ac4719c12042504914cdfaad7ec8d805a06eb4bd0d"} Oct 30 19:00:19.735677 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:19.735583 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889780-qkctp" Oct 30 19:00:19.769002 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:19.768975 4442 scope.go:122] "RemoveContainer" containerID="7d9647608b1d512f3eed071f0504e94b91fa41051cdc81bb3664360c5404baeb" Oct 30 19:00:19.795622 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:19.795605 4442 scope.go:122] "RemoveContainer" containerID="c4b00b6eecaa267a1cd1a860d910fa883ccd9713feb6570d0038debab4031fba" Oct 30 19:00:19.796099 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:19.796073 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889780-vnmx9"] Oct 30 19:00:19.857057 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:19.856246 4442 scope.go:122] "RemoveContainer" containerID="53d61493663c21940b45f5bd949cda482c0f0c2c7578366d10c72c72fd989f2b" Oct 30 19:00:19.882365 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:19.882333 4442 scope.go:122] "RemoveContainer" containerID="02a8c9cae50e24ef176091696fcb817d7b257fa0aa610e2e13e20407e045b07e" Oct 30 19:00:19.927007 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:19.926985 4442 scope.go:122] "RemoveContainer" containerID="981ec931ea42bd0b0439c791c052545c585ba28be5907d69078fcae12e6da488" Oct 30 19:00:19.960459 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:19.959525 4442 scope.go:122] "RemoveContainer" containerID="2e75dbc5e2b8e79dae323da489dc6dde7a6c15271882c1631d51cc2fef733b99" Oct 30 19:00:19.970641 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:19.970539 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889780-2hsw5" Oct 30 19:00:20.036655 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:20.036623 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2ba9730-da34-4745-875f-cd941e32eb79-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2ba9730-da34-4745-875f-cd941e32eb79-maas-api-service-ca\") pod \"a2ba9730-da34-4745-875f-cd941e32eb79\" (UID: \"a2ba9730-da34-4745-875f-cd941e32eb79\") " Oct 30 19:00:20.036854 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:20.036707 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2ba9730-da34-4745-875f-cd941e32eb79-kube-api-access-z5rr5\" (UniqueName: \"kubernetes.io/projected/a2ba9730-da34-4745-875f-cd941e32eb79-kube-api-access-z5rr5\") pod \"a2ba9730-da34-4745-875f-cd941e32eb79\" (UID: \"a2ba9730-da34-4745-875f-cd941e32eb79\") " Oct 30 19:00:20.037996 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:20.037957 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2ba9730-da34-4745-875f-cd941e32eb79-maas-api-service-ca" pod "a2ba9730-da34-4745-875f-cd941e32eb79" (UID: "a2ba9730-da34-4745-875f-cd941e32eb79"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:20.045106 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:20.045081 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2ba9730-da34-4745-875f-cd941e32eb79-kube-api-access-z5rr5" pod "a2ba9730-da34-4745-875f-cd941e32eb79" (UID: "a2ba9730-da34-4745-875f-cd941e32eb79"). InnerVolumeSpecName "kube-api-access-z5rr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:20.113728 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:20.113702 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889780-h6xgl" Oct 30 19:00:20.145428 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:20.145391 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5rr5\" (UniqueName: \"kubernetes.io/projected/a2ba9730-da34-4745-875f-cd941e32eb79-kube-api-access-z5rr5\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:20.145428 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:20.145429 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2ba9730-da34-4745-875f-cd941e32eb79-maas-api-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:20.247903 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:20.247818 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bb028e5-0be3-4923-9894-dc8f89889799-kube-api-access-cp969\" (UniqueName: \"kubernetes.io/projected/2bb028e5-0be3-4923-9894-dc8f89889799-kube-api-access-cp969\") pod \"2bb028e5-0be3-4923-9894-dc8f89889799\" (UID: \"2bb028e5-0be3-4923-9894-dc8f89889799\") " Oct 30 19:00:20.248255 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:20.248232 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bb028e5-0be3-4923-9894-dc8f89889799-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bb028e5-0be3-4923-9894-dc8f89889799-maas-api-service-ca\") pod \"2bb028e5-0be3-4923-9894-dc8f89889799\" (UID: \"2bb028e5-0be3-4923-9894-dc8f89889799\") " Oct 30 19:00:20.250510 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:20.250486 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bb028e5-0be3-4923-9894-dc8f89889799-maas-api-service-ca" pod "2bb028e5-0be3-4923-9894-dc8f89889799" (UID: "2bb028e5-0be3-4923-9894-dc8f89889799"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:20.275717 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:20.275671 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889780-qkctp"] Oct 30 19:00:20.276619 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:20.276587 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bb028e5-0be3-4923-9894-dc8f89889799-kube-api-access-cp969" pod "2bb028e5-0be3-4923-9894-dc8f89889799" (UID: "2bb028e5-0be3-4923-9894-dc8f89889799"). InnerVolumeSpecName "kube-api-access-cp969". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:20.285692 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:20.285665 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-24b124-29889780-qkctp"] Oct 30 19:00:20.351389 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:20.351350 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp969\" (UniqueName: \"kubernetes.io/projected/2bb028e5-0be3-4923-9894-dc8f89889799-kube-api-access-cp969\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:20.351389 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:20.351383 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bb028e5-0be3-4923-9894-dc8f89889799-maas-api-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:20.759608 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:20.759576 4442 generic.go:356] "Generic (PLEG): container finished" podID="2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81" containerID="4e8c94b121b60090e7e7cd53d081b8911bc3a8302b60b0898e89999ecdeba781" exitCode=60 Oct 30 19:00:20.760171 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:20.759655 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889780-kp5fc" event={"ID":"2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81","Type":"ContainerDied","Data":"4e8c94b121b60090e7e7cd53d081b8911bc3a8302b60b0898e89999ecdeba781"} Oct 30 19:00:20.760171 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:20.759772 4442 scope.go:122] "RemoveContainer" containerID="bed6c23990b5fbc950e083dc49cf78746bcf8609f2b2d06ebfac7934c328083f" Oct 30 19:00:20.760436 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:20.760338 4442 scope.go:122] "RemoveContainer" containerID="4e8c94b121b60090e7e7cd53d081b8911bc3a8302b60b0898e89999ecdeba781" Oct 30 19:00:20.761046 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:20.760710 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889780-kp5fc_odh-ai-gateway-infra(2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889780-kp5fc" podUID="2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81" Oct 30 19:00:20.770424 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:20.770402 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889780-2hsw5" Oct 30 19:00:20.773067 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:20.770588 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889780-2hsw5" event={"ID":"a2ba9730-da34-4745-875f-cd941e32eb79","Type":"ContainerDied","Data":"b3b5e5f1b9446bc089cbc2a5f43e54400f6a481980135e7372a65c77d4f7bce3"} Oct 30 19:00:20.773252 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:20.773231 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889780-xztsk" event={"ID":"5abb7ef5-cc99-469a-a165-9e9394370418","Type":"ContainerDied","Data":"9caaafc14f8b58738833cc3d109f28e24a1318927cb39db5a455819adfe19dfb"} Oct 30 19:00:20.773339 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:20.773025 4442 generic.go:356] "Generic (PLEG): container finished" podID="5abb7ef5-cc99-469a-a165-9e9394370418" containerID="9caaafc14f8b58738833cc3d109f28e24a1318927cb39db5a455819adfe19dfb" exitCode=60 Oct 30 19:00:20.773734 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:20.773713 4442 scope.go:122] "RemoveContainer" containerID="9caaafc14f8b58738833cc3d109f28e24a1318927cb39db5a455819adfe19dfb" Oct 30 19:00:20.774222 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:20.774101 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889780-xztsk_odh-ai-gateway-infra(5abb7ef5-cc99-469a-a165-9e9394370418)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889780-xztsk" podUID="5abb7ef5-cc99-469a-a165-9e9394370418" Oct 30 19:00:20.778080 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:20.777993 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889780-h6xgl" Oct 30 19:00:20.778080 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:20.778016 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889780-h6xgl" event={"ID":"2bb028e5-0be3-4923-9894-dc8f89889799","Type":"ContainerDied","Data":"9ff790f08dd1e910993c719a6adaa5555844ebfa2c3ed5db0bc79d0577a45c45"} Oct 30 19:00:20.780402 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:20.780379 4442 generic.go:356] "Generic (PLEG): container finished" podID="88daa438-101a-4adc-8307-727c4176c705" containerID="a2d0cb9de593fb49f4e37f4f016141db890adce67e37c5b00c187440c88b18d5" exitCode=60 Oct 30 19:00:20.780507 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:20.780447 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2820d9-29889780-x8hc4" event={"ID":"88daa438-101a-4adc-8307-727c4176c705","Type":"ContainerDied","Data":"a2d0cb9de593fb49f4e37f4f016141db890adce67e37c5b00c187440c88b18d5"} Oct 30 19:00:20.780866 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:20.780850 4442 scope.go:122] "RemoveContainer" containerID="a2d0cb9de593fb49f4e37f4f016141db890adce67e37c5b00c187440c88b18d5" Oct 30 19:00:20.781531 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:20.781490 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-2820d9-29889780-x8hc4_odh-ai-gateway-infra(88daa438-101a-4adc-8307-727c4176c705)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2820d9-29889780-x8hc4" podUID="88daa438-101a-4adc-8307-727c4176c705" Oct 30 19:00:20.815553 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:20.815525 4442 scope.go:122] "RemoveContainer" containerID="4fd9ce2c01adebe9ef897f77cf0ab8a99b2cb1c78d7132121729edb0bedfb4f1" Oct 30 19:00:20.844098 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:20.844070 4442 scope.go:122] "RemoveContainer" containerID="56036f5498cdd52d2b5a261c9b72bd9406acfd052381e44ec3c5a80337a49758" Oct 30 19:00:20.892838 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:20.892813 4442 scope.go:122] "RemoveContainer" containerID="d603451329e208c4e5a782fa813d3b63310044c242503006d5764dcffafd6922" Oct 30 19:00:20.902284 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:20.902256 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889780-bpkmv"] Oct 30 19:00:21.038165 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.038119 4442 scope.go:122] "RemoveContainer" containerID="c83e53807de77746da4bd90f979125e4e54ac2188de143bdafa02c48dc17f624" Oct 30 19:00:21.081045 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.079311 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889780-vnmx9" Oct 30 19:00:21.081306 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.081287 4442 scope.go:122] "RemoveContainer" containerID="2155792d1bff789bcf3694382ee657c4df4ba1758c7743d78d67b611523474f8" Oct 30 19:00:21.081949 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.081928 4442 scope.go:122] "RemoveContainer" containerID="884e8e2d0639c60f86eb19b9321784376270cab9f2f3554b1c7e478da4df04e8" Oct 30 19:00:21.083188 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.082666 4442 scope.go:122] "RemoveContainer" containerID="cf4e9770f01d04d12c1fcfa31983c6bd9add848982c43ab026926afbf12c89da" Oct 30 19:00:21.083188 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.083115 4442 scope.go:122] "RemoveContainer" containerID="df03fc3017b1b19ae972815bbade48c2f6fe3d23eccac93f34aa48ac602f7164" Oct 30 19:00:21.083782 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.083765 4442 scope.go:122] "RemoveContainer" containerID="bd091ebf501fa386172839cd120584d9cf257de6d0b7b9ed01a60abbfb03e594" Oct 30 19:00:21.096647 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.096392 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb187f07-08dd-40b1-bb89-7f4993d8494f" path="/var/lib/kubelet/pods/fb187f07-08dd-40b1-bb89-7f4993d8494f/volumes" Oct 30 19:00:21.097228 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.097210 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889780-jsvns"] Oct 30 19:00:21.097387 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.097376 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1aa59a-29889780-bw98v"] Oct 30 19:00:21.105272 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.105253 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9f6b7c-29889780-7dcw7"] Oct 30 19:00:21.141603 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.141275 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889780-z79ct"] Oct 30 19:00:21.191918 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.191864 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889780-2hsw5"] Oct 30 19:00:21.203325 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.203236 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889780-2hsw5"] Oct 30 19:00:21.254175 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.252777 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fcfbcab-a54b-47d4-957b-76d2cb2444f1-kube-api-access-rsltm\" (UniqueName: \"kubernetes.io/projected/9fcfbcab-a54b-47d4-957b-76d2cb2444f1-kube-api-access-rsltm\") pod \"9fcfbcab-a54b-47d4-957b-76d2cb2444f1\" (UID: \"9fcfbcab-a54b-47d4-957b-76d2cb2444f1\") " Oct 30 19:00:21.255161 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.254444 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fcfbcab-a54b-47d4-957b-76d2cb2444f1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fcfbcab-a54b-47d4-957b-76d2cb2444f1-maas-api-service-ca\") pod \"9fcfbcab-a54b-47d4-957b-76d2cb2444f1\" (UID: \"9fcfbcab-a54b-47d4-957b-76d2cb2444f1\") " Oct 30 19:00:21.267036 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.266997 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fcfbcab-a54b-47d4-957b-76d2cb2444f1-kube-api-access-rsltm" pod "9fcfbcab-a54b-47d4-957b-76d2cb2444f1" (UID: "9fcfbcab-a54b-47d4-957b-76d2cb2444f1"). InnerVolumeSpecName "kube-api-access-rsltm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:21.267462 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.267432 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fcfbcab-a54b-47d4-957b-76d2cb2444f1-maas-api-service-ca" pod "9fcfbcab-a54b-47d4-957b-76d2cb2444f1" (UID: "9fcfbcab-a54b-47d4-957b-76d2cb2444f1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:21.358978 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.358889 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsltm\" (UniqueName: \"kubernetes.io/projected/9fcfbcab-a54b-47d4-957b-76d2cb2444f1-kube-api-access-rsltm\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:21.358978 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.358920 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fcfbcab-a54b-47d4-957b-76d2cb2444f1-maas-api-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:21.365747 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.365649 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889780-bpkmv" Oct 30 19:00:21.488859 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.488822 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889780-h6xgl"] Oct 30 19:00:21.490618 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.490589 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2-maas-api-service-ca\") pod \"4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2\" (UID: \"4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2\") " Oct 30 19:00:21.502004 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.501967 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2-maas-api-service-ca" pod "4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2" (UID: "4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:21.538426 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.535227 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889780-h6xgl"] Oct 30 19:00:21.593180 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.593094 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2-kube-api-access-wsmdt\" (UniqueName: \"kubernetes.io/projected/4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2-kube-api-access-wsmdt\") pod \"4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2\" (UID: \"4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2\") " Oct 30 19:00:21.594897 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.594856 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2-maas-api-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:21.597510 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.597476 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2-kube-api-access-wsmdt" pod "4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2" (UID: "4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2"). InnerVolumeSpecName "kube-api-access-wsmdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:21.602475 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.602445 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9f6b7c-29889780-7dcw7" Oct 30 19:00:21.670918 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.670889 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889780-jsvns" Oct 30 19:00:21.696175 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.696144 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/322ab2aa-d4f3-42ef-9820-9438ced76d6b-kube-api-access-g2njt\" (UniqueName: \"kubernetes.io/projected/322ab2aa-d4f3-42ef-9820-9438ced76d6b-kube-api-access-g2njt\") pod \"322ab2aa-d4f3-42ef-9820-9438ced76d6b\" (UID: \"322ab2aa-d4f3-42ef-9820-9438ced76d6b\") " Oct 30 19:00:21.696750 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.696718 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/322ab2aa-d4f3-42ef-9820-9438ced76d6b-maas-api-service-ca" pod "322ab2aa-d4f3-42ef-9820-9438ced76d6b" (UID: "322ab2aa-d4f3-42ef-9820-9438ced76d6b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:21.697222 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.696291 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/322ab2aa-d4f3-42ef-9820-9438ced76d6b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/322ab2aa-d4f3-42ef-9820-9438ced76d6b-maas-api-service-ca\") pod \"322ab2aa-d4f3-42ef-9820-9438ced76d6b\" (UID: \"322ab2aa-d4f3-42ef-9820-9438ced76d6b\") " Oct 30 19:00:21.697403 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.697383 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4072bb06-edab-49c8-b213-a381823ec644-kube-api-access-hvz6c\" (UniqueName: \"kubernetes.io/projected/4072bb06-edab-49c8-b213-a381823ec644-kube-api-access-hvz6c\") pod \"4072bb06-edab-49c8-b213-a381823ec644\" (UID: \"4072bb06-edab-49c8-b213-a381823ec644\") " Oct 30 19:00:21.697494 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.697437 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4072bb06-edab-49c8-b213-a381823ec644-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4072bb06-edab-49c8-b213-a381823ec644-maas-api-service-ca\") pod \"4072bb06-edab-49c8-b213-a381823ec644\" (UID: \"4072bb06-edab-49c8-b213-a381823ec644\") " Oct 30 19:00:21.698345 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.698031 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4072bb06-edab-49c8-b213-a381823ec644-maas-api-service-ca" pod "4072bb06-edab-49c8-b213-a381823ec644" (UID: "4072bb06-edab-49c8-b213-a381823ec644"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:21.698345 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.698288 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4072bb06-edab-49c8-b213-a381823ec644-maas-api-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:21.698345 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.698309 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/322ab2aa-d4f3-42ef-9820-9438ced76d6b-maas-api-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:21.698345 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.698324 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsmdt\" (UniqueName: \"kubernetes.io/projected/4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2-kube-api-access-wsmdt\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:21.800834 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.800807 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9f6b7c-29889780-7dcw7" Oct 30 19:00:21.801348 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.800837 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9f6b7c-29889780-7dcw7" event={"ID":"4072bb06-edab-49c8-b213-a381823ec644","Type":"ContainerDied","Data":"27ab561b798f39a5082b62634f54f4a189b033e801565a9a4312732e32cb781a"} Oct 30 19:00:21.801348 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.800949 4442 scope.go:122] "RemoveContainer" containerID="e95ff8f2d2455013e2d16225a765dec79e351b762c1c824dab2186236e40a00f" Oct 30 19:00:21.804563 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.804539 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889780-vnmx9" event={"ID":"9fcfbcab-a54b-47d4-957b-76d2cb2444f1","Type":"ContainerDied","Data":"2ff17a72cd2ad36112e7c8c0cb1d43a2442a2dfdeff5c5375f656b6989b61ec8"} Oct 30 19:00:21.804700 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.804627 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889780-vnmx9" Oct 30 19:00:21.807252 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.807218 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889780-jsvns" Oct 30 19:00:21.807386 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.807251 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889780-jsvns" event={"ID":"322ab2aa-d4f3-42ef-9820-9438ced76d6b","Type":"ContainerDied","Data":"ef56fe24750db9c5f1676fe6ef9ded5a0afbd7da2a1b87a9d81e1ecb1045c417"} Oct 30 19:00:21.813198 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.813175 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889780-bpkmv" Oct 30 19:00:21.813353 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.813171 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889780-bpkmv" event={"ID":"4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2","Type":"ContainerDied","Data":"1cfe6587840938d23e8880be9c7a98943e75d4e243ff580538408179dfb2c766"} Oct 30 19:00:21.816575 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.816445 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889780-kp5fc"] Oct 30 19:00:21.885362 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.885335 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889780-xztsk"] Oct 30 19:00:21.944582 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.944505 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889780-vnmx9"] Oct 30 19:00:21.949887 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:21.949859 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889780-vnmx9"] Oct 30 19:00:22.000586 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.000552 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2820d9-29889780-x8hc4"] Oct 30 19:00:22.015463 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.015437 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889780-bpkmv"] Oct 30 19:00:22.022645 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.022595 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889780-bpkmv"] Oct 30 19:00:22.033020 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.032999 4442 scope.go:122] "RemoveContainer" containerID="aea98d3f54c4367eb203d1c0e0f7391f96610e5b89fc2d35a0050462cc2851e1" Oct 30 19:00:22.039753 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.039707 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/322ab2aa-d4f3-42ef-9820-9438ced76d6b-kube-api-access-g2njt" pod "322ab2aa-d4f3-42ef-9820-9438ced76d6b" (UID: "322ab2aa-d4f3-42ef-9820-9438ced76d6b"). InnerVolumeSpecName "kube-api-access-g2njt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:22.040065 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.040032 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4072bb06-edab-49c8-b213-a381823ec644-kube-api-access-hvz6c" pod "4072bb06-edab-49c8-b213-a381823ec644" (UID: "4072bb06-edab-49c8-b213-a381823ec644"). InnerVolumeSpecName "kube-api-access-hvz6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:22.104968 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.104926 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvz6c\" (UniqueName: \"kubernetes.io/projected/4072bb06-edab-49c8-b213-a381823ec644-kube-api-access-hvz6c\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:22.104968 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.104960 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2njt\" (UniqueName: \"kubernetes.io/projected/322ab2aa-d4f3-42ef-9820-9438ced76d6b-kube-api-access-g2njt\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:22.164291 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.158991 4442 scope.go:122] "RemoveContainer" containerID="bbf390fea71b273e8d6c9e7a9dc336e420a00fa15ed53b89166f3043d61acb44" Oct 30 19:00:22.197313 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.197030 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889780-z79ct" Oct 30 19:00:22.210922 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.210896 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff29d9e8-03a0-4716-840d-a32e8a5aa628-kube-api-access-hxffz\" (UniqueName: \"kubernetes.io/projected/ff29d9e8-03a0-4716-840d-a32e8a5aa628-kube-api-access-hxffz\") pod \"ff29d9e8-03a0-4716-840d-a32e8a5aa628\" (UID: \"ff29d9e8-03a0-4716-840d-a32e8a5aa628\") " Oct 30 19:00:22.214512 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.214478 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff29d9e8-03a0-4716-840d-a32e8a5aa628-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff29d9e8-03a0-4716-840d-a32e8a5aa628-maas-api-service-ca\") pod \"ff29d9e8-03a0-4716-840d-a32e8a5aa628\" (UID: \"ff29d9e8-03a0-4716-840d-a32e8a5aa628\") " Oct 30 19:00:22.228860 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.215476 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff29d9e8-03a0-4716-840d-a32e8a5aa628-maas-api-service-ca" pod "ff29d9e8-03a0-4716-840d-a32e8a5aa628" (UID: "ff29d9e8-03a0-4716-840d-a32e8a5aa628"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:22.229072 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.222985 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff29d9e8-03a0-4716-840d-a32e8a5aa628-kube-api-access-hxffz" pod "ff29d9e8-03a0-4716-840d-a32e8a5aa628" (UID: "ff29d9e8-03a0-4716-840d-a32e8a5aa628"). InnerVolumeSpecName "kube-api-access-hxffz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:22.234355 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.232804 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxffz\" (UniqueName: \"kubernetes.io/projected/ff29d9e8-03a0-4716-840d-a32e8a5aa628-kube-api-access-hxffz\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:22.234355 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.232918 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff29d9e8-03a0-4716-840d-a32e8a5aa628-maas-api-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:22.234588 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.234453 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889780-jsvns"] Oct 30 19:00:22.263190 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.255806 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889780-jsvns"] Oct 30 19:00:22.288363 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.288342 4442 scope.go:122] "RemoveContainer" containerID="1b37fe81ab2064434fa5118348f93e1abcbb0220baaf2bd7f9c984fb238728a7" Oct 30 19:00:22.297607 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.297564 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889780-kp5fc" Oct 30 19:00:22.300749 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.300343 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1aa59a-29889780-bw98v" Oct 30 19:00:22.308903 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.308849 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9f6b7c-29889780-7dcw7"] Oct 30 19:00:22.316732 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.316636 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9f6b7c-29889780-7dcw7"] Oct 30 19:00:22.335156 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.333505 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00d6e25c-6e65-45de-a894-a6d1989bfd46-kube-api-access-2pvr4\" (UniqueName: \"kubernetes.io/projected/00d6e25c-6e65-45de-a894-a6d1989bfd46-kube-api-access-2pvr4\") pod \"00d6e25c-6e65-45de-a894-a6d1989bfd46\" (UID: \"00d6e25c-6e65-45de-a894-a6d1989bfd46\") " Oct 30 19:00:22.335156 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.333709 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00d6e25c-6e65-45de-a894-a6d1989bfd46-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00d6e25c-6e65-45de-a894-a6d1989bfd46-maas-api-service-ca\") pod \"00d6e25c-6e65-45de-a894-a6d1989bfd46\" (UID: \"00d6e25c-6e65-45de-a894-a6d1989bfd46\") " Oct 30 19:00:22.335156 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.333782 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81-maas-api-service-ca\") pod \"2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81\" (UID: \"2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81\") " Oct 30 19:00:22.335156 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.333995 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81-kube-api-access-cb85x\" (UniqueName: \"kubernetes.io/projected/2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81-kube-api-access-cb85x\") pod \"2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81\" (UID: \"2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81\") " Oct 30 19:00:22.335156 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.334846 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00d6e25c-6e65-45de-a894-a6d1989bfd46-maas-api-service-ca" pod "00d6e25c-6e65-45de-a894-a6d1989bfd46" (UID: "00d6e25c-6e65-45de-a894-a6d1989bfd46"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:22.335671 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.335610 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81-maas-api-service-ca" pod "2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81" (UID: "2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:22.380457 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.380413 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00d6e25c-6e65-45de-a894-a6d1989bfd46-kube-api-access-2pvr4" pod "00d6e25c-6e65-45de-a894-a6d1989bfd46" (UID: "00d6e25c-6e65-45de-a894-a6d1989bfd46"). InnerVolumeSpecName "kube-api-access-2pvr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:22.382989 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.382961 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81-kube-api-access-cb85x" pod "2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81" (UID: "2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81"). InnerVolumeSpecName "kube-api-access-cb85x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:22.436663 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.436604 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81-maas-api-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:22.436937 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.436913 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb85x\" (UniqueName: \"kubernetes.io/projected/2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81-kube-api-access-cb85x\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:22.437091 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.437067 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pvr4\" (UniqueName: \"kubernetes.io/projected/00d6e25c-6e65-45de-a894-a6d1989bfd46-kube-api-access-2pvr4\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:22.437274 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.437257 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00d6e25c-6e65-45de-a894-a6d1989bfd46-maas-api-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:22.473791 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.473432 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889780-xztsk" Oct 30 19:00:22.501902 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.501877 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2820d9-29889780-x8hc4" Oct 30 19:00:22.538315 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.538288 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88daa438-101a-4adc-8307-727c4176c705-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88daa438-101a-4adc-8307-727c4176c705-maas-api-service-ca\") pod \"88daa438-101a-4adc-8307-727c4176c705\" (UID: \"88daa438-101a-4adc-8307-727c4176c705\") " Oct 30 19:00:22.538518 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.538429 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5abb7ef5-cc99-469a-a165-9e9394370418-kube-api-access-llpgp\" (UniqueName: \"kubernetes.io/projected/5abb7ef5-cc99-469a-a165-9e9394370418-kube-api-access-llpgp\") pod \"5abb7ef5-cc99-469a-a165-9e9394370418\" (UID: \"5abb7ef5-cc99-469a-a165-9e9394370418\") " Oct 30 19:00:22.538605 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.538578 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5abb7ef5-cc99-469a-a165-9e9394370418-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5abb7ef5-cc99-469a-a165-9e9394370418-maas-api-service-ca\") pod \"5abb7ef5-cc99-469a-a165-9e9394370418\" (UID: \"5abb7ef5-cc99-469a-a165-9e9394370418\") " Oct 30 19:00:22.538667 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.538639 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88daa438-101a-4adc-8307-727c4176c705-kube-api-access-lms5n\" (UniqueName: \"kubernetes.io/projected/88daa438-101a-4adc-8307-727c4176c705-kube-api-access-lms5n\") pod \"88daa438-101a-4adc-8307-727c4176c705\" (UID: \"88daa438-101a-4adc-8307-727c4176c705\") " Oct 30 19:00:22.539120 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.539092 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5abb7ef5-cc99-469a-a165-9e9394370418-maas-api-service-ca" pod "5abb7ef5-cc99-469a-a165-9e9394370418" (UID: "5abb7ef5-cc99-469a-a165-9e9394370418"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:22.539445 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.539428 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5abb7ef5-cc99-469a-a165-9e9394370418-maas-api-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:22.539681 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.539665 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88daa438-101a-4adc-8307-727c4176c705-maas-api-service-ca" pod "88daa438-101a-4adc-8307-727c4176c705" (UID: "88daa438-101a-4adc-8307-727c4176c705"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:22.556515 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.556477 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5abb7ef5-cc99-469a-a165-9e9394370418-kube-api-access-llpgp" pod "5abb7ef5-cc99-469a-a165-9e9394370418" (UID: "5abb7ef5-cc99-469a-a165-9e9394370418"). InnerVolumeSpecName "kube-api-access-llpgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:22.557609 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.557568 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88daa438-101a-4adc-8307-727c4176c705-kube-api-access-lms5n" pod "88daa438-101a-4adc-8307-727c4176c705" (UID: "88daa438-101a-4adc-8307-727c4176c705"). InnerVolumeSpecName "kube-api-access-lms5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:22.641037 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.641003 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88daa438-101a-4adc-8307-727c4176c705-maas-api-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:22.641037 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.641035 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llpgp\" (UniqueName: \"kubernetes.io/projected/5abb7ef5-cc99-469a-a165-9e9394370418-kube-api-access-llpgp\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:22.641289 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.641049 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lms5n\" (UniqueName: \"kubernetes.io/projected/88daa438-101a-4adc-8307-727c4176c705-kube-api-access-lms5n\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:22.840161 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.839781 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f37ca7-29889780-j2k5m" event={"ID":"1e7491f9-b178-456d-9a56-3eb6b5ee2e40","Type":"ContainerStarted","Data":"1a1618f197296f2190540af3f485fe89a511376c52b9b071155d9d13f8c679bf"} Oct 30 19:00:22.845384 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.845319 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889780-bqnkh" event={"ID":"6e894ed2-7f33-42f4-aa19-9ba21268f296","Type":"ContainerStarted","Data":"8d7ce1761b0432faff3da600980ab3da166757e2d3a7f573a0faab503503c29f"} Oct 30 19:00:22.847368 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.847339 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2820d9-29889780-x8hc4" event={"ID":"88daa438-101a-4adc-8307-727c4176c705","Type":"ContainerDied","Data":"209a92d135a6f973fcac4d44b8478333177e947ae85406a746205175de23df5a"} Oct 30 19:00:22.847494 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.847400 4442 scope.go:122] "RemoveContainer" containerID="a2d0cb9de593fb49f4e37f4f016141db890adce67e37c5b00c187440c88b18d5" Oct 30 19:00:22.847494 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.847493 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2820d9-29889780-x8hc4" Oct 30 19:00:22.856115 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.856088 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889780-kp5fc" event={"ID":"2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81","Type":"ContainerDied","Data":"6b84ea7f685f4173ebb5a869e7c08afa46bcc07ba728ccc9664ad2a4edd96849"} Oct 30 19:00:22.857630 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.857608 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889780-kp5fc" Oct 30 19:00:22.861417 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.861391 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889780-z79ct" Oct 30 19:00:22.861417 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.861400 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889780-z79ct" event={"ID":"ff29d9e8-03a0-4716-840d-a32e8a5aa628","Type":"ContainerDied","Data":"f72d8c7e12e7aac3c1999e133cfa2b4777021401466720bcdb81128d4a273853"} Oct 30 19:00:22.866764 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.866740 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1aa59a-29889780-bw98v" event={"ID":"00d6e25c-6e65-45de-a894-a6d1989bfd46","Type":"ContainerDied","Data":"e320981fb539ad0027273e788b548690b9bbb4a8812a3ee85b7ae40509f248ec"} Oct 30 19:00:22.866873 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.866841 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1aa59a-29889780-bw98v" Oct 30 19:00:22.871147 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.871105 4442 generic.go:356] "Generic (PLEG): container finished" podID="525901b4-d3b5-4bd3-8458-b4f27ec0aef2" containerID="6df369bd5d8962cb26cc3fed388c63bb6a29cbc8e28fc14f719e070a9bf58b0a" exitCode=60 Oct 30 19:00:22.871262 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.871209 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-m94vs" event={"ID":"525901b4-d3b5-4bd3-8458-b4f27ec0aef2","Type":"ContainerDied","Data":"6df369bd5d8962cb26cc3fed388c63bb6a29cbc8e28fc14f719e070a9bf58b0a"} Oct 30 19:00:22.871714 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.871693 4442 scope.go:122] "RemoveContainer" containerID="6df369bd5d8962cb26cc3fed388c63bb6a29cbc8e28fc14f719e070a9bf58b0a" Oct 30 19:00:22.872220 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:22.872193 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889780-m94vs_odh-ai-gateway-infra(525901b4-d3b5-4bd3-8458-b4f27ec0aef2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-m94vs" podUID="525901b4-d3b5-4bd3-8458-b4f27ec0aef2" Oct 30 19:00:22.875534 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.875516 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889780-xztsk" Oct 30 19:00:22.875617 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.875534 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889780-xztsk" event={"ID":"5abb7ef5-cc99-469a-a165-9e9394370418","Type":"ContainerDied","Data":"9abb0c82ca6de0fe59ef6fa6981772e0b121e96b37ae8de7c06e149c42c629c9"} Oct 30 19:00:22.877843 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.877822 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889780-7xcv7" event={"ID":"39a6d924-7557-4f9b-b481-4398a402b1bd","Type":"ContainerStarted","Data":"4a540f43ab7d5174f4b732b71720496a93d70117efeca97d68e8c5c7fafe18e2"} Oct 30 19:00:22.881212 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.881185 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889780-xvmhb" event={"ID":"9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1","Type":"ContainerStarted","Data":"be5fe3725c20d1cd578f5712fd95d3b9b297d8141334443fc69161e306c497d5"} Oct 30 19:00:22.983696 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:22.983670 4442 scope.go:122] "RemoveContainer" containerID="4e8c94b121b60090e7e7cd53d081b8911bc3a8302b60b0898e89999ecdeba781" Oct 30 19:00:23.071881 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:23.071845 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bb028e5-0be3-4923-9894-dc8f89889799" path="/var/lib/kubelet/pods/2bb028e5-0be3-4923-9894-dc8f89889799/volumes" Oct 30 19:00:23.072696 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:23.072671 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="322ab2aa-d4f3-42ef-9820-9438ced76d6b" path="/var/lib/kubelet/pods/322ab2aa-d4f3-42ef-9820-9438ced76d6b/volumes" Oct 30 19:00:23.073185 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:23.073154 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4072bb06-edab-49c8-b213-a381823ec644" path="/var/lib/kubelet/pods/4072bb06-edab-49c8-b213-a381823ec644/volumes" Oct 30 19:00:23.074872 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:23.074850 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2" path="/var/lib/kubelet/pods/4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2/volumes" Oct 30 19:00:23.075431 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:23.075411 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fcfbcab-a54b-47d4-957b-76d2cb2444f1" path="/var/lib/kubelet/pods/9fcfbcab-a54b-47d4-957b-76d2cb2444f1/volumes" Oct 30 19:00:23.075908 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:23.075892 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2ba9730-da34-4745-875f-cd941e32eb79" path="/var/lib/kubelet/pods/a2ba9730-da34-4745-875f-cd941e32eb79/volumes" Oct 30 19:00:23.132696 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:23.132654 4442 scope.go:122] "RemoveContainer" containerID="5c3641646a5513898dd658f5f5982aa0f5d10fa339285ae26bbe345df16b2657" Oct 30 19:00:23.156078 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:23.156052 4442 scope.go:122] "RemoveContainer" containerID="25d6c35d3c77abdd5d1318310c4443c17bfea69f32de7e86c26ba4ae01925b10" Oct 30 19:00:23.206253 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:23.206225 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1aa59a-29889780-bw98v"] Oct 30 19:00:23.210722 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:23.210683 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1aa59a-765b7b94b7-s9pmv"] Oct 30 19:00:23.213667 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:23.213646 4442 scope.go:122] "RemoveContainer" containerID="884e8e2d0639c60f86eb19b9321784376270cab9f2f3554b1c7e478da4df04e8" Oct 30 19:00:23.216292 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:23.216273 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1aa59a-29889780-bw98v"] Oct 30 19:00:23.236894 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:23.232892 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889780-z79ct"] Oct 30 19:00:23.236894 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:23.236127 4442 scope.go:122] "RemoveContainer" containerID="9caaafc14f8b58738833cc3d109f28e24a1318927cb39db5a455819adfe19dfb" Oct 30 19:00:23.253669 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:23.253642 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889780-z79ct"] Oct 30 19:00:23.287477 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:23.287402 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2820d9-29889780-x8hc4"] Oct 30 19:00:23.297892 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:23.297856 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2820d9-29889780-x8hc4"] Oct 30 19:00:23.323990 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:23.323385 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889780-kp5fc"] Oct 30 19:00:23.333458 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:23.333429 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889780-kp5fc"] Oct 30 19:00:23.949228 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:23.949191 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f37ca7-29889780-j2k5m"] Oct 30 19:00:23.974424 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:23.974391 4442 generic.go:356] "Generic (PLEG): container finished" podID="6e894ed2-7f33-42f4-aa19-9ba21268f296" containerID="8d7ce1761b0432faff3da600980ab3da166757e2d3a7f573a0faab503503c29f" exitCode=60 Oct 30 19:00:23.974789 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:23.974760 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889780-bqnkh" event={"ID":"6e894ed2-7f33-42f4-aa19-9ba21268f296","Type":"ContainerDied","Data":"8d7ce1761b0432faff3da600980ab3da166757e2d3a7f573a0faab503503c29f"} Oct 30 19:00:23.974989 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:23.974970 4442 scope.go:122] "RemoveContainer" containerID="cf4e9770f01d04d12c1fcfa31983c6bd9add848982c43ab026926afbf12c89da" Oct 30 19:00:23.975595 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:23.975577 4442 scope.go:122] "RemoveContainer" containerID="8d7ce1761b0432faff3da600980ab3da166757e2d3a7f573a0faab503503c29f" Oct 30 19:00:23.976167 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:23.976121 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889780-bqnkh_odh-ai-gateway-infra(6e894ed2-7f33-42f4-aa19-9ba21268f296)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889780-bqnkh" podUID="6e894ed2-7f33-42f4-aa19-9ba21268f296" Oct 30 19:00:24.040066 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:24.040028 4442 generic.go:356] "Generic (PLEG): container finished" podID="39a6d924-7557-4f9b-b481-4398a402b1bd" containerID="4a540f43ab7d5174f4b732b71720496a93d70117efeca97d68e8c5c7fafe18e2" exitCode=60 Oct 30 19:00:24.040422 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:24.040399 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889780-7xcv7" event={"ID":"39a6d924-7557-4f9b-b481-4398a402b1bd","Type":"ContainerDied","Data":"4a540f43ab7d5174f4b732b71720496a93d70117efeca97d68e8c5c7fafe18e2"} Oct 30 19:00:24.041044 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:24.041023 4442 scope.go:122] "RemoveContainer" containerID="4a540f43ab7d5174f4b732b71720496a93d70117efeca97d68e8c5c7fafe18e2" Oct 30 19:00:24.048598 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:24.048568 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-39c99d-29889780-7xcv7_odh-ai-gateway-infra(39a6d924-7557-4f9b-b481-4398a402b1bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889780-7xcv7" podUID="39a6d924-7557-4f9b-b481-4398a402b1bd" Oct 30 19:00:24.067603 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:24.067568 4442 generic.go:356] "Generic (PLEG): container finished" podID="9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1" containerID="be5fe3725c20d1cd578f5712fd95d3b9b297d8141334443fc69161e306c497d5" exitCode=60 Oct 30 19:00:24.094237 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:24.094210 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889780-bqnkh"] Oct 30 19:00:24.094470 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:24.094449 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889780-xvmhb" event={"ID":"9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1","Type":"ContainerDied","Data":"be5fe3725c20d1cd578f5712fd95d3b9b297d8141334443fc69161e306c497d5"} Oct 30 19:00:24.095062 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:24.095041 4442 scope.go:122] "RemoveContainer" containerID="be5fe3725c20d1cd578f5712fd95d3b9b297d8141334443fc69161e306c497d5" Oct 30 19:00:24.095679 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:24.095651 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889780-xvmhb_odh-ai-gateway-infra(9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889780-xvmhb" podUID="9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1" Oct 30 19:00:24.096197 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:24.096177 4442 scope.go:122] "RemoveContainer" containerID="bd091ebf501fa386172839cd120584d9cf257de6d0b7b9ed01a60abbfb03e594" Oct 30 19:00:24.106280 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:24.106254 4442 generic.go:356] "Generic (PLEG): container finished" podID="1e7491f9-b178-456d-9a56-3eb6b5ee2e40" containerID="1a1618f197296f2190540af3f485fe89a511376c52b9b071155d9d13f8c679bf" exitCode=60 Oct 30 19:00:24.110704 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:24.107035 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f37ca7-29889780-j2k5m" event={"ID":"1e7491f9-b178-456d-9a56-3eb6b5ee2e40","Type":"ContainerDied","Data":"1a1618f197296f2190540af3f485fe89a511376c52b9b071155d9d13f8c679bf"} Oct 30 19:00:24.142750 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:24.142724 4442 scope.go:122] "RemoveContainer" containerID="df03fc3017b1b19ae972815bbade48c2f6fe3d23eccac93f34aa48ac602f7164" Oct 30 19:00:24.158116 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:24.158078 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889780-7xcv7"] Oct 30 19:00:24.204902 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:24.204603 4442 scope.go:122] "RemoveContainer" containerID="2155792d1bff789bcf3694382ee657c4df4ba1758c7743d78d67b611523474f8" Oct 30 19:00:24.211852 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:24.211769 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889780-m94vs"] Oct 30 19:00:24.398786 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:24.398757 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889780-xvmhb"] Oct 30 19:00:24.467561 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:24.467477 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f37ca7-29889780-j2k5m" Oct 30 19:00:24.471852 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:24.471825 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-m94vs" Oct 30 19:00:24.572218 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:24.572178 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/525901b4-d3b5-4bd3-8458-b4f27ec0aef2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/525901b4-d3b5-4bd3-8458-b4f27ec0aef2-maas-api-service-ca\") pod \"525901b4-d3b5-4bd3-8458-b4f27ec0aef2\" (UID: \"525901b4-d3b5-4bd3-8458-b4f27ec0aef2\") " Oct 30 19:00:24.572478 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:24.572456 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e7491f9-b178-456d-9a56-3eb6b5ee2e40-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e7491f9-b178-456d-9a56-3eb6b5ee2e40-maas-api-service-ca\") pod \"1e7491f9-b178-456d-9a56-3eb6b5ee2e40\" (UID: \"1e7491f9-b178-456d-9a56-3eb6b5ee2e40\") " Oct 30 19:00:24.572547 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:24.572533 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e7491f9-b178-456d-9a56-3eb6b5ee2e40-kube-api-access-97wqt\" (UniqueName: \"kubernetes.io/projected/1e7491f9-b178-456d-9a56-3eb6b5ee2e40-kube-api-access-97wqt\") pod \"1e7491f9-b178-456d-9a56-3eb6b5ee2e40\" (UID: \"1e7491f9-b178-456d-9a56-3eb6b5ee2e40\") " Oct 30 19:00:24.572605 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:24.572576 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/525901b4-d3b5-4bd3-8458-b4f27ec0aef2-kube-api-access-7gzkn\" (UniqueName: \"kubernetes.io/projected/525901b4-d3b5-4bd3-8458-b4f27ec0aef2-kube-api-access-7gzkn\") pod \"525901b4-d3b5-4bd3-8458-b4f27ec0aef2\" (UID: \"525901b4-d3b5-4bd3-8458-b4f27ec0aef2\") " Oct 30 19:00:24.577692 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:24.577663 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/525901b4-d3b5-4bd3-8458-b4f27ec0aef2-maas-api-service-ca" pod "525901b4-d3b5-4bd3-8458-b4f27ec0aef2" (UID: "525901b4-d3b5-4bd3-8458-b4f27ec0aef2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:24.580653 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:24.580615 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/525901b4-d3b5-4bd3-8458-b4f27ec0aef2-kube-api-access-7gzkn" pod "525901b4-d3b5-4bd3-8458-b4f27ec0aef2" (UID: "525901b4-d3b5-4bd3-8458-b4f27ec0aef2"). InnerVolumeSpecName "kube-api-access-7gzkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:24.581418 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:24.581377 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e7491f9-b178-456d-9a56-3eb6b5ee2e40-maas-api-service-ca" pod "1e7491f9-b178-456d-9a56-3eb6b5ee2e40" (UID: "1e7491f9-b178-456d-9a56-3eb6b5ee2e40"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:24.582288 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:24.582260 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e7491f9-b178-456d-9a56-3eb6b5ee2e40-kube-api-access-97wqt" pod "1e7491f9-b178-456d-9a56-3eb6b5ee2e40" (UID: "1e7491f9-b178-456d-9a56-3eb6b5ee2e40"). InnerVolumeSpecName "kube-api-access-97wqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:24.674852 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:24.674817 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e7491f9-b178-456d-9a56-3eb6b5ee2e40-maas-api-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:24.674852 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:24.674848 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97wqt\" (UniqueName: \"kubernetes.io/projected/1e7491f9-b178-456d-9a56-3eb6b5ee2e40-kube-api-access-97wqt\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:24.674852 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:24.674859 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gzkn\" (UniqueName: \"kubernetes.io/projected/525901b4-d3b5-4bd3-8458-b4f27ec0aef2-kube-api-access-7gzkn\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:24.675073 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:24.674867 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/525901b4-d3b5-4bd3-8458-b4f27ec0aef2-maas-api-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:25.093712 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:25.093665 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00d6e25c-6e65-45de-a894-a6d1989bfd46" path="/var/lib/kubelet/pods/00d6e25c-6e65-45de-a894-a6d1989bfd46/volumes" Oct 30 19:00:25.094432 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:25.094407 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81" path="/var/lib/kubelet/pods/2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81/volumes" Oct 30 19:00:25.095000 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:25.094978 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88daa438-101a-4adc-8307-727c4176c705" path="/var/lib/kubelet/pods/88daa438-101a-4adc-8307-727c4176c705/volumes" Oct 30 19:00:25.095931 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:25.095911 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff29d9e8-03a0-4716-840d-a32e8a5aa628" path="/var/lib/kubelet/pods/ff29d9e8-03a0-4716-840d-a32e8a5aa628/volumes" Oct 30 19:00:25.120960 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:25.120931 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f37ca7-29889780-j2k5m" Oct 30 19:00:25.121123 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:25.120951 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f37ca7-29889780-j2k5m" event={"ID":"1e7491f9-b178-456d-9a56-3eb6b5ee2e40","Type":"ContainerDied","Data":"9b410e2c97ef0f9507a69d6d2254d4aaca52e657c1f5bef3dea9919f22384533"} Oct 30 19:00:25.121123 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:25.121021 4442 scope.go:122] "RemoveContainer" containerID="1a1618f197296f2190540af3f485fe89a511376c52b9b071155d9d13f8c679bf" Oct 30 19:00:25.126024 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:25.125994 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-m94vs" event={"ID":"525901b4-d3b5-4bd3-8458-b4f27ec0aef2","Type":"ContainerDied","Data":"e06b83314f88ea0bd06ccdeaccaaa13be24f8e426983faaada0b9d0fee794f14"} Oct 30 19:00:25.126148 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:25.126114 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-m94vs" Oct 30 19:00:25.136837 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:25.136819 4442 scope.go:122] "RemoveContainer" containerID="6df369bd5d8962cb26cc3fed388c63bb6a29cbc8e28fc14f719e070a9bf58b0a" Oct 30 19:00:25.184196 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:25.182481 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f37ca7-29889780-j2k5m"] Oct 30 19:00:25.189125 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:25.189081 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f37ca7-29889780-j2k5m"] Oct 30 19:00:25.220521 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:25.220302 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889780-m94vs"] Oct 30 19:00:25.241315 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:25.241281 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889780-m94vs"] Oct 30 19:00:25.346935 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:25.346398 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889780-bqnkh" Oct 30 19:00:25.471747 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:25.471712 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889780-xvmhb" Oct 30 19:00:25.476589 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:25.476564 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889780-7xcv7" Oct 30 19:00:25.494890 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:25.494862 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e894ed2-7f33-42f4-aa19-9ba21268f296-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e894ed2-7f33-42f4-aa19-9ba21268f296-maas-api-service-ca\") pod \"6e894ed2-7f33-42f4-aa19-9ba21268f296\" (UID: \"6e894ed2-7f33-42f4-aa19-9ba21268f296\") " Oct 30 19:00:25.495055 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:25.494929 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e894ed2-7f33-42f4-aa19-9ba21268f296-kube-api-access-qmt2t\" (UniqueName: \"kubernetes.io/projected/6e894ed2-7f33-42f4-aa19-9ba21268f296-kube-api-access-qmt2t\") pod \"6e894ed2-7f33-42f4-aa19-9ba21268f296\" (UID: \"6e894ed2-7f33-42f4-aa19-9ba21268f296\") " Oct 30 19:00:25.495199 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:25.495177 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e894ed2-7f33-42f4-aa19-9ba21268f296-maas-api-service-ca" pod "6e894ed2-7f33-42f4-aa19-9ba21268f296" (UID: "6e894ed2-7f33-42f4-aa19-9ba21268f296"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:25.495788 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:25.495769 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e894ed2-7f33-42f4-aa19-9ba21268f296-maas-api-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:25.497367 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:25.497340 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e894ed2-7f33-42f4-aa19-9ba21268f296-kube-api-access-qmt2t" pod "6e894ed2-7f33-42f4-aa19-9ba21268f296" (UID: "6e894ed2-7f33-42f4-aa19-9ba21268f296"). InnerVolumeSpecName "kube-api-access-qmt2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:25.597443 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:25.597344 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1-maas-api-service-ca\") pod \"9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1\" (UID: \"9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1\") " Oct 30 19:00:25.597443 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:25.597412 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39a6d924-7557-4f9b-b481-4398a402b1bd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39a6d924-7557-4f9b-b481-4398a402b1bd-maas-api-service-ca\") pod \"39a6d924-7557-4f9b-b481-4398a402b1bd\" (UID: \"39a6d924-7557-4f9b-b481-4398a402b1bd\") " Oct 30 19:00:25.597783 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:25.597589 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1-kube-api-access-bxl9w\" (UniqueName: \"kubernetes.io/projected/9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1-kube-api-access-bxl9w\") pod \"9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1\" (UID: \"9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1\") " Oct 30 19:00:25.597783 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:25.597696 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39a6d924-7557-4f9b-b481-4398a402b1bd-kube-api-access-s2zr6\" (UniqueName: \"kubernetes.io/projected/39a6d924-7557-4f9b-b481-4398a402b1bd-kube-api-access-s2zr6\") pod \"39a6d924-7557-4f9b-b481-4398a402b1bd\" (UID: \"39a6d924-7557-4f9b-b481-4398a402b1bd\") " Oct 30 19:00:25.598351 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:25.598325 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1-maas-api-service-ca" pod "9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1" (UID: "9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:25.598504 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:25.598420 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmt2t\" (UniqueName: \"kubernetes.io/projected/6e894ed2-7f33-42f4-aa19-9ba21268f296-kube-api-access-qmt2t\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:25.599027 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:25.598987 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39a6d924-7557-4f9b-b481-4398a402b1bd-maas-api-service-ca" pod "39a6d924-7557-4f9b-b481-4398a402b1bd" (UID: "39a6d924-7557-4f9b-b481-4398a402b1bd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:25.600980 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:25.600953 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1-kube-api-access-bxl9w" pod "9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1" (UID: "9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1"). InnerVolumeSpecName "kube-api-access-bxl9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:25.601282 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:25.601118 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39a6d924-7557-4f9b-b481-4398a402b1bd-kube-api-access-s2zr6" pod "39a6d924-7557-4f9b-b481-4398a402b1bd" (UID: "39a6d924-7557-4f9b-b481-4398a402b1bd"). InnerVolumeSpecName "kube-api-access-s2zr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:25.700440 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:25.700401 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1-maas-api-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:25.700440 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:25.700435 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39a6d924-7557-4f9b-b481-4398a402b1bd-maas-api-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:25.700440 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:25.700449 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxl9w\" (UniqueName: \"kubernetes.io/projected/9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1-kube-api-access-bxl9w\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:25.700697 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:25.700458 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2zr6\" (UniqueName: \"kubernetes.io/projected/39a6d924-7557-4f9b-b481-4398a402b1bd-kube-api-access-s2zr6\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:25.879182 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:25.879123 4442 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5abb7ef5_cc99_469a_a165_9e9394370418.slice\": RecentStats: unable to find data in memory cache]" Oct 30 19:00:25.879437 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:25.879410 4442 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5abb7ef5_cc99_469a_a165_9e9394370418.slice\": RecentStats: unable to find data in memory cache]" Oct 30 19:00:26.147402 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:26.147314 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889780-7xcv7" Oct 30 19:00:26.147402 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:26.147332 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889780-7xcv7" event={"ID":"39a6d924-7557-4f9b-b481-4398a402b1bd","Type":"ContainerDied","Data":"d4d98dccaa8581b683ab56189c150d3e2eec04428938415806a92a5ae3e17dc9"} Oct 30 19:00:26.147402 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:26.147401 4442 scope.go:122] "RemoveContainer" containerID="4a540f43ab7d5174f4b732b71720496a93d70117efeca97d68e8c5c7fafe18e2" Oct 30 19:00:26.149321 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:26.149296 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889780-xvmhb" Oct 30 19:00:26.149321 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:26.149309 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889780-xvmhb" event={"ID":"9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1","Type":"ContainerDied","Data":"d55ce94070dfce30947f5552869a72291b76899312072e2dd2927b2ca9288f23"} Oct 30 19:00:26.152673 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:26.152646 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889780-bqnkh" Oct 30 19:00:26.152840 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:26.152809 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889780-bqnkh" event={"ID":"6e894ed2-7f33-42f4-aa19-9ba21268f296","Type":"ContainerDied","Data":"d4d36fc7fbcf75774f79b42561560887510c741ff420b9d0b2cbbd62ba1e1c54"} Oct 30 19:00:26.163053 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:26.163033 4442 scope.go:122] "RemoveContainer" containerID="be5fe3725c20d1cd578f5712fd95d3b9b297d8141334443fc69161e306c497d5" Oct 30 19:00:26.176872 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:26.176850 4442 scope.go:122] "RemoveContainer" containerID="8d7ce1761b0432faff3da600980ab3da166757e2d3a7f573a0faab503503c29f" Oct 30 19:00:26.200602 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:26.200573 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889780-xvmhb"] Oct 30 19:00:26.206189 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:26.206164 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889780-xvmhb"] Oct 30 19:00:26.253501 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:26.253469 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889780-7xcv7"] Oct 30 19:00:26.258514 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:26.258488 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-39c99d-29889780-7xcv7"] Oct 30 19:00:26.289321 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:26.289291 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889780-bqnkh"] Oct 30 19:00:26.293919 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:26.293898 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c095fa-29889780-bqnkh"] Oct 30 19:00:27.071173 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:27.071118 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e7491f9-b178-456d-9a56-3eb6b5ee2e40" path="/var/lib/kubelet/pods/1e7491f9-b178-456d-9a56-3eb6b5ee2e40/volumes" Oct 30 19:00:27.071897 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:27.071876 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39a6d924-7557-4f9b-b481-4398a402b1bd" path="/var/lib/kubelet/pods/39a6d924-7557-4f9b-b481-4398a402b1bd/volumes" Oct 30 19:00:27.072524 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:27.072505 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="525901b4-d3b5-4bd3-8458-b4f27ec0aef2" path="/var/lib/kubelet/pods/525901b4-d3b5-4bd3-8458-b4f27ec0aef2/volumes" Oct 30 19:00:27.073577 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:27.073555 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e894ed2-7f33-42f4-aa19-9ba21268f296" path="/var/lib/kubelet/pods/6e894ed2-7f33-42f4-aa19-9ba21268f296/volumes" Oct 30 19:00:27.074232 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:27.074213 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1" path="/var/lib/kubelet/pods/9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1/volumes" Oct 30 19:00:27.283692 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:27.283651 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-551d0b/test-model-551d0b-kserve-65db7ffb5-b828f" podUID="2ce11bc9-b3ab-4302-8338-4fa4d3db2afb" containerName="main" containerID="cri-o://5d1e353a8ff4ca413e7790a9440a52cd8e1f91766aa5424f6fbb8d2c7a4d4896" gracePeriod=45 Oct 30 19:00:27.609860 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:27.609832 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-551d0b/test-model-551d0b-kserve-65db7ffb5-b828f" Oct 30 19:00:27.735216 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:27.734623 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ce11bc9-b3ab-4302-8338-4fa4d3db2afb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ce11bc9-b3ab-4302-8338-4fa4d3db2afb-dshm\") pod \"2ce11bc9-b3ab-4302-8338-4fa4d3db2afb\" (UID: \"2ce11bc9-b3ab-4302-8338-4fa4d3db2afb\") " Oct 30 19:00:27.735216 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:27.734736 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ce11bc9-b3ab-4302-8338-4fa4d3db2afb-home\" (UniqueName: \"kubernetes.io/empty-dir/2ce11bc9-b3ab-4302-8338-4fa4d3db2afb-home\") pod \"2ce11bc9-b3ab-4302-8338-4fa4d3db2afb\" (UID: \"2ce11bc9-b3ab-4302-8338-4fa4d3db2afb\") " Oct 30 19:00:27.735216 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:27.734894 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ce11bc9-b3ab-4302-8338-4fa4d3db2afb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ce11bc9-b3ab-4302-8338-4fa4d3db2afb-tmp-dir\") pod \"2ce11bc9-b3ab-4302-8338-4fa4d3db2afb\" (UID: \"2ce11bc9-b3ab-4302-8338-4fa4d3db2afb\") " Oct 30 19:00:27.735216 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:27.734969 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ce11bc9-b3ab-4302-8338-4fa4d3db2afb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ce11bc9-b3ab-4302-8338-4fa4d3db2afb-model-cache\") pod \"2ce11bc9-b3ab-4302-8338-4fa4d3db2afb\" (UID: \"2ce11bc9-b3ab-4302-8338-4fa4d3db2afb\") " Oct 30 19:00:27.735216 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:27.735047 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ce11bc9-b3ab-4302-8338-4fa4d3db2afb-kube-api-access-x47sr\" (UniqueName: \"kubernetes.io/projected/2ce11bc9-b3ab-4302-8338-4fa4d3db2afb-kube-api-access-x47sr\") pod \"2ce11bc9-b3ab-4302-8338-4fa4d3db2afb\" (UID: \"2ce11bc9-b3ab-4302-8338-4fa4d3db2afb\") " Oct 30 19:00:27.735621 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:27.735228 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ce11bc9-b3ab-4302-8338-4fa4d3db2afb-tls-certs\" (UniqueName: \"kubernetes.io/secret/2ce11bc9-b3ab-4302-8338-4fa4d3db2afb-tls-certs\") pod \"2ce11bc9-b3ab-4302-8338-4fa4d3db2afb\" (UID: \"2ce11bc9-b3ab-4302-8338-4fa4d3db2afb\") " Oct 30 19:00:27.841162 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:27.824446 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ce11bc9-b3ab-4302-8338-4fa4d3db2afb-home" pod "2ce11bc9-b3ab-4302-8338-4fa4d3db2afb" (UID: "2ce11bc9-b3ab-4302-8338-4fa4d3db2afb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:27.841162 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:27.824664 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ce11bc9-b3ab-4302-8338-4fa4d3db2afb-model-cache" pod "2ce11bc9-b3ab-4302-8338-4fa4d3db2afb" (UID: "2ce11bc9-b3ab-4302-8338-4fa4d3db2afb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:27.857157 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:27.843235 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ce11bc9-b3ab-4302-8338-4fa4d3db2afb-tmp-dir" pod "2ce11bc9-b3ab-4302-8338-4fa4d3db2afb" (UID: "2ce11bc9-b3ab-4302-8338-4fa4d3db2afb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:27.857157 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:27.844798 4442 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ce11bc9-b3ab-4302-8338-4fa4d3db2afb-home\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:27.857157 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:27.844822 4442 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ce11bc9-b3ab-4302-8338-4fa4d3db2afb-tmp-dir\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:27.857157 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:27.844836 4442 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ce11bc9-b3ab-4302-8338-4fa4d3db2afb-model-cache\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:27.899156 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:27.889817 4442 scope.go:122] "RemoveContainer" containerID="5d1e353a8ff4ca413e7790a9440a52cd8e1f91766aa5424f6fbb8d2c7a4d4896" Oct 30 19:00:27.965158 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:27.952439 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ce11bc9-b3ab-4302-8338-4fa4d3db2afb-kube-api-access-x47sr" pod "2ce11bc9-b3ab-4302-8338-4fa4d3db2afb" (UID: "2ce11bc9-b3ab-4302-8338-4fa4d3db2afb"). InnerVolumeSpecName "kube-api-access-x47sr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:28.072448 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:28.053662 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x47sr\" (UniqueName: \"kubernetes.io/projected/2ce11bc9-b3ab-4302-8338-4fa4d3db2afb-kube-api-access-x47sr\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:28.208165 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:28.200320 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ce11bc9-b3ab-4302-8338-4fa4d3db2afb-tls-certs" pod "2ce11bc9-b3ab-4302-8338-4fa4d3db2afb" (UID: "2ce11bc9-b3ab-4302-8338-4fa4d3db2afb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:00:28.237165 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:28.219289 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1aa59a-765b7b94b7-s9pmv" podUID="8b79c781-9ddc-473d-8285-120dffb8e016" containerName="payload-pre-processing" containerID="cri-o://9195959c568f767238f3d4791ee1266e18d693a2225e13cfc308e4e4a0e84c07" gracePeriod=25 Oct 30 19:00:28.270219 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:28.265456 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ce11bc9-b3ab-4302-8338-4fa4d3db2afb-dshm" pod "2ce11bc9-b3ab-4302-8338-4fa4d3db2afb" (UID: "2ce11bc9-b3ab-4302-8338-4fa4d3db2afb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:28.284164 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:28.278111 4442 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ce11bc9-b3ab-4302-8338-4fa4d3db2afb-tls-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:28.284164 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:28.278159 4442 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ce11bc9-b3ab-4302-8338-4fa4d3db2afb-dshm\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:28.401334 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:28.401302 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-551d0b/test-model-551d0b-kserve-65db7ffb5-b828f" Oct 30 19:00:28.401615 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:28.401299 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-551d0b/test-model-551d0b-kserve-65db7ffb5-b828f" event={"ID":"2ce11bc9-b3ab-4302-8338-4fa4d3db2afb","Type":"ContainerDied","Data":"5d1e353a8ff4ca413e7790a9440a52cd8e1f91766aa5424f6fbb8d2c7a4d4896"} Oct 30 19:00:28.401615 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:28.401585 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-551d0b/test-model-551d0b-kserve-65db7ffb5-b828f" event={"ID":"2ce11bc9-b3ab-4302-8338-4fa4d3db2afb","Type":"ContainerDied","Data":"a0d093c2b2d91ebbc030e6a6d8a503ea2ca5227b5584a5e37a50ddd480c5222a"} Oct 30 19:00:28.460232 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:28.460191 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-551d0b/test-model-551d0b-kserve-65db7ffb5-b828f"] Oct 30 19:00:28.475852 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:28.475824 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-551d0b/test-model-551d0b-kserve-65db7ffb5-b828f"] Oct 30 19:00:28.516887 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:28.516836 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-551d0b/test-model-551d0b-kserve-65db7ffb5-b828f" podUID="2ce11bc9-b3ab-4302-8338-4fa4d3db2afb" containerName="main" probeResult="failure" output="Get \"https://10.134.1.10:8000/ready\": context deadline exceeded" Oct 30 19:00:29.068921 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:29.068881 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ce11bc9-b3ab-4302-8338-4fa4d3db2afb" path="/var/lib/kubelet/pods/2ce11bc9-b3ab-4302-8338-4fa4d3db2afb/volumes" Oct 30 19:00:31.625935 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:31.625899 4442 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5abb7ef5_cc99_469a_a165_9e9394370418.slice\": RecentStats: unable to find data in memory cache]" Oct 30 19:00:31.681991 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:31.681954 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1aa59a-765b7b94b7-s9pmv" podUID="8b79c781-9ddc-473d-8285-120dffb8e016" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:00:34.537766 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.537685 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-daf3ba/test-model-daf3ba-kserve-84cd9f6df-bfn9m"] Oct 30 19:00:34.539012 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.538992 4442 cpu_manager.go:414] "containerMap: removing container" podUID="39a6d924-7557-4f9b-b481-4398a402b1bd" containerName="cleanup" Oct 30 19:00:34.539106 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539016 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39a6d924-7557-4f9b-b481-4398a402b1bd" containerName="cleanup" Oct 30 19:00:34.539106 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539034 4442 cpu_manager.go:414] "containerMap: removing container" podUID="9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1" containerName="cleanup" Oct 30 19:00:34.539106 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539042 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1" containerName="cleanup" Oct 30 19:00:34.539106 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539052 4442 cpu_manager.go:414] "containerMap: removing container" podUID="2bb028e5-0be3-4923-9894-dc8f89889799" containerName="cleanup" Oct 30 19:00:34.539106 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539061 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bb028e5-0be3-4923-9894-dc8f89889799" containerName="cleanup" Oct 30 19:00:34.539106 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539074 4442 cpu_manager.go:414] "containerMap: removing container" podUID="4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2" containerName="cleanup" Oct 30 19:00:34.539106 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539082 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2" containerName="cleanup" Oct 30 19:00:34.539106 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539097 4442 cpu_manager.go:414] "containerMap: removing container" podUID="322ab2aa-d4f3-42ef-9820-9438ced76d6b" containerName="cleanup" Oct 30 19:00:34.539106 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539104 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="322ab2aa-d4f3-42ef-9820-9438ced76d6b" containerName="cleanup" Oct 30 19:00:34.539438 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539115 4442 cpu_manager.go:414] "containerMap: removing container" podUID="1e7491f9-b178-456d-9a56-3eb6b5ee2e40" containerName="cleanup" Oct 30 19:00:34.539438 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539122 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e7491f9-b178-456d-9a56-3eb6b5ee2e40" containerName="cleanup" Oct 30 19:00:34.539438 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539153 4442 cpu_manager.go:414] "containerMap: removing container" podUID="2ce11bc9-b3ab-4302-8338-4fa4d3db2afb" containerName="main" Oct 30 19:00:34.539438 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539160 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ce11bc9-b3ab-4302-8338-4fa4d3db2afb" containerName="main" Oct 30 19:00:34.539438 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539171 4442 cpu_manager.go:414] "containerMap: removing container" podUID="88daa438-101a-4adc-8307-727c4176c705" containerName="cleanup" Oct 30 19:00:34.539438 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539178 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88daa438-101a-4adc-8307-727c4176c705" containerName="cleanup" Oct 30 19:00:34.539438 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539189 4442 cpu_manager.go:414] "containerMap: removing container" podUID="c7e7ff65-c137-48d2-a189-eb7f64bb1ff2" containerName="istio-proxy" Oct 30 19:00:34.539438 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539197 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7e7ff65-c137-48d2-a189-eb7f64bb1ff2" containerName="istio-proxy" Oct 30 19:00:34.539438 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539210 4442 cpu_manager.go:414] "containerMap: removing container" podUID="a2ba9730-da34-4745-875f-cd941e32eb79" containerName="cleanup" Oct 30 19:00:34.539438 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539216 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2ba9730-da34-4745-875f-cd941e32eb79" containerName="cleanup" Oct 30 19:00:34.539438 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539224 4442 cpu_manager.go:414] "containerMap: removing container" podUID="525901b4-d3b5-4bd3-8458-b4f27ec0aef2" containerName="cleanup" Oct 30 19:00:34.539438 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539232 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="525901b4-d3b5-4bd3-8458-b4f27ec0aef2" containerName="cleanup" Oct 30 19:00:34.539438 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539248 4442 cpu_manager.go:414] "containerMap: removing container" podUID="6e894ed2-7f33-42f4-aa19-9ba21268f296" containerName="cleanup" Oct 30 19:00:34.539438 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539255 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e894ed2-7f33-42f4-aa19-9ba21268f296" containerName="cleanup" Oct 30 19:00:34.539438 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539269 4442 cpu_manager.go:414] "containerMap: removing container" podUID="39a6d924-7557-4f9b-b481-4398a402b1bd" containerName="cleanup" Oct 30 19:00:34.539438 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539276 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39a6d924-7557-4f9b-b481-4398a402b1bd" containerName="cleanup" Oct 30 19:00:34.539438 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539288 4442 cpu_manager.go:414] "containerMap: removing container" podUID="6e894ed2-7f33-42f4-aa19-9ba21268f296" containerName="cleanup" Oct 30 19:00:34.539438 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539296 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e894ed2-7f33-42f4-aa19-9ba21268f296" containerName="cleanup" Oct 30 19:00:34.539438 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539308 4442 cpu_manager.go:414] "containerMap: removing container" podUID="c7e7ff65-c137-48d2-a189-eb7f64bb1ff2" containerName="istio-proxy" Oct 30 19:00:34.539438 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539315 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7e7ff65-c137-48d2-a189-eb7f64bb1ff2" containerName="istio-proxy" Oct 30 19:00:34.539438 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539324 4442 cpu_manager.go:414] "containerMap: removing container" podUID="5abb7ef5-cc99-469a-a165-9e9394370418" containerName="cleanup" Oct 30 19:00:34.539438 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539330 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5abb7ef5-cc99-469a-a165-9e9394370418" containerName="cleanup" Oct 30 19:00:34.539438 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539349 4442 cpu_manager.go:414] "containerMap: removing container" podUID="00d6e25c-6e65-45de-a894-a6d1989bfd46" containerName="cleanup" Oct 30 19:00:34.539438 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539357 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00d6e25c-6e65-45de-a894-a6d1989bfd46" containerName="cleanup" Oct 30 19:00:34.539438 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539369 4442 cpu_manager.go:414] "containerMap: removing container" podUID="9fcfbcab-a54b-47d4-957b-76d2cb2444f1" containerName="cleanup" Oct 30 19:00:34.539438 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539377 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fcfbcab-a54b-47d4-957b-76d2cb2444f1" containerName="cleanup" Oct 30 19:00:34.539438 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539388 4442 cpu_manager.go:414] "containerMap: removing container" podUID="422f6f0d-871f-44d2-8fba-2dc65d684ced" containerName="revoke-keys" Oct 30 19:00:34.539438 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539395 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="422f6f0d-871f-44d2-8fba-2dc65d684ced" containerName="revoke-keys" Oct 30 19:00:34.539438 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539405 4442 cpu_manager.go:414] "containerMap: removing container" podUID="2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81" containerName="cleanup" Oct 30 19:00:34.539438 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539411 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81" containerName="cleanup" Oct 30 19:00:34.539438 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539431 4442 cpu_manager.go:414] "containerMap: removing container" podUID="aac37dcc-c395-40e0-8d20-ec7733106b01" containerName="istio-proxy" Oct 30 19:00:34.539438 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539438 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aac37dcc-c395-40e0-8d20-ec7733106b01" containerName="istio-proxy" Oct 30 19:00:34.539438 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539451 4442 cpu_manager.go:414] "containerMap: removing container" podUID="1e7491f9-b178-456d-9a56-3eb6b5ee2e40" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539458 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e7491f9-b178-456d-9a56-3eb6b5ee2e40" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539470 4442 cpu_manager.go:414] "containerMap: removing container" podUID="ff29d9e8-03a0-4716-840d-a32e8a5aa628" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539477 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff29d9e8-03a0-4716-840d-a32e8a5aa628" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539486 4442 cpu_manager.go:414] "containerMap: removing container" podUID="00d6e25c-6e65-45de-a894-a6d1989bfd46" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539494 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00d6e25c-6e65-45de-a894-a6d1989bfd46" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539511 4442 cpu_manager.go:414] "containerMap: removing container" podUID="fb187f07-08dd-40b1-bb89-7f4993d8494f" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539518 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb187f07-08dd-40b1-bb89-7f4993d8494f" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539528 4442 cpu_manager.go:414] "containerMap: removing container" podUID="2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539535 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539549 4442 cpu_manager.go:414] "containerMap: removing container" podUID="88daa438-101a-4adc-8307-727c4176c705" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539556 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88daa438-101a-4adc-8307-727c4176c705" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539566 4442 cpu_manager.go:414] "containerMap: removing container" podUID="4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539574 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539587 4442 cpu_manager.go:414] "containerMap: removing container" podUID="ff29d9e8-03a0-4716-840d-a32e8a5aa628" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539594 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff29d9e8-03a0-4716-840d-a32e8a5aa628" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539605 4442 cpu_manager.go:414] "containerMap: removing container" podUID="4072bb06-edab-49c8-b213-a381823ec644" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539613 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4072bb06-edab-49c8-b213-a381823ec644" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539625 4442 cpu_manager.go:414] "containerMap: removing container" podUID="88daa438-101a-4adc-8307-727c4176c705" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539632 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88daa438-101a-4adc-8307-727c4176c705" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539647 4442 cpu_manager.go:414] "containerMap: removing container" podUID="322ab2aa-d4f3-42ef-9820-9438ced76d6b" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539654 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="322ab2aa-d4f3-42ef-9820-9438ced76d6b" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539667 4442 cpu_manager.go:414] "containerMap: removing container" podUID="4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539674 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539686 4442 cpu_manager.go:414] "containerMap: removing container" podUID="00d6e25c-6e65-45de-a894-a6d1989bfd46" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539693 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00d6e25c-6e65-45de-a894-a6d1989bfd46" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539722 4442 cpu_manager.go:414] "containerMap: removing container" podUID="9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539729 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539740 4442 cpu_manager.go:414] "containerMap: removing container" podUID="4072bb06-edab-49c8-b213-a381823ec644" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539747 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4072bb06-edab-49c8-b213-a381823ec644" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539778 4442 cpu_manager.go:414] "containerMap: removing container" podUID="fb187f07-08dd-40b1-bb89-7f4993d8494f" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539785 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb187f07-08dd-40b1-bb89-7f4993d8494f" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539800 4442 cpu_manager.go:414] "containerMap: removing container" podUID="fb187f07-08dd-40b1-bb89-7f4993d8494f" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539807 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb187f07-08dd-40b1-bb89-7f4993d8494f" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539816 4442 cpu_manager.go:414] "containerMap: removing container" podUID="2bb028e5-0be3-4923-9894-dc8f89889799" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539823 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bb028e5-0be3-4923-9894-dc8f89889799" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539833 4442 cpu_manager.go:414] "containerMap: removing container" podUID="a2ba9730-da34-4745-875f-cd941e32eb79" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539839 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2ba9730-da34-4745-875f-cd941e32eb79" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539850 4442 cpu_manager.go:414] "containerMap: removing container" podUID="525901b4-d3b5-4bd3-8458-b4f27ec0aef2" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539857 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="525901b4-d3b5-4bd3-8458-b4f27ec0aef2" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539868 4442 cpu_manager.go:414] "containerMap: removing container" podUID="5abb7ef5-cc99-469a-a165-9e9394370418" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539875 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5abb7ef5-cc99-469a-a165-9e9394370418" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539884 4442 cpu_manager.go:414] "containerMap: removing container" podUID="9fcfbcab-a54b-47d4-957b-76d2cb2444f1" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.539891 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fcfbcab-a54b-47d4-957b-76d2cb2444f1" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540079 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb187f07-08dd-40b1-bb89-7f4993d8494f" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540093 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540103 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="88daa438-101a-4adc-8307-727c4176c705" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540111 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540120 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff29d9e8-03a0-4716-840d-a32e8a5aa628" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540145 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="322ab2aa-d4f3-42ef-9820-9438ced76d6b" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540156 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="4072bb06-edab-49c8-b213-a381823ec644" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540167 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="88daa438-101a-4adc-8307-727c4176c705" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540176 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540186 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="00d6e25c-6e65-45de-a894-a6d1989bfd46" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540197 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e894ed2-7f33-42f4-aa19-9ba21268f296" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540207 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="39a6d924-7557-4f9b-b481-4398a402b1bd" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540215 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fcfbcab-a54b-47d4-957b-76d2cb2444f1" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540224 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540233 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="4072bb06-edab-49c8-b213-a381823ec644" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540245 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bb028e5-0be3-4923-9894-dc8f89889799" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540254 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="4072bb06-edab-49c8-b213-a381823ec644" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540262 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb187f07-08dd-40b1-bb89-7f4993d8494f" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540272 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2ba9730-da34-4745-875f-cd941e32eb79" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540280 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2ba9730-da34-4745-875f-cd941e32eb79" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540289 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e7491f9-b178-456d-9a56-3eb6b5ee2e40" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540297 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="5abb7ef5-cc99-469a-a165-9e9394370418" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540305 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="525901b4-d3b5-4bd3-8458-b4f27ec0aef2" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540314 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="525901b4-d3b5-4bd3-8458-b4f27ec0aef2" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540323 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="5abb7ef5-cc99-469a-a165-9e9394370418" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540331 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fcfbcab-a54b-47d4-957b-76d2cb2444f1" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540341 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="39a6d924-7557-4f9b-b481-4398a402b1bd" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540349 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540359 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bb028e5-0be3-4923-9894-dc8f89889799" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540368 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540377 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="322ab2aa-d4f3-42ef-9820-9438ced76d6b" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540386 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e7491f9-b178-456d-9a56-3eb6b5ee2e40" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540394 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ce11bc9-b3ab-4302-8338-4fa4d3db2afb" containerName="main" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540404 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7e7ff65-c137-48d2-a189-eb7f64bb1ff2" containerName="istio-proxy" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540412 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2ba9730-da34-4745-875f-cd941e32eb79" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540422 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="525901b4-d3b5-4bd3-8458-b4f27ec0aef2" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540430 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e894ed2-7f33-42f4-aa19-9ba21268f296" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540438 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="322ab2aa-d4f3-42ef-9820-9438ced76d6b" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540448 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="39a6d924-7557-4f9b-b481-4398a402b1bd" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540458 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e894ed2-7f33-42f4-aa19-9ba21268f296" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540466 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7e7ff65-c137-48d2-a189-eb7f64bb1ff2" containerName="istio-proxy" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540476 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="5abb7ef5-cc99-469a-a165-9e9394370418" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540488 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="00d6e25c-6e65-45de-a894-a6d1989bfd46" containerName="cleanup" Oct 30 19:00:34.540454 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540498 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="422f6f0d-871f-44d2-8fba-2dc65d684ced" containerName="revoke-keys" Oct 30 19:00:34.543259 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540507 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81" containerName="cleanup" Oct 30 19:00:34.543259 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540517 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="aac37dcc-c395-40e0-8d20-ec7733106b01" containerName="istio-proxy" Oct 30 19:00:34.543259 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.540526 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff29d9e8-03a0-4716-840d-a32e8a5aa628" containerName="cleanup" Oct 30 19:00:34.573424 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.573368 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-daf3ba/test-model-daf3ba-kserve-84cd9f6df-bfn9m"] Oct 30 19:00:34.573684 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.573471 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-daf3ba/test-model-daf3ba-kserve-84cd9f6df-bfn9m" Oct 30 19:00:34.599738 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.599689 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-daf3ba\"/\"openshift-service-ca.crt\"" Oct 30 19:00:34.599738 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.599696 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-daf3ba\"/\"test-model-daf3ba-kserve-self-signed-certs\"" Oct 30 19:00:34.599738 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.599710 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-daf3ba\"/\"default-dockercfg-jqns5\"" Oct 30 19:00:34.600070 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.600039 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-daf3ba\"/\"kube-root-ca.crt\"" Oct 30 19:00:34.614038 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.613986 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4wsm\" (UniqueName: \"kubernetes.io/projected/69d3fb60-519d-4101-a848-17c74270fca7-kube-api-access-n4wsm\") pod \"test-model-daf3ba-kserve-84cd9f6df-bfn9m\" (UID: \"69d3fb60-519d-4101-a848-17c74270fca7\") " pod="ai-tenant-e2e-inf-b-daf3ba/test-model-daf3ba-kserve-84cd9f6df-bfn9m" Oct 30 19:00:34.614215 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.614046 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69d3fb60-519d-4101-a848-17c74270fca7-dshm\") pod \"test-model-daf3ba-kserve-84cd9f6df-bfn9m\" (UID: \"69d3fb60-519d-4101-a848-17c74270fca7\") " pod="ai-tenant-e2e-inf-b-daf3ba/test-model-daf3ba-kserve-84cd9f6df-bfn9m" Oct 30 19:00:34.614215 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.614102 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69d3fb60-519d-4101-a848-17c74270fca7-tls-certs\") pod \"test-model-daf3ba-kserve-84cd9f6df-bfn9m\" (UID: \"69d3fb60-519d-4101-a848-17c74270fca7\") " pod="ai-tenant-e2e-inf-b-daf3ba/test-model-daf3ba-kserve-84cd9f6df-bfn9m" Oct 30 19:00:34.614382 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.614334 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69d3fb60-519d-4101-a848-17c74270fca7-model-cache\") pod \"test-model-daf3ba-kserve-84cd9f6df-bfn9m\" (UID: \"69d3fb60-519d-4101-a848-17c74270fca7\") " pod="ai-tenant-e2e-inf-b-daf3ba/test-model-daf3ba-kserve-84cd9f6df-bfn9m" Oct 30 19:00:34.614447 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.614385 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69d3fb60-519d-4101-a848-17c74270fca7-tmp-dir\") pod \"test-model-daf3ba-kserve-84cd9f6df-bfn9m\" (UID: \"69d3fb60-519d-4101-a848-17c74270fca7\") " pod="ai-tenant-e2e-inf-b-daf3ba/test-model-daf3ba-kserve-84cd9f6df-bfn9m" Oct 30 19:00:34.614548 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.614525 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69d3fb60-519d-4101-a848-17c74270fca7-home\") pod \"test-model-daf3ba-kserve-84cd9f6df-bfn9m\" (UID: \"69d3fb60-519d-4101-a848-17c74270fca7\") " pod="ai-tenant-e2e-inf-b-daf3ba/test-model-daf3ba-kserve-84cd9f6df-bfn9m" Oct 30 19:00:34.716453 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.716423 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69d3fb60-519d-4101-a848-17c74270fca7-tls-certs\") pod \"test-model-daf3ba-kserve-84cd9f6df-bfn9m\" (UID: \"69d3fb60-519d-4101-a848-17c74270fca7\") " pod="ai-tenant-e2e-inf-b-daf3ba/test-model-daf3ba-kserve-84cd9f6df-bfn9m" Oct 30 19:00:34.716673 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.716511 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69d3fb60-519d-4101-a848-17c74270fca7-model-cache\") pod \"test-model-daf3ba-kserve-84cd9f6df-bfn9m\" (UID: \"69d3fb60-519d-4101-a848-17c74270fca7\") " pod="ai-tenant-e2e-inf-b-daf3ba/test-model-daf3ba-kserve-84cd9f6df-bfn9m" Oct 30 19:00:34.716673 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.716550 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69d3fb60-519d-4101-a848-17c74270fca7-tmp-dir\") pod \"test-model-daf3ba-kserve-84cd9f6df-bfn9m\" (UID: \"69d3fb60-519d-4101-a848-17c74270fca7\") " pod="ai-tenant-e2e-inf-b-daf3ba/test-model-daf3ba-kserve-84cd9f6df-bfn9m" Oct 30 19:00:34.716673 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.716620 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69d3fb60-519d-4101-a848-17c74270fca7-home\") pod \"test-model-daf3ba-kserve-84cd9f6df-bfn9m\" (UID: \"69d3fb60-519d-4101-a848-17c74270fca7\") " pod="ai-tenant-e2e-inf-b-daf3ba/test-model-daf3ba-kserve-84cd9f6df-bfn9m" Oct 30 19:00:34.716846 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.716788 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4wsm\" (UniqueName: \"kubernetes.io/projected/69d3fb60-519d-4101-a848-17c74270fca7-kube-api-access-n4wsm\") pod \"test-model-daf3ba-kserve-84cd9f6df-bfn9m\" (UID: \"69d3fb60-519d-4101-a848-17c74270fca7\") " pod="ai-tenant-e2e-inf-b-daf3ba/test-model-daf3ba-kserve-84cd9f6df-bfn9m" Oct 30 19:00:34.716846 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.716831 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69d3fb60-519d-4101-a848-17c74270fca7-dshm\") pod \"test-model-daf3ba-kserve-84cd9f6df-bfn9m\" (UID: \"69d3fb60-519d-4101-a848-17c74270fca7\") " pod="ai-tenant-e2e-inf-b-daf3ba/test-model-daf3ba-kserve-84cd9f6df-bfn9m" Oct 30 19:00:34.717022 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.716997 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69d3fb60-519d-4101-a848-17c74270fca7-tmp-dir\") pod \"test-model-daf3ba-kserve-84cd9f6df-bfn9m\" (UID: \"69d3fb60-519d-4101-a848-17c74270fca7\") " pod="ai-tenant-e2e-inf-b-daf3ba/test-model-daf3ba-kserve-84cd9f6df-bfn9m" Oct 30 19:00:34.717063 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.716998 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69d3fb60-519d-4101-a848-17c74270fca7-model-cache\") pod \"test-model-daf3ba-kserve-84cd9f6df-bfn9m\" (UID: \"69d3fb60-519d-4101-a848-17c74270fca7\") " pod="ai-tenant-e2e-inf-b-daf3ba/test-model-daf3ba-kserve-84cd9f6df-bfn9m" Oct 30 19:00:34.717063 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.717052 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69d3fb60-519d-4101-a848-17c74270fca7-home\") pod \"test-model-daf3ba-kserve-84cd9f6df-bfn9m\" (UID: \"69d3fb60-519d-4101-a848-17c74270fca7\") " pod="ai-tenant-e2e-inf-b-daf3ba/test-model-daf3ba-kserve-84cd9f6df-bfn9m" Oct 30 19:00:34.719040 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.719015 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69d3fb60-519d-4101-a848-17c74270fca7-dshm\") pod \"test-model-daf3ba-kserve-84cd9f6df-bfn9m\" (UID: \"69d3fb60-519d-4101-a848-17c74270fca7\") " pod="ai-tenant-e2e-inf-b-daf3ba/test-model-daf3ba-kserve-84cd9f6df-bfn9m" Oct 30 19:00:34.719200 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.719185 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69d3fb60-519d-4101-a848-17c74270fca7-tls-certs\") pod \"test-model-daf3ba-kserve-84cd9f6df-bfn9m\" (UID: \"69d3fb60-519d-4101-a848-17c74270fca7\") " pod="ai-tenant-e2e-inf-b-daf3ba/test-model-daf3ba-kserve-84cd9f6df-bfn9m" Oct 30 19:00:34.736249 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.736218 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4wsm\" (UniqueName: \"kubernetes.io/projected/69d3fb60-519d-4101-a848-17c74270fca7-kube-api-access-n4wsm\") pod \"test-model-daf3ba-kserve-84cd9f6df-bfn9m\" (UID: \"69d3fb60-519d-4101-a848-17c74270fca7\") " pod="ai-tenant-e2e-inf-b-daf3ba/test-model-daf3ba-kserve-84cd9f6df-bfn9m" Oct 30 19:00:34.903764 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:34.903367 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-daf3ba/test-model-daf3ba-kserve-84cd9f6df-bfn9m" Oct 30 19:00:35.284240 ip-10-0-138-176 kubenswrapper[4442]: W1030 19:00:35.284208 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69d3fb60_519d_4101_a848_17c74270fca7.slice/crio-c3c34df61c69513bbd6e18221dc1cd2a98dee56232f0583de9079b36589b0e0d WatchSource:0}: Error finding container c3c34df61c69513bbd6e18221dc1cd2a98dee56232f0583de9079b36589b0e0d: Status 404 returned error can't find the container with id c3c34df61c69513bbd6e18221dc1cd2a98dee56232f0583de9079b36589b0e0d Oct 30 19:00:35.284599 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:35.284573 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-daf3ba/test-model-daf3ba-kserve-84cd9f6df-bfn9m"] Oct 30 19:00:35.551052 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:35.550943 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-daf3ba/test-model-daf3ba-kserve-84cd9f6df-bfn9m" event={"ID":"69d3fb60-519d-4101-a848-17c74270fca7","Type":"ContainerStarted","Data":"331ea596a4f2dd27567ddc312b77651908996c90d8ce9738e6227576409b806d"} Oct 30 19:00:35.551052 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:35.551012 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-daf3ba/test-model-daf3ba-kserve-84cd9f6df-bfn9m" event={"ID":"69d3fb60-519d-4101-a848-17c74270fca7","Type":"ContainerStarted","Data":"c3c34df61c69513bbd6e18221dc1cd2a98dee56232f0583de9079b36589b0e0d"} Oct 30 19:00:35.590256 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:35.590197 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-daf3ba/test-model-daf3ba-kserve-84cd9f6df-bfn9m" podStartSLOduration=1.590179961 podStartE2EDuration="1.590179961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:35.588535019 +0000 UTC m=+3732.678597358" watchObservedRunningTime="2026-10-30 19:00:35.590179961 +0000 UTC m=+3732.680242302" Oct 30 19:00:35.904595 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:35.904561 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-daf3ba/test-model-daf3ba-kserve-84cd9f6df-bfn9m" Oct 30 19:00:37.702047 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:37.702014 4442 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5abb7ef5_cc99_469a_a165_9e9394370418.slice\": RecentStats: unable to find data in memory cache]" Oct 30 19:00:39.005421 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.005387 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baca322b-pnr26"] Oct 30 19:00:39.006209 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.006193 4442 cpu_manager.go:414] "containerMap: removing container" podUID="39a6d924-7557-4f9b-b481-4398a402b1bd" containerName="cleanup" Oct 30 19:00:39.006269 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.006210 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39a6d924-7557-4f9b-b481-4398a402b1bd" containerName="cleanup" Oct 30 19:00:39.006269 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.006236 4442 cpu_manager.go:414] "containerMap: removing container" podUID="1e7491f9-b178-456d-9a56-3eb6b5ee2e40" containerName="cleanup" Oct 30 19:00:39.006269 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.006241 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e7491f9-b178-456d-9a56-3eb6b5ee2e40" containerName="cleanup" Oct 30 19:00:39.006269 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.006252 4442 cpu_manager.go:414] "containerMap: removing container" podUID="2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81" containerName="cleanup" Oct 30 19:00:39.006269 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.006256 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81" containerName="cleanup" Oct 30 19:00:39.006269 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.006265 4442 cpu_manager.go:414] "containerMap: removing container" podUID="ff29d9e8-03a0-4716-840d-a32e8a5aa628" containerName="cleanup" Oct 30 19:00:39.006269 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.006269 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff29d9e8-03a0-4716-840d-a32e8a5aa628" containerName="cleanup" Oct 30 19:00:39.006503 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.006279 4442 cpu_manager.go:414] "containerMap: removing container" podUID="322ab2aa-d4f3-42ef-9820-9438ced76d6b" containerName="cleanup" Oct 30 19:00:39.006503 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.006284 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="322ab2aa-d4f3-42ef-9820-9438ced76d6b" containerName="cleanup" Oct 30 19:00:39.006503 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.006300 4442 cpu_manager.go:414] "containerMap: removing container" podUID="4072bb06-edab-49c8-b213-a381823ec644" containerName="cleanup" Oct 30 19:00:39.006503 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.006304 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4072bb06-edab-49c8-b213-a381823ec644" containerName="cleanup" Oct 30 19:00:39.006503 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.006324 4442 cpu_manager.go:414] "containerMap: removing container" podUID="5abb7ef5-cc99-469a-a165-9e9394370418" containerName="cleanup" Oct 30 19:00:39.006503 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.006329 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5abb7ef5-cc99-469a-a165-9e9394370418" containerName="cleanup" Oct 30 19:00:39.006503 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.006336 4442 cpu_manager.go:414] "containerMap: removing container" podUID="9fcfbcab-a54b-47d4-957b-76d2cb2444f1" containerName="cleanup" Oct 30 19:00:39.006503 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.006340 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fcfbcab-a54b-47d4-957b-76d2cb2444f1" containerName="cleanup" Oct 30 19:00:39.006503 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.006346 4442 cpu_manager.go:414] "containerMap: removing container" podUID="9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1" containerName="cleanup" Oct 30 19:00:39.006503 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.006350 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b5613a0-fe8a-4ade-9f7d-9ce7485b85d1" containerName="cleanup" Oct 30 19:00:39.006503 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.006356 4442 cpu_manager.go:414] "containerMap: removing container" podUID="2bb028e5-0be3-4923-9894-dc8f89889799" containerName="cleanup" Oct 30 19:00:39.006503 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.006360 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bb028e5-0be3-4923-9894-dc8f89889799" containerName="cleanup" Oct 30 19:00:39.006503 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.006370 4442 cpu_manager.go:414] "containerMap: removing container" podUID="a2ba9730-da34-4745-875f-cd941e32eb79" containerName="cleanup" Oct 30 19:00:39.006503 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.006374 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2ba9730-da34-4745-875f-cd941e32eb79" containerName="cleanup" Oct 30 19:00:39.006503 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.006381 4442 cpu_manager.go:414] "containerMap: removing container" podUID="525901b4-d3b5-4bd3-8458-b4f27ec0aef2" containerName="cleanup" Oct 30 19:00:39.006503 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.006385 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="525901b4-d3b5-4bd3-8458-b4f27ec0aef2" containerName="cleanup" Oct 30 19:00:39.006503 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.006395 4442 cpu_manager.go:414] "containerMap: removing container" podUID="6e894ed2-7f33-42f4-aa19-9ba21268f296" containerName="cleanup" Oct 30 19:00:39.006503 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.006399 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e894ed2-7f33-42f4-aa19-9ba21268f296" containerName="cleanup" Oct 30 19:00:39.006503 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.006503 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fcfbcab-a54b-47d4-957b-76d2cb2444f1" containerName="cleanup" Oct 30 19:00:39.006503 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.006510 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bb028e5-0be3-4923-9894-dc8f89889799" containerName="cleanup" Oct 30 19:00:39.007074 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.006516 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d2ffcd7-66c7-448c-88ce-c85d1ff2a7f2" containerName="cleanup" Oct 30 19:00:39.007074 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.006523 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e7491f9-b178-456d-9a56-3eb6b5ee2e40" containerName="cleanup" Oct 30 19:00:39.007074 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.006536 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb187f07-08dd-40b1-bb89-7f4993d8494f" containerName="cleanup" Oct 30 19:00:39.007074 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.006551 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a9546ae-4ab1-48f2-bb7c-9e5e06c69b81" containerName="cleanup" Oct 30 19:00:39.007074 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.006557 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff29d9e8-03a0-4716-840d-a32e8a5aa628" containerName="cleanup" Oct 30 19:00:39.007074 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.006563 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="88daa438-101a-4adc-8307-727c4176c705" containerName="cleanup" Oct 30 19:00:39.007074 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.006569 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="00d6e25c-6e65-45de-a894-a6d1989bfd46" containerName="cleanup" Oct 30 19:00:39.010550 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.010530 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baca322b-pnr26" Oct 30 19:00:39.039555 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.039522 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baca322b-pnr26"] Oct 30 19:00:39.045494 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.045448 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:00:39.051165 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.049314 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:00:39.175303 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.175264 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgncc\" (UniqueName: \"kubernetes.io/projected/14e77784-722e-4d16-86f7-9fc801b52b4b-kube-api-access-bgncc\") pod \"maas-api-revoke-sub-baca322b-pnr26\" (UID: \"14e77784-722e-4d16-86f7-9fc801b52b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baca322b-pnr26" Oct 30 19:00:39.175566 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.175359 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14e77784-722e-4d16-86f7-9fc801b52b4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baca322b-pnr26\" (UID: \"14e77784-722e-4d16-86f7-9fc801b52b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baca322b-pnr26" Oct 30 19:00:39.175678 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.175651 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14e77784-722e-4d16-86f7-9fc801b52b4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baca322b-pnr26\" (UID: \"14e77784-722e-4d16-86f7-9fc801b52b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baca322b-pnr26" Oct 30 19:00:39.276764 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.276677 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgncc\" (UniqueName: \"kubernetes.io/projected/14e77784-722e-4d16-86f7-9fc801b52b4b-kube-api-access-bgncc\") pod \"maas-api-revoke-sub-baca322b-pnr26\" (UID: \"14e77784-722e-4d16-86f7-9fc801b52b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baca322b-pnr26" Oct 30 19:00:39.276764 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.276759 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14e77784-722e-4d16-86f7-9fc801b52b4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baca322b-pnr26\" (UID: \"14e77784-722e-4d16-86f7-9fc801b52b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baca322b-pnr26" Oct 30 19:00:39.277033 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.277014 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14e77784-722e-4d16-86f7-9fc801b52b4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baca322b-pnr26\" (UID: \"14e77784-722e-4d16-86f7-9fc801b52b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baca322b-pnr26" Oct 30 19:00:39.279122 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.277550 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14e77784-722e-4d16-86f7-9fc801b52b4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baca322b-pnr26\" (UID: \"14e77784-722e-4d16-86f7-9fc801b52b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baca322b-pnr26" Oct 30 19:00:39.280059 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.280018 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14e77784-722e-4d16-86f7-9fc801b52b4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baca322b-pnr26\" (UID: \"14e77784-722e-4d16-86f7-9fc801b52b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baca322b-pnr26" Oct 30 19:00:39.297018 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.296993 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgncc\" (UniqueName: \"kubernetes.io/projected/14e77784-722e-4d16-86f7-9fc801b52b4b-kube-api-access-bgncc\") pod \"maas-api-revoke-sub-baca322b-pnr26\" (UID: \"14e77784-722e-4d16-86f7-9fc801b52b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baca322b-pnr26" Oct 30 19:00:39.328384 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.328349 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baca322b-pnr26" Oct 30 19:00:39.642881 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.642850 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baca322b-pnr26"] Oct 30 19:00:39.668347 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:39.668306 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baca322b-pnr26" event={"ID":"14e77784-722e-4d16-86f7-9fc801b52b4b","Type":"ContainerStarted","Data":"167c9ccaebb2b40f3c4a4cde5b8707652d8748dc9d38d8f1a7788fe7489a9f3c"} Oct 30 19:00:40.682307 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:40.682279 4442 generic.go:356] "Generic (PLEG): container finished" podID="14e77784-722e-4d16-86f7-9fc801b52b4b" containerID="cfd677f5f650f0e85f1feda81a61c5eaedab811ea6803a782fcd5876ae0299c7" exitCode=0 Oct 30 19:00:40.682689 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:40.682344 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baca322b-pnr26" event={"ID":"14e77784-722e-4d16-86f7-9fc801b52b4b","Type":"ContainerDied","Data":"cfd677f5f650f0e85f1feda81a61c5eaedab811ea6803a782fcd5876ae0299c7"} Oct 30 19:00:41.643483 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:41.643453 4442 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5abb7ef5_cc99_469a_a165_9e9394370418.slice\": RecentStats: unable to find data in memory cache]" Oct 30 19:00:41.681266 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:41.681230 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1aa59a-765b7b94b7-s9pmv" podUID="8b79c781-9ddc-473d-8285-120dffb8e016" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:00:41.902866 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:41.902810 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baca322b-pnr26" Oct 30 19:00:42.027741 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:42.027706 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14e77784-722e-4d16-86f7-9fc801b52b4b-kube-api-access-bgncc\" (UniqueName: \"kubernetes.io/projected/14e77784-722e-4d16-86f7-9fc801b52b4b-kube-api-access-bgncc\") pod \"14e77784-722e-4d16-86f7-9fc801b52b4b\" (UID: \"14e77784-722e-4d16-86f7-9fc801b52b4b\") " Oct 30 19:00:42.028093 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:42.028074 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14e77784-722e-4d16-86f7-9fc801b52b4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14e77784-722e-4d16-86f7-9fc801b52b4b-maas-api-openshift-ca\") pod \"14e77784-722e-4d16-86f7-9fc801b52b4b\" (UID: \"14e77784-722e-4d16-86f7-9fc801b52b4b\") " Oct 30 19:00:42.028267 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:42.028249 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14e77784-722e-4d16-86f7-9fc801b52b4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14e77784-722e-4d16-86f7-9fc801b52b4b-maas-api-xks-ca\") pod \"14e77784-722e-4d16-86f7-9fc801b52b4b\" (UID: \"14e77784-722e-4d16-86f7-9fc801b52b4b\") " Oct 30 19:00:42.028672 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:42.028642 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14e77784-722e-4d16-86f7-9fc801b52b4b-maas-api-openshift-ca" pod "14e77784-722e-4d16-86f7-9fc801b52b4b" (UID: "14e77784-722e-4d16-86f7-9fc801b52b4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:42.029072 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:42.029046 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14e77784-722e-4d16-86f7-9fc801b52b4b-maas-api-openshift-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:42.030213 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:42.030179 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14e77784-722e-4d16-86f7-9fc801b52b4b-kube-api-access-bgncc" pod "14e77784-722e-4d16-86f7-9fc801b52b4b" (UID: "14e77784-722e-4d16-86f7-9fc801b52b4b"). InnerVolumeSpecName "kube-api-access-bgncc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:42.030761 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:42.030739 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14e77784-722e-4d16-86f7-9fc801b52b4b-maas-api-xks-ca" pod "14e77784-722e-4d16-86f7-9fc801b52b4b" (UID: "14e77784-722e-4d16-86f7-9fc801b52b4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:00:42.131435 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:42.131392 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14e77784-722e-4d16-86f7-9fc801b52b4b-maas-api-xks-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:42.131435 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:42.131427 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgncc\" (UniqueName: \"kubernetes.io/projected/14e77784-722e-4d16-86f7-9fc801b52b4b-kube-api-access-bgncc\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:42.701287 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:42.701253 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baca322b-pnr26" event={"ID":"14e77784-722e-4d16-86f7-9fc801b52b4b","Type":"ContainerDied","Data":"167c9ccaebb2b40f3c4a4cde5b8707652d8748dc9d38d8f1a7788fe7489a9f3c"} Oct 30 19:00:42.701287 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:42.701274 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baca322b-pnr26" Oct 30 19:00:42.701512 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:42.701299 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="167c9ccaebb2b40f3c4a4cde5b8707652d8748dc9d38d8f1a7788fe7489a9f3c" Oct 30 19:00:44.904902 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:44.904866 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-daf3ba/test-model-daf3ba-kserve-84cd9f6df-bfn9m" Oct 30 19:00:44.919026 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:44.918995 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-daf3ba/test-model-daf3ba-kserve-84cd9f6df-bfn9m" Oct 30 19:00:45.740152 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:45.740099 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-daf3ba/test-model-daf3ba-kserve-84cd9f6df-bfn9m" Oct 30 19:00:47.562221 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:47.562178 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 19:00:48.641408 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:48.641376 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1aa59a-765b7b94b7-s9pmv" Oct 30 19:00:48.757318 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:48.757217 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b79c781-9ddc-473d-8285-120dffb8e016-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b79c781-9ddc-473d-8285-120dffb8e016-plugins-config-volume\") pod \"8b79c781-9ddc-473d-8285-120dffb8e016\" (UID: \"8b79c781-9ddc-473d-8285-120dffb8e016\") " Oct 30 19:00:48.757683 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:48.757645 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b79c781-9ddc-473d-8285-120dffb8e016-kube-api-access-45hqs\" (UniqueName: \"kubernetes.io/projected/8b79c781-9ddc-473d-8285-120dffb8e016-kube-api-access-45hqs\") pod \"8b79c781-9ddc-473d-8285-120dffb8e016\" (UID: \"8b79c781-9ddc-473d-8285-120dffb8e016\") " Oct 30 19:00:48.757683 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:48.757661 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b79c781-9ddc-473d-8285-120dffb8e016-plugins-config-volume" pod "8b79c781-9ddc-473d-8285-120dffb8e016" (UID: "8b79c781-9ddc-473d-8285-120dffb8e016"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:48.758205 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:48.758189 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b79c781-9ddc-473d-8285-120dffb8e016-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:48.760120 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:48.760093 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b79c781-9ddc-473d-8285-120dffb8e016-kube-api-access-45hqs" pod "8b79c781-9ddc-473d-8285-120dffb8e016" (UID: "8b79c781-9ddc-473d-8285-120dffb8e016"). InnerVolumeSpecName "kube-api-access-45hqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:48.764113 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:48.764090 4442 generic.go:356] "Generic (PLEG): container finished" podID="8b79c781-9ddc-473d-8285-120dffb8e016" containerID="9195959c568f767238f3d4791ee1266e18d693a2225e13cfc308e4e4a0e84c07" exitCode=0 Oct 30 19:00:48.764243 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:48.764160 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1aa59a-765b7b94b7-s9pmv" Oct 30 19:00:48.764243 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:48.764193 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1aa59a-765b7b94b7-s9pmv" event={"ID":"8b79c781-9ddc-473d-8285-120dffb8e016","Type":"ContainerDied","Data":"9195959c568f767238f3d4791ee1266e18d693a2225e13cfc308e4e4a0e84c07"} Oct 30 19:00:48.764319 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:48.764257 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1aa59a-765b7b94b7-s9pmv" event={"ID":"8b79c781-9ddc-473d-8285-120dffb8e016","Type":"ContainerDied","Data":"1589357258b607f0d59a32572cc2b99f712981f29ea4d7a948d4edee3611025c"} Oct 30 19:00:48.764319 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:48.764289 4442 scope.go:122] "RemoveContainer" containerID="9195959c568f767238f3d4791ee1266e18d693a2225e13cfc308e4e4a0e84c07" Oct 30 19:00:48.785481 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:48.785456 4442 scope.go:122] "RemoveContainer" containerID="9195959c568f767238f3d4791ee1266e18d693a2225e13cfc308e4e4a0e84c07" Oct 30 19:00:48.785874 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:48.785856 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9195959c568f767238f3d4791ee1266e18d693a2225e13cfc308e4e4a0e84c07\": container with ID starting with 9195959c568f767238f3d4791ee1266e18d693a2225e13cfc308e4e4a0e84c07 not found: ID does not exist" containerID="9195959c568f767238f3d4791ee1266e18d693a2225e13cfc308e4e4a0e84c07" Oct 30 19:00:48.785923 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:48.785886 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9195959c568f767238f3d4791ee1266e18d693a2225e13cfc308e4e4a0e84c07"} err="failed to get container status \"9195959c568f767238f3d4791ee1266e18d693a2225e13cfc308e4e4a0e84c07\": rpc error: code = NotFound desc = could not find container \"9195959c568f767238f3d4791ee1266e18d693a2225e13cfc308e4e4a0e84c07\": container with ID starting with 9195959c568f767238f3d4791ee1266e18d693a2225e13cfc308e4e4a0e84c07 not found: ID does not exist" Oct 30 19:00:48.823741 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:48.823710 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1aa59a-765b7b94b7-s9pmv"] Oct 30 19:00:48.830640 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:48.830605 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1aa59a-765b7b94b7-s9pmv"] Oct 30 19:00:48.860531 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:48.860496 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45hqs\" (UniqueName: \"kubernetes.io/projected/8b79c781-9ddc-473d-8285-120dffb8e016-kube-api-access-45hqs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:00:49.069401 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:49.069317 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b79c781-9ddc-473d-8285-120dffb8e016" path="/var/lib/kubelet/pods/8b79c781-9ddc-473d-8285-120dffb8e016/volumes" Oct 30 19:00:51.733210 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:51.733127 4442 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5abb7ef5_cc99_469a_a165_9e9394370418.slice\": RecentStats: unable to find data in memory cache]" Oct 30 19:00:52.603709 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:52.603671 4442 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5abb7ef5_cc99_469a_a165_9e9394370418.slice\": RecentStats: unable to find data in memory cache]" Oct 30 19:00:53.122308 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:53.122253 4442 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","pod5abb7ef5-cc99-469a-a165-9e9394370418"] err="unable to destroy cgroup paths for cgroup [kubepods burstable pod5abb7ef5-cc99-469a-a165-9e9394370418] : Timed out while waiting for systemd to remove kubepods-burstable-pod5abb7ef5_cc99_469a_a165_9e9394370418.slice" Oct 30 19:00:53.122308 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:00:53.122311 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to delete cgroup paths for [kubepods burstable pod5abb7ef5-cc99-469a-a165-9e9394370418] : unable to destroy cgroup paths for cgroup [kubepods burstable pod5abb7ef5-cc99-469a-a165-9e9394370418] : Timed out while waiting for systemd to remove kubepods-burstable-pod5abb7ef5_cc99_469a_a165_9e9394370418.slice" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889780-xztsk" podUID="5abb7ef5-cc99-469a-a165-9e9394370418" Oct 30 19:00:53.826367 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:53.826335 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889780-xztsk" Oct 30 19:00:53.921911 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:53.921875 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889780-xztsk"] Oct 30 19:00:53.926838 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:53.926809 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889780-xztsk"] Oct 30 19:00:55.072178 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:00:55.072113 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5abb7ef5-cc99-469a-a165-9e9394370418" path="/var/lib/kubelet/pods/5abb7ef5-cc99-469a-a165-9e9394370418/volumes" Oct 30 19:01:01.192561 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:01.191198 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8862aa-zccpp"] Oct 30 19:01:01.192561 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:01.192419 4442 cpu_manager.go:414] "containerMap: removing container" podUID="8b79c781-9ddc-473d-8285-120dffb8e016" containerName="payload-pre-processing" Oct 30 19:01:01.192561 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:01.192439 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b79c781-9ddc-473d-8285-120dffb8e016" containerName="payload-pre-processing" Oct 30 19:01:01.192561 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:01.192458 4442 cpu_manager.go:414] "containerMap: removing container" podUID="14e77784-722e-4d16-86f7-9fc801b52b4b" containerName="revoke-keys" Oct 30 19:01:01.192561 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:01.192465 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14e77784-722e-4d16-86f7-9fc801b52b4b" containerName="revoke-keys" Oct 30 19:01:01.193275 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:01.192693 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b79c781-9ddc-473d-8285-120dffb8e016" containerName="payload-pre-processing" Oct 30 19:01:01.193275 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:01.192709 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="14e77784-722e-4d16-86f7-9fc801b52b4b" containerName="revoke-keys" Oct 30 19:01:01.196548 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:01.196459 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8862aa-zccpp" Oct 30 19:01:01.217645 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:01.217617 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8862aa-zccpp"] Oct 30 19:01:01.218302 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:01.218278 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpr7f\" (UniqueName: \"kubernetes.io/projected/e475ae0a-ccb4-41e8-844c-6c566ad3754a-kube-api-access-qpr7f\") pod \"maas-api-revoke-keys-e2e-inf-a-8862aa-zccpp\" (UID: \"e475ae0a-ccb4-41e8-844c-6c566ad3754a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8862aa-zccpp" Oct 30 19:01:01.218441 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:01.218388 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e475ae0a-ccb4-41e8-844c-6c566ad3754a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8862aa-zccpp\" (UID: \"e475ae0a-ccb4-41e8-844c-6c566ad3754a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8862aa-zccpp" Oct 30 19:01:01.218547 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:01.218532 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e475ae0a-ccb4-41e8-844c-6c566ad3754a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8862aa-zccpp\" (UID: \"e475ae0a-ccb4-41e8-844c-6c566ad3754a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8862aa-zccpp" Oct 30 19:01:01.248426 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:01.248388 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:01:01.248642 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:01.248443 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:01:01.319811 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:01.319777 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e475ae0a-ccb4-41e8-844c-6c566ad3754a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8862aa-zccpp\" (UID: \"e475ae0a-ccb4-41e8-844c-6c566ad3754a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8862aa-zccpp" Oct 30 19:01:01.320007 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:01.319918 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpr7f\" (UniqueName: \"kubernetes.io/projected/e475ae0a-ccb4-41e8-844c-6c566ad3754a-kube-api-access-qpr7f\") pod \"maas-api-revoke-keys-e2e-inf-a-8862aa-zccpp\" (UID: \"e475ae0a-ccb4-41e8-844c-6c566ad3754a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8862aa-zccpp" Oct 30 19:01:01.320007 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:01.319983 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e475ae0a-ccb4-41e8-844c-6c566ad3754a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8862aa-zccpp\" (UID: \"e475ae0a-ccb4-41e8-844c-6c566ad3754a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8862aa-zccpp" Oct 30 19:01:01.320785 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:01.320665 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e475ae0a-ccb4-41e8-844c-6c566ad3754a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8862aa-zccpp\" (UID: \"e475ae0a-ccb4-41e8-844c-6c566ad3754a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8862aa-zccpp" Oct 30 19:01:01.322561 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:01.322531 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e475ae0a-ccb4-41e8-844c-6c566ad3754a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8862aa-zccpp\" (UID: \"e475ae0a-ccb4-41e8-844c-6c566ad3754a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8862aa-zccpp" Oct 30 19:01:01.340721 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:01.340682 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpr7f\" (UniqueName: \"kubernetes.io/projected/e475ae0a-ccb4-41e8-844c-6c566ad3754a-kube-api-access-qpr7f\") pod \"maas-api-revoke-keys-e2e-inf-a-8862aa-zccpp\" (UID: \"e475ae0a-ccb4-41e8-844c-6c566ad3754a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8862aa-zccpp" Oct 30 19:01:01.514411 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:01.514328 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8862aa-zccpp" Oct 30 19:01:01.835909 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:01:01.835330 4442 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5abb7ef5_cc99_469a_a165_9e9394370418.slice\": RecentStats: unable to find data in memory cache]" Oct 30 19:01:01.843073 ip-10-0-138-176 kubenswrapper[4442]: W1030 19:01:01.843040 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode475ae0a_ccb4_41e8_844c_6c566ad3754a.slice/crio-7f7da9156cda9ce7f53a38805c787b1cde560493bf1fcfe395db365b2e72349b WatchSource:0}: Error finding container 7f7da9156cda9ce7f53a38805c787b1cde560493bf1fcfe395db365b2e72349b: Status 404 returned error can't find the container with id 7f7da9156cda9ce7f53a38805c787b1cde560493bf1fcfe395db365b2e72349b Oct 30 19:01:01.843380 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:01.843351 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8862aa-zccpp"] Oct 30 19:01:01.927128 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:01.927085 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8862aa-zccpp" event={"ID":"e475ae0a-ccb4-41e8-844c-6c566ad3754a","Type":"ContainerStarted","Data":"7f7da9156cda9ce7f53a38805c787b1cde560493bf1fcfe395db365b2e72349b"} Oct 30 19:01:02.941459 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:02.941418 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8862aa-zccpp" event={"ID":"e475ae0a-ccb4-41e8-844c-6c566ad3754a","Type":"ContainerStarted","Data":"1496ab5158b4a1badc27f63ce11e4f85d8e105be0d66ccc1619c5120585cb232"} Oct 30 19:01:02.990219 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:02.990171 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8862aa-zccpp" podStartSLOduration=1.9901583029999999 podStartE2EDuration="1.990158303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:02.989628518 +0000 UTC m=+3760.079690850" watchObservedRunningTime="2026-10-30 19:01:02.990158303 +0000 UTC m=+3760.080220641" Oct 30 19:01:03.956570 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:03.956526 4442 generic.go:356] "Generic (PLEG): container finished" podID="e475ae0a-ccb4-41e8-844c-6c566ad3754a" containerID="1496ab5158b4a1badc27f63ce11e4f85d8e105be0d66ccc1619c5120585cb232" exitCode=0 Oct 30 19:01:03.957064 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:03.956677 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8862aa-zccpp" event={"ID":"e475ae0a-ccb4-41e8-844c-6c566ad3754a","Type":"ContainerDied","Data":"1496ab5158b4a1badc27f63ce11e4f85d8e105be0d66ccc1619c5120585cb232"} Oct 30 19:01:05.476082 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:05.476046 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8862aa-zccpp" Oct 30 19:01:05.583619 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:05.583583 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e475ae0a-ccb4-41e8-844c-6c566ad3754a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e475ae0a-ccb4-41e8-844c-6c566ad3754a-maas-api-openshift-ca\") pod \"e475ae0a-ccb4-41e8-844c-6c566ad3754a\" (UID: \"e475ae0a-ccb4-41e8-844c-6c566ad3754a\") " Oct 30 19:01:05.583619 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:05.583623 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e475ae0a-ccb4-41e8-844c-6c566ad3754a-kube-api-access-qpr7f\" (UniqueName: \"kubernetes.io/projected/e475ae0a-ccb4-41e8-844c-6c566ad3754a-kube-api-access-qpr7f\") pod \"e475ae0a-ccb4-41e8-844c-6c566ad3754a\" (UID: \"e475ae0a-ccb4-41e8-844c-6c566ad3754a\") " Oct 30 19:01:05.583923 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:05.583858 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e475ae0a-ccb4-41e8-844c-6c566ad3754a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e475ae0a-ccb4-41e8-844c-6c566ad3754a-maas-api-xks-ca\") pod \"e475ae0a-ccb4-41e8-844c-6c566ad3754a\" (UID: \"e475ae0a-ccb4-41e8-844c-6c566ad3754a\") " Oct 30 19:01:05.584109 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:05.584078 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e475ae0a-ccb4-41e8-844c-6c566ad3754a-maas-api-openshift-ca" pod "e475ae0a-ccb4-41e8-844c-6c566ad3754a" (UID: "e475ae0a-ccb4-41e8-844c-6c566ad3754a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:01:05.584601 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:05.584585 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e475ae0a-ccb4-41e8-844c-6c566ad3754a-maas-api-openshift-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:01:05.586684 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:05.586660 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e475ae0a-ccb4-41e8-844c-6c566ad3754a-kube-api-access-qpr7f" pod "e475ae0a-ccb4-41e8-844c-6c566ad3754a" (UID: "e475ae0a-ccb4-41e8-844c-6c566ad3754a"). InnerVolumeSpecName "kube-api-access-qpr7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:01:05.587248 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:05.587226 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e475ae0a-ccb4-41e8-844c-6c566ad3754a-maas-api-xks-ca" pod "e475ae0a-ccb4-41e8-844c-6c566ad3754a" (UID: "e475ae0a-ccb4-41e8-844c-6c566ad3754a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:01:05.686699 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:05.686626 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpr7f\" (UniqueName: \"kubernetes.io/projected/e475ae0a-ccb4-41e8-844c-6c566ad3754a-kube-api-access-qpr7f\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:01:05.686699 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:05.686652 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e475ae0a-ccb4-41e8-844c-6c566ad3754a-maas-api-xks-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:01:05.979471 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:05.979386 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8862aa-zccpp" Oct 30 19:01:05.979471 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:05.979409 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8862aa-zccpp" event={"ID":"e475ae0a-ccb4-41e8-844c-6c566ad3754a","Type":"ContainerDied","Data":"7f7da9156cda9ce7f53a38805c787b1cde560493bf1fcfe395db365b2e72349b"} Oct 30 19:01:05.979471 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:05.979467 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f7da9156cda9ce7f53a38805c787b1cde560493bf1fcfe395db365b2e72349b" Oct 30 19:01:07.696862 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:01:07.696829 4442 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5abb7ef5_cc99_469a_a165_9e9394370418.slice\": RecentStats: unable to find data in memory cache]" Oct 30 19:01:11.980159 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:01:11.954923 4442 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5abb7ef5_cc99_469a_a165_9e9394370418.slice\": RecentStats: unable to find data in memory cache]" Oct 30 19:01:17.561636 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:01:17.561595 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 19:01:18.216864 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:18.216834 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-551d0b-8c8jd"] Oct 30 19:01:18.217678 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:18.217658 4442 cpu_manager.go:414] "containerMap: removing container" podUID="e475ae0a-ccb4-41e8-844c-6c566ad3754a" containerName="revoke-keys" Oct 30 19:01:18.217678 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:18.217676 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e475ae0a-ccb4-41e8-844c-6c566ad3754a" containerName="revoke-keys" Oct 30 19:01:18.217870 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:18.217818 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="e475ae0a-ccb4-41e8-844c-6c566ad3754a" containerName="revoke-keys" Oct 30 19:01:18.222912 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:18.222894 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-551d0b-8c8jd" Oct 30 19:01:18.280381 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:18.280348 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:01:18.280381 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:18.280379 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:01:18.294878 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:18.294849 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-551d0b-8c8jd"] Oct 30 19:01:18.317579 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:18.317545 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2014b449-aa26-47e8-bb93-893588bc6083-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-551d0b-8c8jd\" (UID: \"2014b449-aa26-47e8-bb93-893588bc6083\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-551d0b-8c8jd" Oct 30 19:01:18.317733 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:18.317605 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4gkg\" (UniqueName: \"kubernetes.io/projected/2014b449-aa26-47e8-bb93-893588bc6083-kube-api-access-r4gkg\") pod \"maas-api-revoke-keys-e2e-inf-b-551d0b-8c8jd\" (UID: \"2014b449-aa26-47e8-bb93-893588bc6083\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-551d0b-8c8jd" Oct 30 19:01:18.317862 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:18.317836 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2014b449-aa26-47e8-bb93-893588bc6083-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-551d0b-8c8jd\" (UID: \"2014b449-aa26-47e8-bb93-893588bc6083\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-551d0b-8c8jd" Oct 30 19:01:18.420050 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:18.420002 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2014b449-aa26-47e8-bb93-893588bc6083-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-551d0b-8c8jd\" (UID: \"2014b449-aa26-47e8-bb93-893588bc6083\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-551d0b-8c8jd" Oct 30 19:01:18.420312 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:18.420292 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2014b449-aa26-47e8-bb93-893588bc6083-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-551d0b-8c8jd\" (UID: \"2014b449-aa26-47e8-bb93-893588bc6083\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-551d0b-8c8jd" Oct 30 19:01:18.420437 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:18.420355 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4gkg\" (UniqueName: \"kubernetes.io/projected/2014b449-aa26-47e8-bb93-893588bc6083-kube-api-access-r4gkg\") pod \"maas-api-revoke-keys-e2e-inf-b-551d0b-8c8jd\" (UID: \"2014b449-aa26-47e8-bb93-893588bc6083\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-551d0b-8c8jd" Oct 30 19:01:18.420832 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:18.420810 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2014b449-aa26-47e8-bb93-893588bc6083-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-551d0b-8c8jd\" (UID: \"2014b449-aa26-47e8-bb93-893588bc6083\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-551d0b-8c8jd" Oct 30 19:01:18.422917 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:18.422900 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2014b449-aa26-47e8-bb93-893588bc6083-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-551d0b-8c8jd\" (UID: \"2014b449-aa26-47e8-bb93-893588bc6083\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-551d0b-8c8jd" Oct 30 19:01:18.470558 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:18.470483 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4gkg\" (UniqueName: \"kubernetes.io/projected/2014b449-aa26-47e8-bb93-893588bc6083-kube-api-access-r4gkg\") pod \"maas-api-revoke-keys-e2e-inf-b-551d0b-8c8jd\" (UID: \"2014b449-aa26-47e8-bb93-893588bc6083\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-551d0b-8c8jd" Oct 30 19:01:18.539456 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:18.539411 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-551d0b-8c8jd" Oct 30 19:01:18.956080 ip-10-0-138-176 kubenswrapper[4442]: W1030 19:01:18.956044 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2014b449_aa26_47e8_bb93_893588bc6083.slice/crio-d8a6c4e786e8ffb1f7a66e4714fd9d8fe44aa05afc34bb2f1cb7a5e66eed8c0b WatchSource:0}: Error finding container d8a6c4e786e8ffb1f7a66e4714fd9d8fe44aa05afc34bb2f1cb7a5e66eed8c0b: Status 404 returned error can't find the container with id d8a6c4e786e8ffb1f7a66e4714fd9d8fe44aa05afc34bb2f1cb7a5e66eed8c0b Oct 30 19:01:18.956553 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:18.956458 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-551d0b-8c8jd"] Oct 30 19:01:19.252276 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:19.252244 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-551d0b-8c8jd" event={"ID":"2014b449-aa26-47e8-bb93-893588bc6083","Type":"ContainerStarted","Data":"d8a6c4e786e8ffb1f7a66e4714fd9d8fe44aa05afc34bb2f1cb7a5e66eed8c0b"} Oct 30 19:01:20.264292 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:20.264200 4442 generic.go:356] "Generic (PLEG): container finished" podID="2014b449-aa26-47e8-bb93-893588bc6083" containerID="846b422026467ec817b01e85b3fcce1e3c195afd96f8dd4e453ffb4d7e053cba" exitCode=0 Oct 30 19:01:20.264682 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:20.264289 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-551d0b-8c8jd" event={"ID":"2014b449-aa26-47e8-bb93-893588bc6083","Type":"ContainerDied","Data":"846b422026467ec817b01e85b3fcce1e3c195afd96f8dd4e453ffb4d7e053cba"} Oct 30 19:01:21.445919 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:21.445883 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-551d0b-8c8jd" Oct 30 19:01:21.570985 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:21.570880 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2014b449-aa26-47e8-bb93-893588bc6083-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2014b449-aa26-47e8-bb93-893588bc6083-maas-api-openshift-ca\") pod \"2014b449-aa26-47e8-bb93-893588bc6083\" (UID: \"2014b449-aa26-47e8-bb93-893588bc6083\") " Oct 30 19:01:21.571232 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:21.571016 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2014b449-aa26-47e8-bb93-893588bc6083-kube-api-access-r4gkg\" (UniqueName: \"kubernetes.io/projected/2014b449-aa26-47e8-bb93-893588bc6083-kube-api-access-r4gkg\") pod \"2014b449-aa26-47e8-bb93-893588bc6083\" (UID: \"2014b449-aa26-47e8-bb93-893588bc6083\") " Oct 30 19:01:21.571340 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:21.571300 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2014b449-aa26-47e8-bb93-893588bc6083-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2014b449-aa26-47e8-bb93-893588bc6083-maas-api-xks-ca\") pod \"2014b449-aa26-47e8-bb93-893588bc6083\" (UID: \"2014b449-aa26-47e8-bb93-893588bc6083\") " Oct 30 19:01:21.571502 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:21.571358 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2014b449-aa26-47e8-bb93-893588bc6083-maas-api-openshift-ca" pod "2014b449-aa26-47e8-bb93-893588bc6083" (UID: "2014b449-aa26-47e8-bb93-893588bc6083"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:01:21.572271 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:21.572245 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2014b449-aa26-47e8-bb93-893588bc6083-maas-api-openshift-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:01:21.573844 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:21.573815 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2014b449-aa26-47e8-bb93-893588bc6083-maas-api-xks-ca" pod "2014b449-aa26-47e8-bb93-893588bc6083" (UID: "2014b449-aa26-47e8-bb93-893588bc6083"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:01:21.573942 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:21.573881 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2014b449-aa26-47e8-bb93-893588bc6083-kube-api-access-r4gkg" pod "2014b449-aa26-47e8-bb93-893588bc6083" (UID: "2014b449-aa26-47e8-bb93-893588bc6083"). InnerVolumeSpecName "kube-api-access-r4gkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:01:21.677897 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:21.673937 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4gkg\" (UniqueName: \"kubernetes.io/projected/2014b449-aa26-47e8-bb93-893588bc6083-kube-api-access-r4gkg\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:01:21.677897 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:21.674021 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2014b449-aa26-47e8-bb93-893588bc6083-maas-api-xks-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:01:22.282657 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:22.282624 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-551d0b-8c8jd" Oct 30 19:01:22.282657 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:22.282648 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-551d0b-8c8jd" event={"ID":"2014b449-aa26-47e8-bb93-893588bc6083","Type":"ContainerDied","Data":"d8a6c4e786e8ffb1f7a66e4714fd9d8fe44aa05afc34bb2f1cb7a5e66eed8c0b"} Oct 30 19:01:22.282889 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:22.282701 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8a6c4e786e8ffb1f7a66e4714fd9d8fe44aa05afc34bb2f1cb7a5e66eed8c0b" Oct 30 19:01:22.414658 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:01:22.414625 4442 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5abb7ef5_cc99_469a_a165_9e9394370418.slice\": RecentStats: unable to find data in memory cache]" Oct 30 19:01:22.612938 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:01:22.612850 4442 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5abb7ef5_cc99_469a_a165_9e9394370418.slice\": RecentStats: unable to find data in memory cache]" Oct 30 19:01:28.441648 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:28.441616 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s"] Oct 30 19:01:28.442506 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:28.442488 4442 cpu_manager.go:414] "containerMap: removing container" podUID="2014b449-aa26-47e8-bb93-893588bc6083" containerName="revoke-keys" Oct 30 19:01:28.442565 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:28.442508 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2014b449-aa26-47e8-bb93-893588bc6083" containerName="revoke-keys" Oct 30 19:01:28.442686 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:28.442675 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="2014b449-aa26-47e8-bb93-893588bc6083" containerName="revoke-keys" Oct 30 19:01:28.446448 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:28.446424 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" Oct 30 19:01:28.519311 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:28.519257 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/921aaf3a-c623-4688-8c39-601e42b9d567-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-qwf7s\" (UID: \"921aaf3a-c623-4688-8c39-601e42b9d567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" Oct 30 19:01:28.519720 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:28.519670 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/921aaf3a-c623-4688-8c39-601e42b9d567-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-qwf7s\" (UID: \"921aaf3a-c623-4688-8c39-601e42b9d567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" Oct 30 19:01:28.531597 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:28.531515 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxdcc\" (UniqueName: \"kubernetes.io/projected/921aaf3a-c623-4688-8c39-601e42b9d567-kube-api-access-zxdcc\") pod \"maas-default-gateway-openshift-default-76597d6855-qwf7s\" (UID: \"921aaf3a-c623-4688-8c39-601e42b9d567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" Oct 30 19:01:28.548162 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:28.531974 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/921aaf3a-c623-4688-8c39-601e42b9d567-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-qwf7s\" (UID: \"921aaf3a-c623-4688-8c39-601e42b9d567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" Oct 30 19:01:28.548779 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:28.548746 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/921aaf3a-c623-4688-8c39-601e42b9d567-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-qwf7s\" (UID: \"921aaf3a-c623-4688-8c39-601e42b9d567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" Oct 30 19:01:28.549111 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:28.549080 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/921aaf3a-c623-4688-8c39-601e42b9d567-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-qwf7s\" (UID: \"921aaf3a-c623-4688-8c39-601e42b9d567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" Oct 30 19:01:28.549433 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:28.549409 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/921aaf3a-c623-4688-8c39-601e42b9d567-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-qwf7s\" (UID: \"921aaf3a-c623-4688-8c39-601e42b9d567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" Oct 30 19:01:28.550890 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:28.550857 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/921aaf3a-c623-4688-8c39-601e42b9d567-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-qwf7s\" (UID: \"921aaf3a-c623-4688-8c39-601e42b9d567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" Oct 30 19:01:28.551023 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:28.550945 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/921aaf3a-c623-4688-8c39-601e42b9d567-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-qwf7s\" (UID: \"921aaf3a-c623-4688-8c39-601e42b9d567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" Oct 30 19:01:28.565740 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:28.565699 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s"] Oct 30 19:01:28.653441 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:28.653263 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/921aaf3a-c623-4688-8c39-601e42b9d567-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-qwf7s\" (UID: \"921aaf3a-c623-4688-8c39-601e42b9d567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" Oct 30 19:01:28.653441 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:28.653412 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/921aaf3a-c623-4688-8c39-601e42b9d567-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-qwf7s\" (UID: \"921aaf3a-c623-4688-8c39-601e42b9d567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" Oct 30 19:01:28.653783 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:28.653549 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/921aaf3a-c623-4688-8c39-601e42b9d567-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-qwf7s\" (UID: \"921aaf3a-c623-4688-8c39-601e42b9d567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" Oct 30 19:01:28.653783 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:28.653614 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/921aaf3a-c623-4688-8c39-601e42b9d567-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-qwf7s\" (UID: \"921aaf3a-c623-4688-8c39-601e42b9d567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" Oct 30 19:01:28.653783 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:28.653648 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/921aaf3a-c623-4688-8c39-601e42b9d567-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-qwf7s\" (UID: \"921aaf3a-c623-4688-8c39-601e42b9d567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" Oct 30 19:01:28.653783 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:28.653712 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/921aaf3a-c623-4688-8c39-601e42b9d567-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-qwf7s\" (UID: \"921aaf3a-c623-4688-8c39-601e42b9d567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" Oct 30 19:01:28.653783 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:28.653751 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/921aaf3a-c623-4688-8c39-601e42b9d567-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-qwf7s\" (UID: \"921aaf3a-c623-4688-8c39-601e42b9d567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" Oct 30 19:01:28.654126 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:28.653823 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxdcc\" (UniqueName: \"kubernetes.io/projected/921aaf3a-c623-4688-8c39-601e42b9d567-kube-api-access-zxdcc\") pod \"maas-default-gateway-openshift-default-76597d6855-qwf7s\" (UID: \"921aaf3a-c623-4688-8c39-601e42b9d567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" Oct 30 19:01:28.654126 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:28.653924 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/921aaf3a-c623-4688-8c39-601e42b9d567-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-qwf7s\" (UID: \"921aaf3a-c623-4688-8c39-601e42b9d567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" Oct 30 19:01:28.654126 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:28.654030 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/921aaf3a-c623-4688-8c39-601e42b9d567-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-qwf7s\" (UID: \"921aaf3a-c623-4688-8c39-601e42b9d567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" Oct 30 19:01:28.654957 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:28.654925 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/921aaf3a-c623-4688-8c39-601e42b9d567-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-qwf7s\" (UID: \"921aaf3a-c623-4688-8c39-601e42b9d567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" Oct 30 19:01:28.655283 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:28.655254 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/921aaf3a-c623-4688-8c39-601e42b9d567-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-qwf7s\" (UID: \"921aaf3a-c623-4688-8c39-601e42b9d567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" Oct 30 19:01:28.657280 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:28.655535 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/921aaf3a-c623-4688-8c39-601e42b9d567-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-qwf7s\" (UID: \"921aaf3a-c623-4688-8c39-601e42b9d567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" Oct 30 19:01:28.659163 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:28.659114 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/921aaf3a-c623-4688-8c39-601e42b9d567-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-qwf7s\" (UID: \"921aaf3a-c623-4688-8c39-601e42b9d567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" Oct 30 19:01:28.660624 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:28.660600 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/921aaf3a-c623-4688-8c39-601e42b9d567-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-qwf7s\" (UID: \"921aaf3a-c623-4688-8c39-601e42b9d567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" Oct 30 19:01:28.672008 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:28.671973 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/921aaf3a-c623-4688-8c39-601e42b9d567-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-qwf7s\" (UID: \"921aaf3a-c623-4688-8c39-601e42b9d567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" Oct 30 19:01:28.683979 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:28.683944 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/921aaf3a-c623-4688-8c39-601e42b9d567-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-qwf7s\" (UID: \"921aaf3a-c623-4688-8c39-601e42b9d567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" Oct 30 19:01:28.684449 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:28.684220 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxdcc\" (UniqueName: \"kubernetes.io/projected/921aaf3a-c623-4688-8c39-601e42b9d567-kube-api-access-zxdcc\") pod \"maas-default-gateway-openshift-default-76597d6855-qwf7s\" (UID: \"921aaf3a-c623-4688-8c39-601e42b9d567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" Oct 30 19:01:28.764370 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:28.764269 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" Oct 30 19:01:29.105356 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:29.105279 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-551d0b-6686dd876d-qvsgq"] Oct 30 19:01:29.181173 ip-10-0-138-176 kubenswrapper[4442]: W1030 19:01:29.181143 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod921aaf3a_c623_4688_8c39_601e42b9d567.slice/crio-7f92ab7fc67a7c5455d2eedbb808564df45050c6acd6d093c64eece3d4488036 WatchSource:0}: Error finding container 7f92ab7fc67a7c5455d2eedbb808564df45050c6acd6d093c64eece3d4488036: Status 404 returned error can't find the container with id 7f92ab7fc67a7c5455d2eedbb808564df45050c6acd6d093c64eece3d4488036 Oct 30 19:01:29.183857 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:29.183819 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s"] Oct 30 19:01:29.184499 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:29.184464 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:01:29.184590 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:29.184534 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:01:29.184590 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:29.184566 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:01:29.355268 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:29.355221 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" event={"ID":"921aaf3a-c623-4688-8c39-601e42b9d567","Type":"ContainerStarted","Data":"2943958ae2bc7e045eec02109beb2100116eb274e165f40726bdc0dac7c787c8"} Oct 30 19:01:29.355491 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:29.355311 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" event={"ID":"921aaf3a-c623-4688-8c39-601e42b9d567","Type":"ContainerStarted","Data":"7f92ab7fc67a7c5455d2eedbb808564df45050c6acd6d093c64eece3d4488036"} Oct 30 19:01:29.467557 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:29.457384 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" podStartSLOduration=1.457364366 podStartE2EDuration="1.457364366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:29.456093875 +0000 UTC m=+3786.546156215" watchObservedRunningTime="2026-10-30 19:01:29.457364366 +0000 UTC m=+3786.547426706" Oct 30 19:01:29.766027 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:29.765997 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" Oct 30 19:01:30.766681 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:30.766634 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-qwf7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" start-of-body= Oct 30 19:01:30.767239 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:30.766690 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" podUID="921aaf3a-c623-4688-8c39-601e42b9d567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" Oct 30 19:01:31.765757 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:31.765717 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-qwf7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" start-of-body= Oct 30 19:01:31.765977 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:31.765784 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" podUID="921aaf3a-c623-4688-8c39-601e42b9d567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" Oct 30 19:01:32.765715 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:32.765674 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-qwf7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" start-of-body= Oct 30 19:01:32.766192 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:32.765740 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" podUID="921aaf3a-c623-4688-8c39-601e42b9d567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" Oct 30 19:01:33.764997 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:33.764960 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-qwf7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" start-of-body= Oct 30 19:01:33.765211 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:33.765024 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" podUID="921aaf3a-c623-4688-8c39-601e42b9d567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" Oct 30 19:01:34.105685 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:34.105588 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-551d0b-6686dd876d-qvsgq" podUID="dd895ac2-c8b9-47df-93cd-0da009ab4c1e" containerName="payload-processing" containerID="cri-o://9d3c41c9902d6e1db4462b26df0b49780580fcd329c3006f0c5a0c642347f813" gracePeriod=25 Oct 30 19:01:34.765723 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:34.765686 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-qwf7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" start-of-body= Oct 30 19:01:34.765954 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:34.765756 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" podUID="921aaf3a-c623-4688-8c39-601e42b9d567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" Oct 30 19:01:35.765231 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:35.765196 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-qwf7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" start-of-body= Oct 30 19:01:35.765757 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:35.765248 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" podUID="921aaf3a-c623-4688-8c39-601e42b9d567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" Oct 30 19:01:36.765088 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:36.765048 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-qwf7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" start-of-body= Oct 30 19:01:36.765320 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:36.765181 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" podUID="921aaf3a-c623-4688-8c39-601e42b9d567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" Oct 30 19:01:37.659160 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:37.658284 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-551d0b-6686dd876d-qvsgq" podUID="dd895ac2-c8b9-47df-93cd-0da009ab4c1e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:01:38.285960 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:38.285929 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" Oct 30 19:01:38.286443 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:38.286405 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" Oct 30 19:01:38.287576 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:38.287554 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" Oct 30 19:01:41.181972 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:41.181935 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8862aa-zccpp"] Oct 30 19:01:41.193547 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:41.193514 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8862aa-zccpp"] Oct 30 19:01:43.067884 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:43.067844 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e475ae0a-ccb4-41e8-844c-6c566ad3754a" path="/var/lib/kubelet/pods/e475ae0a-ccb4-41e8-844c-6c566ad3754a/volumes" Oct 30 19:01:43.448947 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:43.448901 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb"] Oct 30 19:01:43.457931 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:43.457900 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" Oct 30 19:01:43.483498 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:43.483403 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb"] Oct 30 19:01:43.598049 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:43.597188 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d854e8-50eb-4747-9c43-29f5fcdbc955-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-xkggb\" (UID: \"49d854e8-50eb-4747-9c43-29f5fcdbc955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" Oct 30 19:01:43.598049 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:43.597276 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49d854e8-50eb-4747-9c43-29f5fcdbc955-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-xkggb\" (UID: \"49d854e8-50eb-4747-9c43-29f5fcdbc955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" Oct 30 19:01:43.598049 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:43.597318 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49d854e8-50eb-4747-9c43-29f5fcdbc955-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-xkggb\" (UID: \"49d854e8-50eb-4747-9c43-29f5fcdbc955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" Oct 30 19:01:43.598049 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:43.597381 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d854e8-50eb-4747-9c43-29f5fcdbc955-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-xkggb\" (UID: \"49d854e8-50eb-4747-9c43-29f5fcdbc955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" Oct 30 19:01:43.598049 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:43.597480 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h92ms\" (UniqueName: \"kubernetes.io/projected/49d854e8-50eb-4747-9c43-29f5fcdbc955-kube-api-access-h92ms\") pod \"maas-default-gateway-openshift-default-76597d6855-xkggb\" (UID: \"49d854e8-50eb-4747-9c43-29f5fcdbc955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" Oct 30 19:01:43.598049 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:43.597532 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49d854e8-50eb-4747-9c43-29f5fcdbc955-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-xkggb\" (UID: \"49d854e8-50eb-4747-9c43-29f5fcdbc955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" Oct 30 19:01:43.598049 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:43.597585 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49d854e8-50eb-4747-9c43-29f5fcdbc955-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-xkggb\" (UID: \"49d854e8-50eb-4747-9c43-29f5fcdbc955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" Oct 30 19:01:43.598049 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:43.597793 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49d854e8-50eb-4747-9c43-29f5fcdbc955-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-xkggb\" (UID: \"49d854e8-50eb-4747-9c43-29f5fcdbc955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" Oct 30 19:01:43.598049 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:43.597854 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49d854e8-50eb-4747-9c43-29f5fcdbc955-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-xkggb\" (UID: \"49d854e8-50eb-4747-9c43-29f5fcdbc955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" Oct 30 19:01:43.699878 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:43.699794 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49d854e8-50eb-4747-9c43-29f5fcdbc955-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-xkggb\" (UID: \"49d854e8-50eb-4747-9c43-29f5fcdbc955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" Oct 30 19:01:43.699878 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:43.699839 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49d854e8-50eb-4747-9c43-29f5fcdbc955-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-xkggb\" (UID: \"49d854e8-50eb-4747-9c43-29f5fcdbc955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" Oct 30 19:01:43.700101 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:43.699898 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d854e8-50eb-4747-9c43-29f5fcdbc955-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-xkggb\" (UID: \"49d854e8-50eb-4747-9c43-29f5fcdbc955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" Oct 30 19:01:43.700101 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:43.699994 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h92ms\" (UniqueName: \"kubernetes.io/projected/49d854e8-50eb-4747-9c43-29f5fcdbc955-kube-api-access-h92ms\") pod \"maas-default-gateway-openshift-default-76597d6855-xkggb\" (UID: \"49d854e8-50eb-4747-9c43-29f5fcdbc955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" Oct 30 19:01:43.700101 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:43.700049 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49d854e8-50eb-4747-9c43-29f5fcdbc955-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-xkggb\" (UID: \"49d854e8-50eb-4747-9c43-29f5fcdbc955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" Oct 30 19:01:43.700363 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:43.700116 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49d854e8-50eb-4747-9c43-29f5fcdbc955-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-xkggb\" (UID: \"49d854e8-50eb-4747-9c43-29f5fcdbc955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" Oct 30 19:01:43.700363 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:43.700303 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49d854e8-50eb-4747-9c43-29f5fcdbc955-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-xkggb\" (UID: \"49d854e8-50eb-4747-9c43-29f5fcdbc955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" Oct 30 19:01:43.700477 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:43.700361 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49d854e8-50eb-4747-9c43-29f5fcdbc955-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-xkggb\" (UID: \"49d854e8-50eb-4747-9c43-29f5fcdbc955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" Oct 30 19:01:43.700477 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:43.700378 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49d854e8-50eb-4747-9c43-29f5fcdbc955-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-xkggb\" (UID: \"49d854e8-50eb-4747-9c43-29f5fcdbc955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" Oct 30 19:01:43.700477 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:43.700406 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d854e8-50eb-4747-9c43-29f5fcdbc955-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-xkggb\" (UID: \"49d854e8-50eb-4747-9c43-29f5fcdbc955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" Oct 30 19:01:43.700666 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:43.700642 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49d854e8-50eb-4747-9c43-29f5fcdbc955-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-xkggb\" (UID: \"49d854e8-50eb-4747-9c43-29f5fcdbc955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" Oct 30 19:01:43.700768 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:43.700750 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d854e8-50eb-4747-9c43-29f5fcdbc955-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-xkggb\" (UID: \"49d854e8-50eb-4747-9c43-29f5fcdbc955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" Oct 30 19:01:43.700878 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:43.700859 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49d854e8-50eb-4747-9c43-29f5fcdbc955-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-xkggb\" (UID: \"49d854e8-50eb-4747-9c43-29f5fcdbc955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" Oct 30 19:01:43.701088 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:43.701072 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d854e8-50eb-4747-9c43-29f5fcdbc955-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-xkggb\" (UID: \"49d854e8-50eb-4747-9c43-29f5fcdbc955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" Oct 30 19:01:43.702778 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:43.702755 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49d854e8-50eb-4747-9c43-29f5fcdbc955-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-xkggb\" (UID: \"49d854e8-50eb-4747-9c43-29f5fcdbc955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" Oct 30 19:01:43.703063 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:43.703046 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49d854e8-50eb-4747-9c43-29f5fcdbc955-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-xkggb\" (UID: \"49d854e8-50eb-4747-9c43-29f5fcdbc955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" Oct 30 19:01:43.730720 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:43.730656 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h92ms\" (UniqueName: \"kubernetes.io/projected/49d854e8-50eb-4747-9c43-29f5fcdbc955-kube-api-access-h92ms\") pod \"maas-default-gateway-openshift-default-76597d6855-xkggb\" (UID: \"49d854e8-50eb-4747-9c43-29f5fcdbc955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" Oct 30 19:01:43.732950 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:43.732918 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49d854e8-50eb-4747-9c43-29f5fcdbc955-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-xkggb\" (UID: \"49d854e8-50eb-4747-9c43-29f5fcdbc955\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" Oct 30 19:01:43.778837 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:43.778801 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" Oct 30 19:01:44.111584 ip-10-0-138-176 kubenswrapper[4442]: W1030 19:01:44.111552 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49d854e8_50eb_4747_9c43_29f5fcdbc955.slice/crio-0cc5bd37e8d0b9954b0dfe1f6de559128c3b763cf99d9ea2a7dc09f2e0b5edc1 WatchSource:0}: Error finding container 0cc5bd37e8d0b9954b0dfe1f6de559128c3b763cf99d9ea2a7dc09f2e0b5edc1: Status 404 returned error can't find the container with id 0cc5bd37e8d0b9954b0dfe1f6de559128c3b763cf99d9ea2a7dc09f2e0b5edc1 Oct 30 19:01:44.114782 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:44.114745 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:01:44.114917 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:44.114826 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:01:44.114917 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:44.114862 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:01:44.114917 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:44.114874 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb"] Oct 30 19:01:44.717963 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:44.717921 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" event={"ID":"49d854e8-50eb-4747-9c43-29f5fcdbc955","Type":"ContainerStarted","Data":"05f1580b0d3e567e290b24af7fb44afb12c172e7868e2d5d7ee392cea5de669a"} Oct 30 19:01:44.718145 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:44.717994 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" event={"ID":"49d854e8-50eb-4747-9c43-29f5fcdbc955","Type":"ContainerStarted","Data":"0cc5bd37e8d0b9954b0dfe1f6de559128c3b763cf99d9ea2a7dc09f2e0b5edc1"} Oct 30 19:01:44.769313 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:44.769264 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" podStartSLOduration=1.7692496260000001 podStartE2EDuration="1.769249626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:44.764534233 +0000 UTC m=+3801.854596609" watchObservedRunningTime="2026-10-30 19:01:44.769249626 +0000 UTC m=+3801.859312032" Oct 30 19:01:44.779643 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:44.779613 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" Oct 30 19:01:45.781752 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:45.781717 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-xkggb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" start-of-body= Oct 30 19:01:45.782182 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:45.781774 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" podUID="49d854e8-50eb-4747-9c43-29f5fcdbc955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" Oct 30 19:01:46.780075 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:46.780036 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-xkggb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" start-of-body= Oct 30 19:01:46.780296 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:46.780097 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" podUID="49d854e8-50eb-4747-9c43-29f5fcdbc955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" Oct 30 19:01:47.585295 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:01:47.585244 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 19:01:47.706157 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:47.700320 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-551d0b-6686dd876d-qvsgq" podUID="dd895ac2-c8b9-47df-93cd-0da009ab4c1e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:01:47.779351 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:47.779319 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-xkggb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" start-of-body= Oct 30 19:01:47.779563 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:47.779367 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" podUID="49d854e8-50eb-4747-9c43-29f5fcdbc955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" Oct 30 19:01:48.779877 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:48.779839 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-xkggb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" start-of-body= Oct 30 19:01:48.780410 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:48.779898 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" podUID="49d854e8-50eb-4747-9c43-29f5fcdbc955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" Oct 30 19:01:49.290081 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:49.290055 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm"] Oct 30 19:01:49.290913 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:49.290487 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" podUID="b15a372b-64ad-449b-8f2c-c7b49dbe4600" containerName="istio-proxy" containerID="cri-o://20151f26f52bf7959f6411cb8f2ba1b8ad0ba6f5579d1969b47e1e4b272d859d" gracePeriod=30 Oct 30 19:01:49.780032 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:49.779669 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-xkggb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" start-of-body= Oct 30 19:01:49.780032 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:49.779715 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" podUID="49d854e8-50eb-4747-9c43-29f5fcdbc955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" Oct 30 19:01:49.969780 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:49.969743 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f37ca7-xdc9j"] Oct 30 19:01:49.975695 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:49.975669 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f37ca7-xdc9j" Oct 30 19:01:49.995124 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:49.995098 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f37ca7-xdc9j"] Oct 30 19:01:50.021325 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:50.018031 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:01:50.021325 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:50.018182 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:01:50.098819 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:50.098738 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf20e69b-9704-4362-8906-cadb6ab06fde-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f37ca7-xdc9j\" (UID: \"cf20e69b-9704-4362-8906-cadb6ab06fde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f37ca7-xdc9j" Oct 30 19:01:50.098819 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:50.098783 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf20e69b-9704-4362-8906-cadb6ab06fde-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f37ca7-xdc9j\" (UID: \"cf20e69b-9704-4362-8906-cadb6ab06fde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f37ca7-xdc9j" Oct 30 19:01:50.099005 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:50.098979 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgr9s\" (UniqueName: \"kubernetes.io/projected/cf20e69b-9704-4362-8906-cadb6ab06fde-kube-api-access-zgr9s\") pod \"maas-api-revoke-keys-e2e-inf-b-f37ca7-xdc9j\" (UID: \"cf20e69b-9704-4362-8906-cadb6ab06fde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f37ca7-xdc9j" Oct 30 19:01:50.200209 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:50.200171 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf20e69b-9704-4362-8906-cadb6ab06fde-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f37ca7-xdc9j\" (UID: \"cf20e69b-9704-4362-8906-cadb6ab06fde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f37ca7-xdc9j" Oct 30 19:01:50.200450 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:50.200222 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf20e69b-9704-4362-8906-cadb6ab06fde-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f37ca7-xdc9j\" (UID: \"cf20e69b-9704-4362-8906-cadb6ab06fde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f37ca7-xdc9j" Oct 30 19:01:50.200558 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:50.200531 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgr9s\" (UniqueName: \"kubernetes.io/projected/cf20e69b-9704-4362-8906-cadb6ab06fde-kube-api-access-zgr9s\") pod \"maas-api-revoke-keys-e2e-inf-b-f37ca7-xdc9j\" (UID: \"cf20e69b-9704-4362-8906-cadb6ab06fde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f37ca7-xdc9j" Oct 30 19:01:50.201030 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:50.201007 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf20e69b-9704-4362-8906-cadb6ab06fde-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f37ca7-xdc9j\" (UID: \"cf20e69b-9704-4362-8906-cadb6ab06fde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f37ca7-xdc9j" Oct 30 19:01:50.202942 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:50.202924 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf20e69b-9704-4362-8906-cadb6ab06fde-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f37ca7-xdc9j\" (UID: \"cf20e69b-9704-4362-8906-cadb6ab06fde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f37ca7-xdc9j" Oct 30 19:01:50.221239 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:50.221212 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgr9s\" (UniqueName: \"kubernetes.io/projected/cf20e69b-9704-4362-8906-cadb6ab06fde-kube-api-access-zgr9s\") pod \"maas-api-revoke-keys-e2e-inf-b-f37ca7-xdc9j\" (UID: \"cf20e69b-9704-4362-8906-cadb6ab06fde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f37ca7-xdc9j" Oct 30 19:01:50.297975 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:50.297941 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f37ca7-xdc9j" Oct 30 19:01:50.656414 ip-10-0-138-176 kubenswrapper[4442]: W1030 19:01:50.656377 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf20e69b_9704_4362_8906_cadb6ab06fde.slice/crio-c0236483ae14b53929eaedaed61189c73b37cf6e4be2660d3efa5050cdbdab1f WatchSource:0}: Error finding container c0236483ae14b53929eaedaed61189c73b37cf6e4be2660d3efa5050cdbdab1f: Status 404 returned error can't find the container with id c0236483ae14b53929eaedaed61189c73b37cf6e4be2660d3efa5050cdbdab1f Oct 30 19:01:50.658453 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:50.658351 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f37ca7-xdc9j"] Oct 30 19:01:50.779966 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:50.779928 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-xkggb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" start-of-body= Oct 30 19:01:50.780197 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:50.779995 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" podUID="49d854e8-50eb-4747-9c43-29f5fcdbc955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" Oct 30 19:01:50.811248 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:50.811205 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f37ca7-xdc9j" event={"ID":"cf20e69b-9704-4362-8906-cadb6ab06fde","Type":"ContainerStarted","Data":"c0236483ae14b53929eaedaed61189c73b37cf6e4be2660d3efa5050cdbdab1f"} Oct 30 19:01:51.780163 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:51.780115 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-xkggb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" start-of-body= Oct 30 19:01:51.780380 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:51.780185 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" podUID="49d854e8-50eb-4747-9c43-29f5fcdbc955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" Oct 30 19:01:51.823582 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:51.823546 4442 generic.go:356] "Generic (PLEG): container finished" podID="cf20e69b-9704-4362-8906-cadb6ab06fde" containerID="b9aa1eae205c029255b60967dd88f6f54a52a1f4822ae6a7b894d53c15b30c12" exitCode=0 Oct 30 19:01:51.823756 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:51.823624 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f37ca7-xdc9j" event={"ID":"cf20e69b-9704-4362-8906-cadb6ab06fde","Type":"ContainerDied","Data":"b9aa1eae205c029255b60967dd88f6f54a52a1f4822ae6a7b894d53c15b30c12"} Oct 30 19:01:53.072420 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:53.072370 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f37ca7-xdc9j" Oct 30 19:01:53.178162 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:53.173849 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf20e69b-9704-4362-8906-cadb6ab06fde-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf20e69b-9704-4362-8906-cadb6ab06fde-maas-api-xks-ca\") pod \"cf20e69b-9704-4362-8906-cadb6ab06fde\" (UID: \"cf20e69b-9704-4362-8906-cadb6ab06fde\") " Oct 30 19:01:53.178162 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:53.173933 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf20e69b-9704-4362-8906-cadb6ab06fde-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf20e69b-9704-4362-8906-cadb6ab06fde-maas-api-openshift-ca\") pod \"cf20e69b-9704-4362-8906-cadb6ab06fde\" (UID: \"cf20e69b-9704-4362-8906-cadb6ab06fde\") " Oct 30 19:01:53.189112 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:53.182245 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf20e69b-9704-4362-8906-cadb6ab06fde-kube-api-access-zgr9s\" (UniqueName: \"kubernetes.io/projected/cf20e69b-9704-4362-8906-cadb6ab06fde-kube-api-access-zgr9s\") pod \"cf20e69b-9704-4362-8906-cadb6ab06fde\" (UID: \"cf20e69b-9704-4362-8906-cadb6ab06fde\") " Oct 30 19:01:53.189112 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:53.187200 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf20e69b-9704-4362-8906-cadb6ab06fde-maas-api-openshift-ca" pod "cf20e69b-9704-4362-8906-cadb6ab06fde" (UID: "cf20e69b-9704-4362-8906-cadb6ab06fde"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:01:53.192797 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:53.192756 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf20e69b-9704-4362-8906-cadb6ab06fde-kube-api-access-zgr9s" pod "cf20e69b-9704-4362-8906-cadb6ab06fde" (UID: "cf20e69b-9704-4362-8906-cadb6ab06fde"). InnerVolumeSpecName "kube-api-access-zgr9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:01:53.208407 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:53.208252 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf20e69b-9704-4362-8906-cadb6ab06fde-maas-api-xks-ca" pod "cf20e69b-9704-4362-8906-cadb6ab06fde" (UID: "cf20e69b-9704-4362-8906-cadb6ab06fde"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:01:53.286934 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:53.286895 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgr9s\" (UniqueName: \"kubernetes.io/projected/cf20e69b-9704-4362-8906-cadb6ab06fde-kube-api-access-zgr9s\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:01:53.286934 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:53.286935 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf20e69b-9704-4362-8906-cadb6ab06fde-maas-api-xks-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:01:53.287163 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:53.286951 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf20e69b-9704-4362-8906-cadb6ab06fde-maas-api-openshift-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:01:53.488824 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:53.488725 4442 patch_prober.go:29] interesting pod/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:01:53.489012 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:53.488843 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" podUID="b15a372b-64ad-449b-8f2c-c7b49dbe4600" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:01:53.612351 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:53.612314 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" Oct 30 19:01:53.613092 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:53.613065 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" Oct 30 19:01:53.614410 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:53.614385 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" Oct 30 19:01:53.866703 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:53.866669 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f37ca7-xdc9j" Oct 30 19:01:53.873477 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:53.873423 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f37ca7-xdc9j" event={"ID":"cf20e69b-9704-4362-8906-cadb6ab06fde","Type":"ContainerDied","Data":"c0236483ae14b53929eaedaed61189c73b37cf6e4be2660d3efa5050cdbdab1f"} Oct 30 19:01:53.873700 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:53.873512 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0236483ae14b53929eaedaed61189c73b37cf6e4be2660d3efa5050cdbdab1f" Oct 30 19:01:54.484353 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.484327 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-551d0b-6686dd876d-qvsgq" Oct 30 19:01:54.604630 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.604602 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" Oct 30 19:01:54.619042 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.619007 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd895ac2-c8b9-47df-93cd-0da009ab4c1e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd895ac2-c8b9-47df-93cd-0da009ab4c1e-plugins-config-volume\") pod \"dd895ac2-c8b9-47df-93cd-0da009ab4c1e\" (UID: \"dd895ac2-c8b9-47df-93cd-0da009ab4c1e\") " Oct 30 19:01:54.619256 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.619197 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd895ac2-c8b9-47df-93cd-0da009ab4c1e-kube-api-access-knd28\" (UniqueName: \"kubernetes.io/projected/dd895ac2-c8b9-47df-93cd-0da009ab4c1e-kube-api-access-knd28\") pod \"dd895ac2-c8b9-47df-93cd-0da009ab4c1e\" (UID: \"dd895ac2-c8b9-47df-93cd-0da009ab4c1e\") " Oct 30 19:01:54.620754 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.620693 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd895ac2-c8b9-47df-93cd-0da009ab4c1e-plugins-config-volume" pod "dd895ac2-c8b9-47df-93cd-0da009ab4c1e" (UID: "dd895ac2-c8b9-47df-93cd-0da009ab4c1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:01:54.621987 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.621959 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd895ac2-c8b9-47df-93cd-0da009ab4c1e-kube-api-access-knd28" pod "dd895ac2-c8b9-47df-93cd-0da009ab4c1e" (UID: "dd895ac2-c8b9-47df-93cd-0da009ab4c1e"). InnerVolumeSpecName "kube-api-access-knd28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:01:54.721523 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.721423 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b15a372b-64ad-449b-8f2c-c7b49dbe4600-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b15a372b-64ad-449b-8f2c-c7b49dbe4600-istio-data\") pod \"b15a372b-64ad-449b-8f2c-c7b49dbe4600\" (UID: \"b15a372b-64ad-449b-8f2c-c7b49dbe4600\") " Oct 30 19:01:54.721713 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.721587 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b15a372b-64ad-449b-8f2c-c7b49dbe4600-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b15a372b-64ad-449b-8f2c-c7b49dbe4600-workload-certs\") pod \"b15a372b-64ad-449b-8f2c-c7b49dbe4600\" (UID: \"b15a372b-64ad-449b-8f2c-c7b49dbe4600\") " Oct 30 19:01:54.721713 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.721630 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b15a372b-64ad-449b-8f2c-c7b49dbe4600-istio-token\" (UniqueName: \"kubernetes.io/projected/b15a372b-64ad-449b-8f2c-c7b49dbe4600-istio-token\") pod \"b15a372b-64ad-449b-8f2c-c7b49dbe4600\" (UID: \"b15a372b-64ad-449b-8f2c-c7b49dbe4600\") " Oct 30 19:01:54.721713 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.721662 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b15a372b-64ad-449b-8f2c-c7b49dbe4600-kube-api-access-tvgsk\" (UniqueName: \"kubernetes.io/projected/b15a372b-64ad-449b-8f2c-c7b49dbe4600-kube-api-access-tvgsk\") pod \"b15a372b-64ad-449b-8f2c-c7b49dbe4600\" (UID: \"b15a372b-64ad-449b-8f2c-c7b49dbe4600\") " Oct 30 19:01:54.721935 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.721709 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b15a372b-64ad-449b-8f2c-c7b49dbe4600-istio-data" pod "b15a372b-64ad-449b-8f2c-c7b49dbe4600" (UID: "b15a372b-64ad-449b-8f2c-c7b49dbe4600"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:01:54.721935 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.721802 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b15a372b-64ad-449b-8f2c-c7b49dbe4600-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b15a372b-64ad-449b-8f2c-c7b49dbe4600-istio-podinfo\") pod \"b15a372b-64ad-449b-8f2c-c7b49dbe4600\" (UID: \"b15a372b-64ad-449b-8f2c-c7b49dbe4600\") " Oct 30 19:01:54.721935 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.721855 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b15a372b-64ad-449b-8f2c-c7b49dbe4600-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b15a372b-64ad-449b-8f2c-c7b49dbe4600-istio-envoy\") pod \"b15a372b-64ad-449b-8f2c-c7b49dbe4600\" (UID: \"b15a372b-64ad-449b-8f2c-c7b49dbe4600\") " Oct 30 19:01:54.721935 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.721911 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b15a372b-64ad-449b-8f2c-c7b49dbe4600-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b15a372b-64ad-449b-8f2c-c7b49dbe4600-workload-socket\") pod \"b15a372b-64ad-449b-8f2c-c7b49dbe4600\" (UID: \"b15a372b-64ad-449b-8f2c-c7b49dbe4600\") " Oct 30 19:01:54.722212 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.721944 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b15a372b-64ad-449b-8f2c-c7b49dbe4600-workload-certs" pod "b15a372b-64ad-449b-8f2c-c7b49dbe4600" (UID: "b15a372b-64ad-449b-8f2c-c7b49dbe4600"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:01:54.722212 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.722004 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b15a372b-64ad-449b-8f2c-c7b49dbe4600-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b15a372b-64ad-449b-8f2c-c7b49dbe4600-credential-socket\") pod \"b15a372b-64ad-449b-8f2c-c7b49dbe4600\" (UID: \"b15a372b-64ad-449b-8f2c-c7b49dbe4600\") " Oct 30 19:01:54.722212 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.722069 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b15a372b-64ad-449b-8f2c-c7b49dbe4600-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b15a372b-64ad-449b-8f2c-c7b49dbe4600-istiod-ca-cert\") pod \"b15a372b-64ad-449b-8f2c-c7b49dbe4600\" (UID: \"b15a372b-64ad-449b-8f2c-c7b49dbe4600\") " Oct 30 19:01:54.723014 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.722833 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knd28\" (UniqueName: \"kubernetes.io/projected/dd895ac2-c8b9-47df-93cd-0da009ab4c1e-kube-api-access-knd28\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:01:54.723014 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.722858 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b15a372b-64ad-449b-8f2c-c7b49dbe4600-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:01:54.723014 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.722872 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b15a372b-64ad-449b-8f2c-c7b49dbe4600-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:01:54.723014 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.722884 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd895ac2-c8b9-47df-93cd-0da009ab4c1e-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:01:54.723510 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.723368 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b15a372b-64ad-449b-8f2c-c7b49dbe4600-workload-socket" pod "b15a372b-64ad-449b-8f2c-c7b49dbe4600" (UID: "b15a372b-64ad-449b-8f2c-c7b49dbe4600"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:01:54.723510 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.723428 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b15a372b-64ad-449b-8f2c-c7b49dbe4600-credential-socket" pod "b15a372b-64ad-449b-8f2c-c7b49dbe4600" (UID: "b15a372b-64ad-449b-8f2c-c7b49dbe4600"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:01:54.723648 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.723626 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b15a372b-64ad-449b-8f2c-c7b49dbe4600-istiod-ca-cert" pod "b15a372b-64ad-449b-8f2c-c7b49dbe4600" (UID: "b15a372b-64ad-449b-8f2c-c7b49dbe4600"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:01:54.726544 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.726303 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b15a372b-64ad-449b-8f2c-c7b49dbe4600-istio-envoy" pod "b15a372b-64ad-449b-8f2c-c7b49dbe4600" (UID: "b15a372b-64ad-449b-8f2c-c7b49dbe4600"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:01:54.727328 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.727303 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b15a372b-64ad-449b-8f2c-c7b49dbe4600-kube-api-access-tvgsk" pod "b15a372b-64ad-449b-8f2c-c7b49dbe4600" (UID: "b15a372b-64ad-449b-8f2c-c7b49dbe4600"). InnerVolumeSpecName "kube-api-access-tvgsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:01:54.727900 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.727658 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b15a372b-64ad-449b-8f2c-c7b49dbe4600-istio-token" pod "b15a372b-64ad-449b-8f2c-c7b49dbe4600" (UID: "b15a372b-64ad-449b-8f2c-c7b49dbe4600"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:01:54.728112 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.728077 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b15a372b-64ad-449b-8f2c-c7b49dbe4600-istio-podinfo" pod "b15a372b-64ad-449b-8f2c-c7b49dbe4600" (UID: "b15a372b-64ad-449b-8f2c-c7b49dbe4600"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:01:54.824752 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.824711 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b15a372b-64ad-449b-8f2c-c7b49dbe4600-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:01:54.824752 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.824753 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvgsk\" (UniqueName: \"kubernetes.io/projected/b15a372b-64ad-449b-8f2c-c7b49dbe4600-kube-api-access-tvgsk\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:01:54.824973 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.824769 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b15a372b-64ad-449b-8f2c-c7b49dbe4600-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:01:54.824973 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.824784 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b15a372b-64ad-449b-8f2c-c7b49dbe4600-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:01:54.824973 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.824800 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b15a372b-64ad-449b-8f2c-c7b49dbe4600-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:01:54.824973 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.824814 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b15a372b-64ad-449b-8f2c-c7b49dbe4600-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:01:54.824973 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.824827 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b15a372b-64ad-449b-8f2c-c7b49dbe4600-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:01:54.880933 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.880899 4442 generic.go:356] "Generic (PLEG): container finished" podID="dd895ac2-c8b9-47df-93cd-0da009ab4c1e" containerID="9d3c41c9902d6e1db4462b26df0b49780580fcd329c3006f0c5a0c642347f813" exitCode=0 Oct 30 19:01:54.881168 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.880966 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-551d0b-6686dd876d-qvsgq" Oct 30 19:01:54.881168 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.880990 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-551d0b-6686dd876d-qvsgq" event={"ID":"dd895ac2-c8b9-47df-93cd-0da009ab4c1e","Type":"ContainerDied","Data":"9d3c41c9902d6e1db4462b26df0b49780580fcd329c3006f0c5a0c642347f813"} Oct 30 19:01:54.881168 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.881060 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-551d0b-6686dd876d-qvsgq" event={"ID":"dd895ac2-c8b9-47df-93cd-0da009ab4c1e","Type":"ContainerDied","Data":"f8e2260a4bbbba6017e612339d0501802b3937409527a5e28742e5df1ce6d43c"} Oct 30 19:01:54.881168 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.881089 4442 scope.go:122] "RemoveContainer" containerID="9d3c41c9902d6e1db4462b26df0b49780580fcd329c3006f0c5a0c642347f813" Oct 30 19:01:54.883165 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.883101 4442 generic.go:356] "Generic (PLEG): container finished" podID="b15a372b-64ad-449b-8f2c-c7b49dbe4600" containerID="20151f26f52bf7959f6411cb8f2ba1b8ad0ba6f5579d1969b47e1e4b272d859d" exitCode=0 Oct 30 19:01:54.883300 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.883186 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" Oct 30 19:01:54.883300 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.883184 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" event={"ID":"b15a372b-64ad-449b-8f2c-c7b49dbe4600","Type":"ContainerDied","Data":"20151f26f52bf7959f6411cb8f2ba1b8ad0ba6f5579d1969b47e1e4b272d859d"} Oct 30 19:01:54.883407 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.883317 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm" event={"ID":"b15a372b-64ad-449b-8f2c-c7b49dbe4600","Type":"ContainerDied","Data":"2181dd21bc42987f7b21f70af607a02e0a1340e4362bd3360c6c8b2c9c13b256"} Oct 30 19:01:54.908450 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.908423 4442 scope.go:122] "RemoveContainer" containerID="9d3c41c9902d6e1db4462b26df0b49780580fcd329c3006f0c5a0c642347f813" Oct 30 19:01:54.908855 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:01:54.908821 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d3c41c9902d6e1db4462b26df0b49780580fcd329c3006f0c5a0c642347f813\": container with ID starting with 9d3c41c9902d6e1db4462b26df0b49780580fcd329c3006f0c5a0c642347f813 not found: ID does not exist" containerID="9d3c41c9902d6e1db4462b26df0b49780580fcd329c3006f0c5a0c642347f813" Oct 30 19:01:54.908920 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.908857 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d3c41c9902d6e1db4462b26df0b49780580fcd329c3006f0c5a0c642347f813"} err="failed to get container status \"9d3c41c9902d6e1db4462b26df0b49780580fcd329c3006f0c5a0c642347f813\": rpc error: code = NotFound desc = could not find container \"9d3c41c9902d6e1db4462b26df0b49780580fcd329c3006f0c5a0c642347f813\": container with ID starting with 9d3c41c9902d6e1db4462b26df0b49780580fcd329c3006f0c5a0c642347f813 not found: ID does not exist" Oct 30 19:01:54.908920 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.908880 4442 scope.go:122] "RemoveContainer" containerID="20151f26f52bf7959f6411cb8f2ba1b8ad0ba6f5579d1969b47e1e4b272d859d" Oct 30 19:01:54.921878 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.921855 4442 scope.go:122] "RemoveContainer" containerID="2f400ecb7f14bb45abbd1106184a7c4b1b381edcdc0307e7d9755af418f0e4cd" Oct 30 19:01:54.933364 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.933342 4442 scope.go:122] "RemoveContainer" containerID="20151f26f52bf7959f6411cb8f2ba1b8ad0ba6f5579d1969b47e1e4b272d859d" Oct 30 19:01:54.933716 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:01:54.933697 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20151f26f52bf7959f6411cb8f2ba1b8ad0ba6f5579d1969b47e1e4b272d859d\": container with ID starting with 20151f26f52bf7959f6411cb8f2ba1b8ad0ba6f5579d1969b47e1e4b272d859d not found: ID does not exist" containerID="20151f26f52bf7959f6411cb8f2ba1b8ad0ba6f5579d1969b47e1e4b272d859d" Oct 30 19:01:54.933773 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.933728 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20151f26f52bf7959f6411cb8f2ba1b8ad0ba6f5579d1969b47e1e4b272d859d"} err="failed to get container status \"20151f26f52bf7959f6411cb8f2ba1b8ad0ba6f5579d1969b47e1e4b272d859d\": rpc error: code = NotFound desc = could not find container \"20151f26f52bf7959f6411cb8f2ba1b8ad0ba6f5579d1969b47e1e4b272d859d\": container with ID starting with 20151f26f52bf7959f6411cb8f2ba1b8ad0ba6f5579d1969b47e1e4b272d859d not found: ID does not exist" Oct 30 19:01:54.933773 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.933747 4442 scope.go:122] "RemoveContainer" containerID="2f400ecb7f14bb45abbd1106184a7c4b1b381edcdc0307e7d9755af418f0e4cd" Oct 30 19:01:54.934034 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:01:54.934010 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f400ecb7f14bb45abbd1106184a7c4b1b381edcdc0307e7d9755af418f0e4cd\": container with ID starting with 2f400ecb7f14bb45abbd1106184a7c4b1b381edcdc0307e7d9755af418f0e4cd not found: ID does not exist" containerID="2f400ecb7f14bb45abbd1106184a7c4b1b381edcdc0307e7d9755af418f0e4cd" Oct 30 19:01:54.934088 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.934046 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f400ecb7f14bb45abbd1106184a7c4b1b381edcdc0307e7d9755af418f0e4cd"} err="failed to get container status \"2f400ecb7f14bb45abbd1106184a7c4b1b381edcdc0307e7d9755af418f0e4cd\": rpc error: code = NotFound desc = could not find container \"2f400ecb7f14bb45abbd1106184a7c4b1b381edcdc0307e7d9755af418f0e4cd\": container with ID starting with 2f400ecb7f14bb45abbd1106184a7c4b1b381edcdc0307e7d9755af418f0e4cd not found: ID does not exist" Oct 30 19:01:54.968043 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.968011 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-551d0b-6686dd876d-qvsgq"] Oct 30 19:01:54.990898 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:54.990787 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-551d0b-6686dd876d-qvsgq"] Oct 30 19:01:55.031540 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:55.031512 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm"] Oct 30 19:01:55.037796 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:55.037774 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-7kxhm"] Oct 30 19:01:55.067766 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:55.067733 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b15a372b-64ad-449b-8f2c-c7b49dbe4600" path="/var/lib/kubelet/pods/b15a372b-64ad-449b-8f2c-c7b49dbe4600/volumes" Oct 30 19:01:55.068377 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:01:55.068361 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd895ac2-c8b9-47df-93cd-0da009ab4c1e" path="/var/lib/kubelet/pods/dd895ac2-c8b9-47df-93cd-0da009ab4c1e/volumes" Oct 30 19:02:03.242510 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:03.242472 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eed264df-wdcbd"] Oct 30 19:02:03.243839 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:03.243817 4442 cpu_manager.go:414] "containerMap: removing container" podUID="b15a372b-64ad-449b-8f2c-c7b49dbe4600" containerName="istio-proxy" Oct 30 19:02:03.243893 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:03.243843 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b15a372b-64ad-449b-8f2c-c7b49dbe4600" containerName="istio-proxy" Oct 30 19:02:03.243893 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:03.243862 4442 cpu_manager.go:414] "containerMap: removing container" podUID="b15a372b-64ad-449b-8f2c-c7b49dbe4600" containerName="istio-proxy" Oct 30 19:02:03.243893 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:03.243870 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b15a372b-64ad-449b-8f2c-c7b49dbe4600" containerName="istio-proxy" Oct 30 19:02:03.243893 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:03.243884 4442 cpu_manager.go:414] "containerMap: removing container" podUID="cf20e69b-9704-4362-8906-cadb6ab06fde" containerName="revoke-keys" Oct 30 19:02:03.244028 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:03.243893 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf20e69b-9704-4362-8906-cadb6ab06fde" containerName="revoke-keys" Oct 30 19:02:03.244028 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:03.243911 4442 cpu_manager.go:414] "containerMap: removing container" podUID="dd895ac2-c8b9-47df-93cd-0da009ab4c1e" containerName="payload-processing" Oct 30 19:02:03.244028 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:03.243920 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd895ac2-c8b9-47df-93cd-0da009ab4c1e" containerName="payload-processing" Oct 30 19:02:03.244192 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:03.244176 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd895ac2-c8b9-47df-93cd-0da009ab4c1e" containerName="payload-processing" Oct 30 19:02:03.244237 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:03.244223 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="b15a372b-64ad-449b-8f2c-c7b49dbe4600" containerName="istio-proxy" Oct 30 19:02:03.244272 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:03.244236 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="b15a372b-64ad-449b-8f2c-c7b49dbe4600" containerName="istio-proxy" Oct 30 19:02:03.244272 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:03.244248 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf20e69b-9704-4362-8906-cadb6ab06fde" containerName="revoke-keys" Oct 30 19:02:03.249028 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:03.249004 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eed264df-wdcbd" Oct 30 19:02:03.269353 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:03.269322 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eed264df-wdcbd"] Oct 30 19:02:03.272941 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:03.272916 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:02:03.273317 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:03.273281 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:02:03.390384 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:03.390351 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eed264df-wdcbd\" (UID: \"3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eed264df-wdcbd" Oct 30 19:02:03.390570 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:03.390545 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eed264df-wdcbd\" (UID: \"3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eed264df-wdcbd" Oct 30 19:02:03.390662 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:03.390620 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbbvn\" (UniqueName: \"kubernetes.io/projected/3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764-kube-api-access-sbbvn\") pod \"maas-api-revoke-sub-eed264df-wdcbd\" (UID: \"3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eed264df-wdcbd" Oct 30 19:02:03.491869 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:03.491835 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eed264df-wdcbd\" (UID: \"3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eed264df-wdcbd" Oct 30 19:02:03.492083 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:03.491908 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbbvn\" (UniqueName: \"kubernetes.io/projected/3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764-kube-api-access-sbbvn\") pod \"maas-api-revoke-sub-eed264df-wdcbd\" (UID: \"3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eed264df-wdcbd" Oct 30 19:02:03.492245 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:03.492224 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eed264df-wdcbd\" (UID: \"3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eed264df-wdcbd" Oct 30 19:02:03.492704 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:03.492643 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eed264df-wdcbd\" (UID: \"3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eed264df-wdcbd" Oct 30 19:02:03.494735 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:03.494708 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eed264df-wdcbd\" (UID: \"3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eed264df-wdcbd" Oct 30 19:02:03.513029 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:03.513000 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbbvn\" (UniqueName: \"kubernetes.io/projected/3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764-kube-api-access-sbbvn\") pod \"maas-api-revoke-sub-eed264df-wdcbd\" (UID: \"3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eed264df-wdcbd" Oct 30 19:02:03.570406 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:03.570365 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eed264df-wdcbd" Oct 30 19:02:03.897072 ip-10-0-138-176 kubenswrapper[4442]: W1030 19:02:03.897045 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b7c6bd3_a04b_44c0_bd5c_24e3dcea6764.slice/crio-4d049ac0d65cc80e2cbd7afe1b1ff143149bdc24a53e857de089b1cd350367b5 WatchSource:0}: Error finding container 4d049ac0d65cc80e2cbd7afe1b1ff143149bdc24a53e857de089b1cd350367b5: Status 404 returned error can't find the container with id 4d049ac0d65cc80e2cbd7afe1b1ff143149bdc24a53e857de089b1cd350367b5 Oct 30 19:02:03.898284 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:03.898258 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eed264df-wdcbd"] Oct 30 19:02:04.010941 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:04.010901 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eed264df-wdcbd" event={"ID":"3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764","Type":"ContainerStarted","Data":"4d049ac0d65cc80e2cbd7afe1b1ff143149bdc24a53e857de089b1cd350367b5"} Oct 30 19:02:05.061429 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:05.061389 4442 generic.go:356] "Generic (PLEG): container finished" podID="3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764" containerID="d4f902c2ec0659eddeccc6c758cbac1304222baffd6508ab829a4f374f5b7111" exitCode=0 Oct 30 19:02:05.070625 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:05.070594 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eed264df-wdcbd" event={"ID":"3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764","Type":"ContainerDied","Data":"d4f902c2ec0659eddeccc6c758cbac1304222baffd6508ab829a4f374f5b7111"} Oct 30 19:02:06.249749 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:06.249713 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eed264df-wdcbd" Oct 30 19:02:06.374898 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:06.374863 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764-kube-api-access-sbbvn\" (UniqueName: \"kubernetes.io/projected/3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764-kube-api-access-sbbvn\") pod \"3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764\" (UID: \"3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764\") " Oct 30 19:02:06.374898 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:06.374922 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764-maas-api-xks-ca\") pod \"3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764\" (UID: \"3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764\") " Oct 30 19:02:06.375250 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:06.375077 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764-maas-api-openshift-ca\") pod \"3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764\" (UID: \"3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764\") " Oct 30 19:02:06.375514 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:06.375480 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764-maas-api-openshift-ca" pod "3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764" (UID: "3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:02:06.375844 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:06.375828 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764-maas-api-openshift-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:02:06.377400 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:06.377375 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764-maas-api-xks-ca" pod "3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764" (UID: "3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:02:06.377502 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:06.377459 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764-kube-api-access-sbbvn" pod "3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764" (UID: "3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764"). InnerVolumeSpecName "kube-api-access-sbbvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:02:06.477404 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:06.477373 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbbvn\" (UniqueName: \"kubernetes.io/projected/3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764-kube-api-access-sbbvn\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:02:06.477404 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:06.477399 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764-maas-api-xks-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:02:07.107737 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:07.107705 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eed264df-wdcbd" Oct 30 19:02:07.107737 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:07.107717 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eed264df-wdcbd" event={"ID":"3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764","Type":"ContainerDied","Data":"4d049ac0d65cc80e2cbd7afe1b1ff143149bdc24a53e857de089b1cd350367b5"} Oct 30 19:02:07.107970 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:07.107820 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d049ac0d65cc80e2cbd7afe1b1ff143149bdc24a53e857de089b1cd350367b5" Oct 30 19:02:17.565697 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:02:17.565661 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 19:02:30.730936 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:30.730896 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-551d0b-8c8jd"] Oct 30 19:02:30.738908 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:30.738883 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-551d0b-8c8jd"] Oct 30 19:02:31.070733 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:31.070643 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2014b449-aa26-47e8-bb93-893588bc6083" path="/var/lib/kubelet/pods/2014b449-aa26-47e8-bb93-893588bc6083/volumes" Oct 30 19:02:38.212723 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:38.212687 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9"] Oct 30 19:02:38.213250 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:38.212961 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" podUID="553c4b32-dcd3-494b-9cfc-5014c97c1543" containerName="istio-proxy" containerID="cri-o://715472ff24cafa7875efa47b7a4f983a1354fb73767e9e4ee3d41dd4eff6f75a" gracePeriod=30 Oct 30 19:02:43.622305 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:43.621957 4442 generic.go:356] "Generic (PLEG): container finished" podID="553c4b32-dcd3-494b-9cfc-5014c97c1543" containerID="715472ff24cafa7875efa47b7a4f983a1354fb73767e9e4ee3d41dd4eff6f75a" exitCode=0 Oct 30 19:02:43.622305 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:43.622224 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" event={"ID":"553c4b32-dcd3-494b-9cfc-5014c97c1543","Type":"ContainerDied","Data":"715472ff24cafa7875efa47b7a4f983a1354fb73767e9e4ee3d41dd4eff6f75a"} Oct 30 19:02:44.164013 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:44.163633 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" Oct 30 19:02:44.314819 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:44.314764 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/553c4b32-dcd3-494b-9cfc-5014c97c1543-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/553c4b32-dcd3-494b-9cfc-5014c97c1543-workload-socket\") pod \"553c4b32-dcd3-494b-9cfc-5014c97c1543\" (UID: \"553c4b32-dcd3-494b-9cfc-5014c97c1543\") " Oct 30 19:02:44.315025 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:44.314951 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/553c4b32-dcd3-494b-9cfc-5014c97c1543-istio-token\" (UniqueName: \"kubernetes.io/projected/553c4b32-dcd3-494b-9cfc-5014c97c1543-istio-token\") pod \"553c4b32-dcd3-494b-9cfc-5014c97c1543\" (UID: \"553c4b32-dcd3-494b-9cfc-5014c97c1543\") " Oct 30 19:02:44.315025 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:44.314988 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/553c4b32-dcd3-494b-9cfc-5014c97c1543-kube-api-access-gnv5t\" (UniqueName: \"kubernetes.io/projected/553c4b32-dcd3-494b-9cfc-5014c97c1543-kube-api-access-gnv5t\") pod \"553c4b32-dcd3-494b-9cfc-5014c97c1543\" (UID: \"553c4b32-dcd3-494b-9cfc-5014c97c1543\") " Oct 30 19:02:44.315190 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:44.315035 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/553c4b32-dcd3-494b-9cfc-5014c97c1543-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/553c4b32-dcd3-494b-9cfc-5014c97c1543-istiod-ca-cert\") pod \"553c4b32-dcd3-494b-9cfc-5014c97c1543\" (UID: \"553c4b32-dcd3-494b-9cfc-5014c97c1543\") " Oct 30 19:02:44.315190 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:44.315070 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/553c4b32-dcd3-494b-9cfc-5014c97c1543-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/553c4b32-dcd3-494b-9cfc-5014c97c1543-istio-podinfo\") pod \"553c4b32-dcd3-494b-9cfc-5014c97c1543\" (UID: \"553c4b32-dcd3-494b-9cfc-5014c97c1543\") " Oct 30 19:02:44.315190 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:44.315102 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/553c4b32-dcd3-494b-9cfc-5014c97c1543-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/553c4b32-dcd3-494b-9cfc-5014c97c1543-credential-socket\") pod \"553c4b32-dcd3-494b-9cfc-5014c97c1543\" (UID: \"553c4b32-dcd3-494b-9cfc-5014c97c1543\") " Oct 30 19:02:44.315190 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:44.315179 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/553c4b32-dcd3-494b-9cfc-5014c97c1543-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/553c4b32-dcd3-494b-9cfc-5014c97c1543-workload-certs\") pod \"553c4b32-dcd3-494b-9cfc-5014c97c1543\" (UID: \"553c4b32-dcd3-494b-9cfc-5014c97c1543\") " Oct 30 19:02:44.315495 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:44.315367 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/553c4b32-dcd3-494b-9cfc-5014c97c1543-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/553c4b32-dcd3-494b-9cfc-5014c97c1543-istio-envoy\") pod \"553c4b32-dcd3-494b-9cfc-5014c97c1543\" (UID: \"553c4b32-dcd3-494b-9cfc-5014c97c1543\") " Oct 30 19:02:44.315495 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:44.315460 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/553c4b32-dcd3-494b-9cfc-5014c97c1543-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/553c4b32-dcd3-494b-9cfc-5014c97c1543-istio-data\") pod \"553c4b32-dcd3-494b-9cfc-5014c97c1543\" (UID: \"553c4b32-dcd3-494b-9cfc-5014c97c1543\") " Oct 30 19:02:44.315748 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:44.315718 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/553c4b32-dcd3-494b-9cfc-5014c97c1543-workload-socket" pod "553c4b32-dcd3-494b-9cfc-5014c97c1543" (UID: "553c4b32-dcd3-494b-9cfc-5014c97c1543"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:02:44.316090 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:44.316063 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/553c4b32-dcd3-494b-9cfc-5014c97c1543-istio-data" pod "553c4b32-dcd3-494b-9cfc-5014c97c1543" (UID: "553c4b32-dcd3-494b-9cfc-5014c97c1543"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:02:44.316224 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:44.316075 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/553c4b32-dcd3-494b-9cfc-5014c97c1543-istiod-ca-cert" pod "553c4b32-dcd3-494b-9cfc-5014c97c1543" (UID: "553c4b32-dcd3-494b-9cfc-5014c97c1543"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:02:44.316322 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:44.316299 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/553c4b32-dcd3-494b-9cfc-5014c97c1543-workload-certs" pod "553c4b32-dcd3-494b-9cfc-5014c97c1543" (UID: "553c4b32-dcd3-494b-9cfc-5014c97c1543"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:02:44.316718 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:44.316696 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/553c4b32-dcd3-494b-9cfc-5014c97c1543-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:02:44.316849 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:44.316834 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/553c4b32-dcd3-494b-9cfc-5014c97c1543-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:02:44.316967 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:44.316938 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/553c4b32-dcd3-494b-9cfc-5014c97c1543-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:02:44.317064 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:44.317053 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/553c4b32-dcd3-494b-9cfc-5014c97c1543-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:02:44.317181 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:44.317123 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/553c4b32-dcd3-494b-9cfc-5014c97c1543-credential-socket" pod "553c4b32-dcd3-494b-9cfc-5014c97c1543" (UID: "553c4b32-dcd3-494b-9cfc-5014c97c1543"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:02:44.320341 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:44.319397 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/553c4b32-dcd3-494b-9cfc-5014c97c1543-kube-api-access-gnv5t" pod "553c4b32-dcd3-494b-9cfc-5014c97c1543" (UID: "553c4b32-dcd3-494b-9cfc-5014c97c1543"). InnerVolumeSpecName "kube-api-access-gnv5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:02:44.320341 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:44.319494 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/553c4b32-dcd3-494b-9cfc-5014c97c1543-istio-token" pod "553c4b32-dcd3-494b-9cfc-5014c97c1543" (UID: "553c4b32-dcd3-494b-9cfc-5014c97c1543"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:02:44.320341 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:44.319520 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/553c4b32-dcd3-494b-9cfc-5014c97c1543-istio-podinfo" pod "553c4b32-dcd3-494b-9cfc-5014c97c1543" (UID: "553c4b32-dcd3-494b-9cfc-5014c97c1543"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:02:44.321070 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:44.321044 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/553c4b32-dcd3-494b-9cfc-5014c97c1543-istio-envoy" pod "553c4b32-dcd3-494b-9cfc-5014c97c1543" (UID: "553c4b32-dcd3-494b-9cfc-5014c97c1543"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:02:44.418279 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:44.418249 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/553c4b32-dcd3-494b-9cfc-5014c97c1543-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:02:44.418558 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:44.418546 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnv5t\" (UniqueName: \"kubernetes.io/projected/553c4b32-dcd3-494b-9cfc-5014c97c1543-kube-api-access-gnv5t\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:02:44.418632 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:44.418624 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/553c4b32-dcd3-494b-9cfc-5014c97c1543-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:02:44.418691 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:44.418683 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/553c4b32-dcd3-494b-9cfc-5014c97c1543-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:02:44.418766 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:44.418755 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/553c4b32-dcd3-494b-9cfc-5014c97c1543-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:02:44.713432 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:44.713403 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" Oct 30 19:02:44.716909 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:44.713590 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9" event={"ID":"553c4b32-dcd3-494b-9cfc-5014c97c1543","Type":"ContainerDied","Data":"de5eff019c3a5ebe3d86d33844c535d6a0ffc18b3a758cc46a6d30f29250d555"} Oct 30 19:02:44.717310 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:44.717289 4442 scope.go:122] "RemoveContainer" containerID="715472ff24cafa7875efa47b7a4f983a1354fb73767e9e4ee3d41dd4eff6f75a" Oct 30 19:02:44.787287 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:44.787229 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9"] Oct 30 19:02:44.795610 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:44.795404 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-62sf9"] Oct 30 19:02:45.066724 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:45.066635 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="553c4b32-dcd3-494b-9cfc-5014c97c1543" path="/var/lib/kubelet/pods/553c4b32-dcd3-494b-9cfc-5014c97c1543/volumes" Oct 30 19:02:47.560249 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:02:47.560216 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 19:02:58.359958 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:58.359921 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f37ca7-xdc9j"] Oct 30 19:02:58.369750 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:58.369717 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f37ca7-xdc9j"] Oct 30 19:02:59.072770 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:02:59.072733 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf20e69b-9704-4362-8906-cadb6ab06fde" path="/var/lib/kubelet/pods/cf20e69b-9704-4362-8906-cadb6ab06fde/volumes" Oct 30 19:03:05.102749 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:03:05.102705 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd"] Oct 30 19:03:05.103243 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:03:05.102966 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" podUID="6d8d84cf-1334-4038-8b24-8cbec03e6882" containerName="istio-proxy" containerID="cri-o://a7bb5aa4b35ee7536b17eac0e1f9d3bccf57f8b470396a4befb164ad7965c24e" gracePeriod=30 Oct 30 19:03:10.407992 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:03:10.407963 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" Oct 30 19:03:10.440540 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:03:10.440510 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d8d84cf-1334-4038-8b24-8cbec03e6882-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d8d84cf-1334-4038-8b24-8cbec03e6882-workload-socket\") pod \"6d8d84cf-1334-4038-8b24-8cbec03e6882\" (UID: \"6d8d84cf-1334-4038-8b24-8cbec03e6882\") " Oct 30 19:03:10.440717 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:03:10.440634 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d8d84cf-1334-4038-8b24-8cbec03e6882-istio-token\" (UniqueName: \"kubernetes.io/projected/6d8d84cf-1334-4038-8b24-8cbec03e6882-istio-token\") pod \"6d8d84cf-1334-4038-8b24-8cbec03e6882\" (UID: \"6d8d84cf-1334-4038-8b24-8cbec03e6882\") " Oct 30 19:03:10.440791 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:03:10.440755 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d8d84cf-1334-4038-8b24-8cbec03e6882-workload-socket" pod "6d8d84cf-1334-4038-8b24-8cbec03e6882" (UID: "6d8d84cf-1334-4038-8b24-8cbec03e6882"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:03:10.440791 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:03:10.440774 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d8d84cf-1334-4038-8b24-8cbec03e6882-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d8d84cf-1334-4038-8b24-8cbec03e6882-workload-certs\") pod \"6d8d84cf-1334-4038-8b24-8cbec03e6882\" (UID: \"6d8d84cf-1334-4038-8b24-8cbec03e6882\") " Oct 30 19:03:10.440904 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:03:10.440814 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d8d84cf-1334-4038-8b24-8cbec03e6882-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d8d84cf-1334-4038-8b24-8cbec03e6882-istio-data\") pod \"6d8d84cf-1334-4038-8b24-8cbec03e6882\" (UID: \"6d8d84cf-1334-4038-8b24-8cbec03e6882\") " Oct 30 19:03:10.440960 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:03:10.440929 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d8d84cf-1334-4038-8b24-8cbec03e6882-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d8d84cf-1334-4038-8b24-8cbec03e6882-credential-socket\") pod \"6d8d84cf-1334-4038-8b24-8cbec03e6882\" (UID: \"6d8d84cf-1334-4038-8b24-8cbec03e6882\") " Oct 30 19:03:10.441018 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:03:10.440985 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d8d84cf-1334-4038-8b24-8cbec03e6882-workload-certs" pod "6d8d84cf-1334-4038-8b24-8cbec03e6882" (UID: "6d8d84cf-1334-4038-8b24-8cbec03e6882"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:03:10.441082 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:03:10.441024 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d8d84cf-1334-4038-8b24-8cbec03e6882-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d8d84cf-1334-4038-8b24-8cbec03e6882-istiod-ca-cert\") pod \"6d8d84cf-1334-4038-8b24-8cbec03e6882\" (UID: \"6d8d84cf-1334-4038-8b24-8cbec03e6882\") " Oct 30 19:03:10.441082 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:03:10.441072 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d8d84cf-1334-4038-8b24-8cbec03e6882-kube-api-access-pdhjc\" (UniqueName: \"kubernetes.io/projected/6d8d84cf-1334-4038-8b24-8cbec03e6882-kube-api-access-pdhjc\") pod \"6d8d84cf-1334-4038-8b24-8cbec03e6882\" (UID: \"6d8d84cf-1334-4038-8b24-8cbec03e6882\") " Oct 30 19:03:10.441212 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:03:10.441105 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d8d84cf-1334-4038-8b24-8cbec03e6882-istio-data" pod "6d8d84cf-1334-4038-8b24-8cbec03e6882" (UID: "6d8d84cf-1334-4038-8b24-8cbec03e6882"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:03:10.441212 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:03:10.441198 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d8d84cf-1334-4038-8b24-8cbec03e6882-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d8d84cf-1334-4038-8b24-8cbec03e6882-istio-envoy\") pod \"6d8d84cf-1334-4038-8b24-8cbec03e6882\" (UID: \"6d8d84cf-1334-4038-8b24-8cbec03e6882\") " Oct 30 19:03:10.441331 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:03:10.441229 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d8d84cf-1334-4038-8b24-8cbec03e6882-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d8d84cf-1334-4038-8b24-8cbec03e6882-istio-podinfo\") pod \"6d8d84cf-1334-4038-8b24-8cbec03e6882\" (UID: \"6d8d84cf-1334-4038-8b24-8cbec03e6882\") " Oct 30 19:03:10.442022 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:03:10.441230 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d8d84cf-1334-4038-8b24-8cbec03e6882-credential-socket" pod "6d8d84cf-1334-4038-8b24-8cbec03e6882" (UID: "6d8d84cf-1334-4038-8b24-8cbec03e6882"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:03:10.442022 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:03:10.441949 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d8d84cf-1334-4038-8b24-8cbec03e6882-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:03:10.442022 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:03:10.441970 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d8d84cf-1334-4038-8b24-8cbec03e6882-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:03:10.442022 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:03:10.441984 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d8d84cf-1334-4038-8b24-8cbec03e6882-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:03:10.442022 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:03:10.441998 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d8d84cf-1334-4038-8b24-8cbec03e6882-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:03:10.442559 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:03:10.442532 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d8d84cf-1334-4038-8b24-8cbec03e6882-istiod-ca-cert" pod "6d8d84cf-1334-4038-8b24-8cbec03e6882" (UID: "6d8d84cf-1334-4038-8b24-8cbec03e6882"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:03:10.443743 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:03:10.443714 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d8d84cf-1334-4038-8b24-8cbec03e6882-kube-api-access-pdhjc" pod "6d8d84cf-1334-4038-8b24-8cbec03e6882" (UID: "6d8d84cf-1334-4038-8b24-8cbec03e6882"). InnerVolumeSpecName "kube-api-access-pdhjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:03:10.444090 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:03:10.444062 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d8d84cf-1334-4038-8b24-8cbec03e6882-istio-envoy" pod "6d8d84cf-1334-4038-8b24-8cbec03e6882" (UID: "6d8d84cf-1334-4038-8b24-8cbec03e6882"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:03:10.444358 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:03:10.444336 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d8d84cf-1334-4038-8b24-8cbec03e6882-istio-podinfo" pod "6d8d84cf-1334-4038-8b24-8cbec03e6882" (UID: "6d8d84cf-1334-4038-8b24-8cbec03e6882"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:03:10.444435 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:03:10.444403 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d8d84cf-1334-4038-8b24-8cbec03e6882-istio-token" pod "6d8d84cf-1334-4038-8b24-8cbec03e6882" (UID: "6d8d84cf-1334-4038-8b24-8cbec03e6882"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:03:10.543865 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:03:10.543834 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d8d84cf-1334-4038-8b24-8cbec03e6882-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:03:10.543865 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:03:10.543860 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdhjc\" (UniqueName: \"kubernetes.io/projected/6d8d84cf-1334-4038-8b24-8cbec03e6882-kube-api-access-pdhjc\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:03:10.543865 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:03:10.543870 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d8d84cf-1334-4038-8b24-8cbec03e6882-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:03:10.544106 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:03:10.543879 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d8d84cf-1334-4038-8b24-8cbec03e6882-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:03:10.544106 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:03:10.543887 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d8d84cf-1334-4038-8b24-8cbec03e6882-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:03:11.074107 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:03:11.074069 4442 generic.go:356] "Generic (PLEG): container finished" podID="6d8d84cf-1334-4038-8b24-8cbec03e6882" containerID="a7bb5aa4b35ee7536b17eac0e1f9d3bccf57f8b470396a4befb164ad7965c24e" exitCode=0 Oct 30 19:03:11.074107 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:03:11.074104 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" event={"ID":"6d8d84cf-1334-4038-8b24-8cbec03e6882","Type":"ContainerDied","Data":"a7bb5aa4b35ee7536b17eac0e1f9d3bccf57f8b470396a4befb164ad7965c24e"} Oct 30 19:03:11.074425 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:03:11.074155 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" Oct 30 19:03:11.074425 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:03:11.074212 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd" event={"ID":"6d8d84cf-1334-4038-8b24-8cbec03e6882","Type":"ContainerDied","Data":"814b606c0517cf6a063c1c6095963226d07c5f0954278b162d5034e5d386ecaa"} Oct 30 19:03:11.074425 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:03:11.074244 4442 scope.go:122] "RemoveContainer" containerID="a7bb5aa4b35ee7536b17eac0e1f9d3bccf57f8b470396a4befb164ad7965c24e" Oct 30 19:03:11.094178 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:03:11.094128 4442 scope.go:122] "RemoveContainer" containerID="7ffd7490b3dcbf7b355af0ac00e48d063fa223293713c5445ac06795e4a88324" Oct 30 19:03:11.109888 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:03:11.109864 4442 scope.go:122] "RemoveContainer" containerID="a7bb5aa4b35ee7536b17eac0e1f9d3bccf57f8b470396a4befb164ad7965c24e" Oct 30 19:03:11.110674 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:03:11.110638 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7bb5aa4b35ee7536b17eac0e1f9d3bccf57f8b470396a4befb164ad7965c24e\": container with ID starting with a7bb5aa4b35ee7536b17eac0e1f9d3bccf57f8b470396a4befb164ad7965c24e not found: ID does not exist" containerID="a7bb5aa4b35ee7536b17eac0e1f9d3bccf57f8b470396a4befb164ad7965c24e" Oct 30 19:03:11.110812 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:03:11.110681 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7bb5aa4b35ee7536b17eac0e1f9d3bccf57f8b470396a4befb164ad7965c24e"} err="failed to get container status \"a7bb5aa4b35ee7536b17eac0e1f9d3bccf57f8b470396a4befb164ad7965c24e\": rpc error: code = NotFound desc = could not find container \"a7bb5aa4b35ee7536b17eac0e1f9d3bccf57f8b470396a4befb164ad7965c24e\": container with ID starting with a7bb5aa4b35ee7536b17eac0e1f9d3bccf57f8b470396a4befb164ad7965c24e not found: ID does not exist" Oct 30 19:03:11.110812 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:03:11.110706 4442 scope.go:122] "RemoveContainer" containerID="7ffd7490b3dcbf7b355af0ac00e48d063fa223293713c5445ac06795e4a88324" Oct 30 19:03:11.111054 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:03:11.111024 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ffd7490b3dcbf7b355af0ac00e48d063fa223293713c5445ac06795e4a88324\": container with ID starting with 7ffd7490b3dcbf7b355af0ac00e48d063fa223293713c5445ac06795e4a88324 not found: ID does not exist" containerID="7ffd7490b3dcbf7b355af0ac00e48d063fa223293713c5445ac06795e4a88324" Oct 30 19:03:11.111182 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:03:11.111063 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ffd7490b3dcbf7b355af0ac00e48d063fa223293713c5445ac06795e4a88324"} err="failed to get container status \"7ffd7490b3dcbf7b355af0ac00e48d063fa223293713c5445ac06795e4a88324\": rpc error: code = NotFound desc = could not find container \"7ffd7490b3dcbf7b355af0ac00e48d063fa223293713c5445ac06795e4a88324\": container with ID starting with 7ffd7490b3dcbf7b355af0ac00e48d063fa223293713c5445ac06795e4a88324 not found: ID does not exist" Oct 30 19:03:11.140787 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:03:11.140758 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd"] Oct 30 19:03:11.152150 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:03:11.152104 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-xj7sd"] Oct 30 19:03:13.070364 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:03:13.070334 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d8d84cf-1334-4038-8b24-8cbec03e6882" path="/var/lib/kubelet/pods/6d8d84cf-1334-4038-8b24-8cbec03e6882/volumes" Oct 30 19:03:17.563400 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:03:17.563356 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 19:03:24.189349 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:03:24.189316 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52rn6_516c6b6c-fc40-49cf-93b5-1c4471b02cf0/ovn-acl-logging/0.log" Oct 30 19:03:24.189789 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:03:24.189545 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52rn6_516c6b6c-fc40-49cf-93b5-1c4471b02cf0/ovn-acl-logging/0.log" Oct 30 19:03:47.560912 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:03:47.560876 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 19:04:17.566796 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:04:17.566751 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 19:04:43.145805 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:43.145771 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adc05f24-qlkw6"] Oct 30 19:04:43.202044 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:43.202010 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adc05f24-qlkw6"] Oct 30 19:04:45.068967 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:45.068928 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a54f191-8b3f-4d88-87fb-c84c9e76efe8" path="/var/lib/kubelet/pods/3a54f191-8b3f-4d88-87fb-c84c9e76efe8/volumes" Oct 30 19:04:47.232321 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:47.232273 4442 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-176.ec2.internal" podUID="9e81bd78300430e51b01d30caa1123e7" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded" Oct 30 19:04:47.561994 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:04:47.561898 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 19:04:54.097170 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:54.097094 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7133e51d-zhbt6"] Oct 30 19:04:54.106385 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:54.106358 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7133e51d-zhbt6"] Oct 30 19:04:55.068080 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:55.068050 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68876174-5cb8-4e63-85b2-a37cf5420e8a" path="/var/lib/kubelet/pods/68876174-5cb8-4e63-85b2-a37cf5420e8a/volumes" Oct 30 19:04:58.844870 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:58.844830 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg"] Oct 30 19:04:58.846243 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:58.846217 4442 cpu_manager.go:414] "containerMap: removing container" podUID="6d8d84cf-1334-4038-8b24-8cbec03e6882" containerName="istio-proxy" Oct 30 19:04:58.846243 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:58.846245 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d8d84cf-1334-4038-8b24-8cbec03e6882" containerName="istio-proxy" Oct 30 19:04:58.846433 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:58.846361 4442 cpu_manager.go:414] "containerMap: removing container" podUID="553c4b32-dcd3-494b-9cfc-5014c97c1543" containerName="istio-proxy" Oct 30 19:04:58.846433 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:58.846371 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="553c4b32-dcd3-494b-9cfc-5014c97c1543" containerName="istio-proxy" Oct 30 19:04:58.846433 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:58.846411 4442 cpu_manager.go:414] "containerMap: removing container" podUID="3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764" containerName="revoke-keys" Oct 30 19:04:58.846433 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:58.846418 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764" containerName="revoke-keys" Oct 30 19:04:58.846659 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:58.846627 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="553c4b32-dcd3-494b-9cfc-5014c97c1543" containerName="istio-proxy" Oct 30 19:04:58.846659 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:58.846648 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764" containerName="revoke-keys" Oct 30 19:04:58.846774 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:58.846660 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d8d84cf-1334-4038-8b24-8cbec03e6882" containerName="istio-proxy" Oct 30 19:04:58.846774 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:58.846670 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d8d84cf-1334-4038-8b24-8cbec03e6882" containerName="istio-proxy" Oct 30 19:04:58.851818 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:58.851780 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" Oct 30 19:04:58.873878 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:58.873852 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-15e8827b-openshift-default-dockercfg-zlszd\"" Oct 30 19:04:58.880996 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:58.880051 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg"] Oct 30 19:04:58.964161 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:58.956374 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-workload-certs\") pod \"e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg\" (UID: \"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf\") " pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" Oct 30 19:04:58.964161 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:58.956537 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-istio-data\") pod \"e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg\" (UID: \"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf\") " pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" Oct 30 19:04:58.964161 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:58.956609 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-workload-socket\") pod \"e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg\" (UID: \"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf\") " pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" Oct 30 19:04:58.964161 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:58.956690 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-istio-podinfo\") pod \"e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg\" (UID: \"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf\") " pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" Oct 30 19:04:58.964161 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:58.956846 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-istio-envoy\") pod \"e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg\" (UID: \"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf\") " pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" Oct 30 19:04:58.964161 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:58.956904 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-istiod-ca-cert\") pod \"e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg\" (UID: \"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf\") " pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" Oct 30 19:04:58.964161 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:58.956955 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-istio-token\") pod \"e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg\" (UID: \"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf\") " pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" Oct 30 19:04:58.964161 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:58.956990 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-credential-socket\") pod \"e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg\" (UID: \"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf\") " pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" Oct 30 19:04:58.964161 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:58.957162 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ps2q\" (UniqueName: \"kubernetes.io/projected/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-kube-api-access-5ps2q\") pod \"e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg\" (UID: \"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf\") " pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" Oct 30 19:04:59.106208 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:59.105378 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-workload-certs\") pod \"e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg\" (UID: \"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf\") " pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" Oct 30 19:04:59.119643 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:59.109512 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-istio-data\") pod \"e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg\" (UID: \"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf\") " pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" Oct 30 19:04:59.119643 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:59.109689 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-workload-socket\") pod \"e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg\" (UID: \"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf\") " pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" Oct 30 19:04:59.119643 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:59.109733 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-workload-certs\") pod \"e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg\" (UID: \"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf\") " pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" Oct 30 19:04:59.119643 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:59.110170 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-istio-podinfo\") pod \"e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg\" (UID: \"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf\") " pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" Oct 30 19:04:59.119643 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:59.110930 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-istio-envoy\") pod \"e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg\" (UID: \"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf\") " pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" Oct 30 19:04:59.119643 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:59.111625 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-istio-data\") pod \"e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg\" (UID: \"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf\") " pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" Oct 30 19:04:59.119643 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:59.115414 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-istiod-ca-cert\") pod \"e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg\" (UID: \"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf\") " pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" Oct 30 19:04:59.119643 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:59.118840 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-workload-socket\") pod \"e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg\" (UID: \"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf\") " pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" Oct 30 19:04:59.119643 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:59.119187 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-istio-token\") pod \"e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg\" (UID: \"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf\") " pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" Oct 30 19:04:59.119643 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:59.119590 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-credential-socket\") pod \"e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg\" (UID: \"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf\") " pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" Oct 30 19:04:59.120632 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:59.119765 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ps2q\" (UniqueName: \"kubernetes.io/projected/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-kube-api-access-5ps2q\") pod \"e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg\" (UID: \"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf\") " pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" Oct 30 19:04:59.121966 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:59.121918 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-credential-socket\") pod \"e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg\" (UID: \"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf\") " pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" Oct 30 19:04:59.137161 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:59.125064 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-istiod-ca-cert\") pod \"e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg\" (UID: \"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf\") " pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" Oct 30 19:04:59.229927 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:59.226925 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ps2q\" (UniqueName: \"kubernetes.io/projected/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-kube-api-access-5ps2q\") pod \"e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg\" (UID: \"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf\") " pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" Oct 30 19:04:59.250170 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:59.242705 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-istio-token\") pod \"e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg\" (UID: \"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf\") " pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" Oct 30 19:04:59.250170 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:59.245435 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-istio-envoy\") pod \"e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg\" (UID: \"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf\") " pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" Oct 30 19:04:59.250170 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:59.245929 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-istio-podinfo\") pod \"e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg\" (UID: \"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf\") " pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" Oct 30 19:04:59.483276 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:59.483237 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" Oct 30 19:04:59.820451 ip-10-0-138-176 kubenswrapper[4442]: W1030 19:04:59.820417 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0edbed5d_d9dd_4ccd_8ee5_7af32e7c74cf.slice/crio-6fe3a0ce8c3d6d4e7364470350a85b9ab7bac3e18b3f2553ec41c3e9dcaaceac WatchSource:0}: Error finding container 6fe3a0ce8c3d6d4e7364470350a85b9ab7bac3e18b3f2553ec41c3e9dcaaceac: Status 404 returned error can't find the container with id 6fe3a0ce8c3d6d4e7364470350a85b9ab7bac3e18b3f2553ec41c3e9dcaaceac Oct 30 19:04:59.821500 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:59.821473 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg"] Oct 30 19:04:59.823979 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:59.823944 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:04:59.824056 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:59.824015 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:04:59.824056 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:04:59.824045 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:05:00.506768 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:00.506719 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" event={"ID":"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf","Type":"ContainerStarted","Data":"0c2ef475dfa96e7018d41cac96a0a2285ed2f01cbf2d8bbc0a64d2aa63155315"} Oct 30 19:05:00.507307 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:00.506784 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" event={"ID":"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf","Type":"ContainerStarted","Data":"6fe3a0ce8c3d6d4e7364470350a85b9ab7bac3e18b3f2553ec41c3e9dcaaceac"} Oct 30 19:05:00.577565 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:00.577498 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" podStartSLOduration=2.577484208 podStartE2EDuration="2.577484208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:00.565376461 +0000 UTC m=+3997.655438802" watchObservedRunningTime="2026-10-30 19:05:00.577484208 +0000 UTC m=+3997.667546546" Oct 30 19:05:01.484324 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:01.484288 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" Oct 30 19:05:01.490859 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:01.490833 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" Oct 30 19:05:01.515517 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:01.515478 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" Oct 30 19:05:01.517001 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:01.516976 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" Oct 30 19:05:15.732214 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:15.724501 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78763cdc-7vznh"] Oct 30 19:05:15.818090 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:15.811296 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78763cdc-7vznh"] Oct 30 19:05:17.189525 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:17.189476 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4" podUID="cf7d5f54-e091-4032-ab11-af2687e453c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 19:05:17.357871 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:17.356618 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="422f6f0d-871f-44d2-8fba-2dc65d684ced" path="/var/lib/kubelet/pods/422f6f0d-871f-44d2-8fba-2dc65d684ced/volumes" Oct 30 19:05:17.357871 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:17.357190 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-daf3ba/test-model-daf3ba-kserve-84cd9f6df-bfn9m"] Oct 30 19:05:17.574341 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:05:17.574257 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 19:05:27.717388 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:27.717354 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-869b55-b745cd4f5-7smvc"] Oct 30 19:05:32.417593 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:32.417552 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-daf3ba/test-model-daf3ba-kserve-84cd9f6df-bfn9m" podUID="69d3fb60-519d-4101-a848-17c74270fca7" containerName="main" containerID="cri-o://331ea596a4f2dd27567ddc312b77651908996c90d8ce9738e6227576409b806d" gracePeriod=45 Oct 30 19:05:32.714099 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:32.714066 4442 generic.go:356] "Generic (PLEG): container finished" podID="69d3fb60-519d-4101-a848-17c74270fca7" containerID="331ea596a4f2dd27567ddc312b77651908996c90d8ce9738e6227576409b806d" exitCode=0 Oct 30 19:05:32.714335 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:32.714165 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-daf3ba/test-model-daf3ba-kserve-84cd9f6df-bfn9m" event={"ID":"69d3fb60-519d-4101-a848-17c74270fca7","Type":"ContainerDied","Data":"331ea596a4f2dd27567ddc312b77651908996c90d8ce9738e6227576409b806d"} Oct 30 19:05:32.714335 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:32.714212 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-daf3ba/test-model-daf3ba-kserve-84cd9f6df-bfn9m" event={"ID":"69d3fb60-519d-4101-a848-17c74270fca7","Type":"ContainerDied","Data":"c3c34df61c69513bbd6e18221dc1cd2a98dee56232f0583de9079b36589b0e0d"} Oct 30 19:05:32.714335 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:32.714224 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3c34df61c69513bbd6e18221dc1cd2a98dee56232f0583de9079b36589b0e0d" Oct 30 19:05:32.718293 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:32.718257 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-869b55-b745cd4f5-7smvc" podUID="0709501c-2bbd-4926-8679-0e65a78e09ac" containerName="payload-pre-processing" containerID="cri-o://908cb5d05e08bba2a38762ed63e2737dea0985078888c26d9f7fbe407bb312e7" gracePeriod=25 Oct 30 19:05:32.730896 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:32.730871 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-daf3ba/test-model-daf3ba-kserve-84cd9f6df-bfn9m" Oct 30 19:05:32.815489 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:32.815449 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69d3fb60-519d-4101-a848-17c74270fca7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/69d3fb60-519d-4101-a848-17c74270fca7-dshm\") pod \"69d3fb60-519d-4101-a848-17c74270fca7\" (UID: \"69d3fb60-519d-4101-a848-17c74270fca7\") " Oct 30 19:05:32.815700 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:32.815672 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69d3fb60-519d-4101-a848-17c74270fca7-kube-api-access-n4wsm\" (UniqueName: \"kubernetes.io/projected/69d3fb60-519d-4101-a848-17c74270fca7-kube-api-access-n4wsm\") pod \"69d3fb60-519d-4101-a848-17c74270fca7\" (UID: \"69d3fb60-519d-4101-a848-17c74270fca7\") " Oct 30 19:05:32.815779 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:32.815755 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69d3fb60-519d-4101-a848-17c74270fca7-home\" (UniqueName: \"kubernetes.io/empty-dir/69d3fb60-519d-4101-a848-17c74270fca7-home\") pod \"69d3fb60-519d-4101-a848-17c74270fca7\" (UID: \"69d3fb60-519d-4101-a848-17c74270fca7\") " Oct 30 19:05:32.815841 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:32.815782 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69d3fb60-519d-4101-a848-17c74270fca7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69d3fb60-519d-4101-a848-17c74270fca7-model-cache\") pod \"69d3fb60-519d-4101-a848-17c74270fca7\" (UID: \"69d3fb60-519d-4101-a848-17c74270fca7\") " Oct 30 19:05:32.816057 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:32.816033 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69d3fb60-519d-4101-a848-17c74270fca7-model-cache" pod "69d3fb60-519d-4101-a848-17c74270fca7" (UID: "69d3fb60-519d-4101-a848-17c74270fca7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:32.816299 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:32.816261 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69d3fb60-519d-4101-a848-17c74270fca7-home" pod "69d3fb60-519d-4101-a848-17c74270fca7" (UID: "69d3fb60-519d-4101-a848-17c74270fca7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:32.816453 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:32.816300 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69d3fb60-519d-4101-a848-17c74270fca7-tls-certs\" (UniqueName: \"kubernetes.io/secret/69d3fb60-519d-4101-a848-17c74270fca7-tls-certs\") pod \"69d3fb60-519d-4101-a848-17c74270fca7\" (UID: \"69d3fb60-519d-4101-a848-17c74270fca7\") " Oct 30 19:05:32.816453 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:32.816389 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69d3fb60-519d-4101-a848-17c74270fca7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69d3fb60-519d-4101-a848-17c74270fca7-tmp-dir\") pod \"69d3fb60-519d-4101-a848-17c74270fca7\" (UID: \"69d3fb60-519d-4101-a848-17c74270fca7\") " Oct 30 19:05:32.817215 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:32.817193 4442 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69d3fb60-519d-4101-a848-17c74270fca7-home\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:05:32.817327 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:32.817219 4442 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69d3fb60-519d-4101-a848-17c74270fca7-model-cache\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:05:32.817472 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:32.817452 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69d3fb60-519d-4101-a848-17c74270fca7-tmp-dir" pod "69d3fb60-519d-4101-a848-17c74270fca7" (UID: "69d3fb60-519d-4101-a848-17c74270fca7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:32.818898 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:32.818867 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69d3fb60-519d-4101-a848-17c74270fca7-dshm" pod "69d3fb60-519d-4101-a848-17c74270fca7" (UID: "69d3fb60-519d-4101-a848-17c74270fca7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:32.819760 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:32.819735 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69d3fb60-519d-4101-a848-17c74270fca7-tls-certs" pod "69d3fb60-519d-4101-a848-17c74270fca7" (UID: "69d3fb60-519d-4101-a848-17c74270fca7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:05:32.819889 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:32.819864 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69d3fb60-519d-4101-a848-17c74270fca7-kube-api-access-n4wsm" pod "69d3fb60-519d-4101-a848-17c74270fca7" (UID: "69d3fb60-519d-4101-a848-17c74270fca7"). InnerVolumeSpecName "kube-api-access-n4wsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:05:32.919586 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:32.919554 4442 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69d3fb60-519d-4101-a848-17c74270fca7-dshm\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:05:32.919586 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:32.919584 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4wsm\" (UniqueName: \"kubernetes.io/projected/69d3fb60-519d-4101-a848-17c74270fca7-kube-api-access-n4wsm\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:05:32.919841 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:32.919594 4442 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69d3fb60-519d-4101-a848-17c74270fca7-tls-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:05:32.919841 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:32.919607 4442 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69d3fb60-519d-4101-a848-17c74270fca7-tmp-dir\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:05:33.724873 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:33.724840 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-daf3ba/test-model-daf3ba-kserve-84cd9f6df-bfn9m" Oct 30 19:05:33.779267 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:33.779233 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-daf3ba/test-model-daf3ba-kserve-84cd9f6df-bfn9m"] Oct 30 19:05:33.790220 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:33.790187 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-daf3ba/test-model-daf3ba-kserve-84cd9f6df-bfn9m"] Oct 30 19:05:35.070055 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:35.070023 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69d3fb60-519d-4101-a848-17c74270fca7" path="/var/lib/kubelet/pods/69d3fb60-519d-4101-a848-17c74270fca7/volumes" Oct 30 19:05:38.533283 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:38.533238 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-869b55-b745cd4f5-7smvc" podUID="0709501c-2bbd-4926-8679-0e65a78e09ac" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:05:42.031900 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:42.031856 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baca322b-pnr26"] Oct 30 19:05:42.041227 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:42.041197 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baca322b-pnr26"] Oct 30 19:05:43.070085 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:43.070044 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14e77784-722e-4d16-86f7-9fc801b52b4b" path="/var/lib/kubelet/pods/14e77784-722e-4d16-86f7-9fc801b52b4b/volumes" Oct 30 19:05:44.487066 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:44.487014 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw"] Oct 30 19:05:44.487604 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:44.487254 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" podUID="29ce0e02-4041-4ecb-bb5f-c885099d00b7" containerName="istio-proxy" containerID="cri-o://58b26f522793c6b618d5a220043e9f1e6d9b804468523ef5e80262da8bb8aae1" gracePeriod=30 Oct 30 19:05:46.026123 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:46.026086 4442 patch_prober.go:29] interesting pod/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:05:46.026593 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:46.026200 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" podUID="29ce0e02-4041-4ecb-bb5f-c885099d00b7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:05:47.563575 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:05:47.563526 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 19:05:48.478047 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:48.478011 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-869b55-b745cd4f5-7smvc" podUID="0709501c-2bbd-4926-8679-0e65a78e09ac" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:05:49.794342 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:49.794315 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" Oct 30 19:05:49.865836 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:49.865454 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29ce0e02-4041-4ecb-bb5f-c885099d00b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ce0e02-4041-4ecb-bb5f-c885099d00b7-workload-socket\") pod \"29ce0e02-4041-4ecb-bb5f-c885099d00b7\" (UID: \"29ce0e02-4041-4ecb-bb5f-c885099d00b7\") " Oct 30 19:05:49.866126 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:49.866099 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29ce0e02-4041-4ecb-bb5f-c885099d00b7-istio-token\" (UniqueName: \"kubernetes.io/projected/29ce0e02-4041-4ecb-bb5f-c885099d00b7-istio-token\") pod \"29ce0e02-4041-4ecb-bb5f-c885099d00b7\" (UID: \"29ce0e02-4041-4ecb-bb5f-c885099d00b7\") " Oct 30 19:05:49.866788 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:49.865802 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29ce0e02-4041-4ecb-bb5f-c885099d00b7-workload-socket" pod "29ce0e02-4041-4ecb-bb5f-c885099d00b7" (UID: "29ce0e02-4041-4ecb-bb5f-c885099d00b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:49.866978 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:49.866955 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29ce0e02-4041-4ecb-bb5f-c885099d00b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29ce0e02-4041-4ecb-bb5f-c885099d00b7-istio-podinfo\") pod \"29ce0e02-4041-4ecb-bb5f-c885099d00b7\" (UID: \"29ce0e02-4041-4ecb-bb5f-c885099d00b7\") " Oct 30 19:05:49.867696 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:49.867677 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29ce0e02-4041-4ecb-bb5f-c885099d00b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29ce0e02-4041-4ecb-bb5f-c885099d00b7-workload-certs\") pod \"29ce0e02-4041-4ecb-bb5f-c885099d00b7\" (UID: \"29ce0e02-4041-4ecb-bb5f-c885099d00b7\") " Oct 30 19:05:49.868174 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:49.868054 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29ce0e02-4041-4ecb-bb5f-c885099d00b7-workload-certs" pod "29ce0e02-4041-4ecb-bb5f-c885099d00b7" (UID: "29ce0e02-4041-4ecb-bb5f-c885099d00b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:49.868635 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:49.868573 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29ce0e02-4041-4ecb-bb5f-c885099d00b7-istiod-ca-cert" pod "29ce0e02-4041-4ecb-bb5f-c885099d00b7" (UID: "29ce0e02-4041-4ecb-bb5f-c885099d00b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:05:49.868797 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:49.868781 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29ce0e02-4041-4ecb-bb5f-c885099d00b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29ce0e02-4041-4ecb-bb5f-c885099d00b7-istiod-ca-cert\") pod \"29ce0e02-4041-4ecb-bb5f-c885099d00b7\" (UID: \"29ce0e02-4041-4ecb-bb5f-c885099d00b7\") " Oct 30 19:05:49.868995 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:49.868981 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29ce0e02-4041-4ecb-bb5f-c885099d00b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ce0e02-4041-4ecb-bb5f-c885099d00b7-credential-socket\") pod \"29ce0e02-4041-4ecb-bb5f-c885099d00b7\" (UID: \"29ce0e02-4041-4ecb-bb5f-c885099d00b7\") " Oct 30 19:05:49.870168 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:49.870044 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29ce0e02-4041-4ecb-bb5f-c885099d00b7-credential-socket" pod "29ce0e02-4041-4ecb-bb5f-c885099d00b7" (UID: "29ce0e02-4041-4ecb-bb5f-c885099d00b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:49.870842 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:49.870335 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29ce0e02-4041-4ecb-bb5f-c885099d00b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29ce0e02-4041-4ecb-bb5f-c885099d00b7-istio-envoy\") pod \"29ce0e02-4041-4ecb-bb5f-c885099d00b7\" (UID: \"29ce0e02-4041-4ecb-bb5f-c885099d00b7\") " Oct 30 19:05:49.871111 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:49.871094 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29ce0e02-4041-4ecb-bb5f-c885099d00b7-kube-api-access-sdffg\" (UniqueName: \"kubernetes.io/projected/29ce0e02-4041-4ecb-bb5f-c885099d00b7-kube-api-access-sdffg\") pod \"29ce0e02-4041-4ecb-bb5f-c885099d00b7\" (UID: \"29ce0e02-4041-4ecb-bb5f-c885099d00b7\") " Oct 30 19:05:49.871599 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:49.871582 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29ce0e02-4041-4ecb-bb5f-c885099d00b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29ce0e02-4041-4ecb-bb5f-c885099d00b7-istio-data\") pod \"29ce0e02-4041-4ecb-bb5f-c885099d00b7\" (UID: \"29ce0e02-4041-4ecb-bb5f-c885099d00b7\") " Oct 30 19:05:49.872249 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:49.872000 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29ce0e02-4041-4ecb-bb5f-c885099d00b7-istio-data" pod "29ce0e02-4041-4ecb-bb5f-c885099d00b7" (UID: "29ce0e02-4041-4ecb-bb5f-c885099d00b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:49.873050 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:49.873028 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29ce0e02-4041-4ecb-bb5f-c885099d00b7-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:05:49.873296 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:49.873274 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29ce0e02-4041-4ecb-bb5f-c885099d00b7-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:05:49.873450 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:49.873436 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ce0e02-4041-4ecb-bb5f-c885099d00b7-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:05:49.873585 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:49.873572 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29ce0e02-4041-4ecb-bb5f-c885099d00b7-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:05:49.873707 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:49.873696 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ce0e02-4041-4ecb-bb5f-c885099d00b7-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:05:49.873889 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:49.873310 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29ce0e02-4041-4ecb-bb5f-c885099d00b7-istio-envoy" pod "29ce0e02-4041-4ecb-bb5f-c885099d00b7" (UID: "29ce0e02-4041-4ecb-bb5f-c885099d00b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:49.874344 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:49.874320 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29ce0e02-4041-4ecb-bb5f-c885099d00b7-kube-api-access-sdffg" pod "29ce0e02-4041-4ecb-bb5f-c885099d00b7" (UID: "29ce0e02-4041-4ecb-bb5f-c885099d00b7"). InnerVolumeSpecName "kube-api-access-sdffg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:05:49.885007 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:49.884982 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29ce0e02-4041-4ecb-bb5f-c885099d00b7-istio-token" pod "29ce0e02-4041-4ecb-bb5f-c885099d00b7" (UID: "29ce0e02-4041-4ecb-bb5f-c885099d00b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:05:49.886460 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:49.886433 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29ce0e02-4041-4ecb-bb5f-c885099d00b7-istio-podinfo" pod "29ce0e02-4041-4ecb-bb5f-c885099d00b7" (UID: "29ce0e02-4041-4ecb-bb5f-c885099d00b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:05:49.975563 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:49.975527 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29ce0e02-4041-4ecb-bb5f-c885099d00b7-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:05:49.975563 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:49.975557 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdffg\" (UniqueName: \"kubernetes.io/projected/29ce0e02-4041-4ecb-bb5f-c885099d00b7-kube-api-access-sdffg\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:05:49.975563 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:49.975567 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29ce0e02-4041-4ecb-bb5f-c885099d00b7-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:05:49.975819 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:49.975575 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29ce0e02-4041-4ecb-bb5f-c885099d00b7-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:05:50.232953 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:50.232921 4442 generic.go:356] "Generic (PLEG): container finished" podID="29ce0e02-4041-4ecb-bb5f-c885099d00b7" containerID="58b26f522793c6b618d5a220043e9f1e6d9b804468523ef5e80262da8bb8aae1" exitCode=0 Oct 30 19:05:50.233204 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:50.232998 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" Oct 30 19:05:50.233204 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:50.233001 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" event={"ID":"29ce0e02-4041-4ecb-bb5f-c885099d00b7","Type":"ContainerDied","Data":"58b26f522793c6b618d5a220043e9f1e6d9b804468523ef5e80262da8bb8aae1"} Oct 30 19:05:50.233204 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:50.233064 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw" event={"ID":"29ce0e02-4041-4ecb-bb5f-c885099d00b7","Type":"ContainerDied","Data":"5864beb8eed8f9e9fd71d32e77b649823cd77a4dc05277bfeda807e0ae9923f8"} Oct 30 19:05:50.233204 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:50.233089 4442 scope.go:122] "RemoveContainer" containerID="58b26f522793c6b618d5a220043e9f1e6d9b804468523ef5e80262da8bb8aae1" Oct 30 19:05:50.246599 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:50.246576 4442 scope.go:122] "RemoveContainer" containerID="71f4dbf20f87947e8cfbd5fcbf4435ae557edb9a8619c06379cd334fcc223379" Oct 30 19:05:50.258347 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:50.258328 4442 scope.go:122] "RemoveContainer" containerID="58b26f522793c6b618d5a220043e9f1e6d9b804468523ef5e80262da8bb8aae1" Oct 30 19:05:50.258704 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:05:50.258680 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58b26f522793c6b618d5a220043e9f1e6d9b804468523ef5e80262da8bb8aae1\": container with ID starting with 58b26f522793c6b618d5a220043e9f1e6d9b804468523ef5e80262da8bb8aae1 not found: ID does not exist" containerID="58b26f522793c6b618d5a220043e9f1e6d9b804468523ef5e80262da8bb8aae1" Oct 30 19:05:50.258803 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:50.258718 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58b26f522793c6b618d5a220043e9f1e6d9b804468523ef5e80262da8bb8aae1"} err="failed to get container status \"58b26f522793c6b618d5a220043e9f1e6d9b804468523ef5e80262da8bb8aae1\": rpc error: code = NotFound desc = could not find container \"58b26f522793c6b618d5a220043e9f1e6d9b804468523ef5e80262da8bb8aae1\": container with ID starting with 58b26f522793c6b618d5a220043e9f1e6d9b804468523ef5e80262da8bb8aae1 not found: ID does not exist" Oct 30 19:05:50.258803 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:50.258751 4442 scope.go:122] "RemoveContainer" containerID="71f4dbf20f87947e8cfbd5fcbf4435ae557edb9a8619c06379cd334fcc223379" Oct 30 19:05:50.259070 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:05:50.259050 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71f4dbf20f87947e8cfbd5fcbf4435ae557edb9a8619c06379cd334fcc223379\": container with ID starting with 71f4dbf20f87947e8cfbd5fcbf4435ae557edb9a8619c06379cd334fcc223379 not found: ID does not exist" containerID="71f4dbf20f87947e8cfbd5fcbf4435ae557edb9a8619c06379cd334fcc223379" Oct 30 19:05:50.259167 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:50.259080 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71f4dbf20f87947e8cfbd5fcbf4435ae557edb9a8619c06379cd334fcc223379"} err="failed to get container status \"71f4dbf20f87947e8cfbd5fcbf4435ae557edb9a8619c06379cd334fcc223379\": rpc error: code = NotFound desc = could not find container \"71f4dbf20f87947e8cfbd5fcbf4435ae557edb9a8619c06379cd334fcc223379\": container with ID starting with 71f4dbf20f87947e8cfbd5fcbf4435ae557edb9a8619c06379cd334fcc223379 not found: ID does not exist" Oct 30 19:05:50.288626 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:50.288594 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw"] Oct 30 19:05:50.299975 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:50.299950 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-869b55-openshift-default-675f9b4749-2l7bw"] Oct 30 19:05:51.067896 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:51.067859 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29ce0e02-4041-4ecb-bb5f-c885099d00b7" path="/var/lib/kubelet/pods/29ce0e02-4041-4ecb-bb5f-c885099d00b7/volumes" Oct 30 19:05:52.812664 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:05:52.812628 4442 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0709501c_2bbd_4926_8679_0e65a78e09ac.slice/crio-908cb5d05e08bba2a38762ed63e2737dea0985078888c26d9f7fbe407bb312e7.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:05:53.230548 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:53.230507 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-869b55-b745cd4f5-7smvc" Oct 30 19:05:53.268517 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:53.268475 4442 generic.go:356] "Generic (PLEG): container finished" podID="0709501c-2bbd-4926-8679-0e65a78e09ac" containerID="908cb5d05e08bba2a38762ed63e2737dea0985078888c26d9f7fbe407bb312e7" exitCode=0 Oct 30 19:05:53.268517 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:53.268519 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-869b55-b745cd4f5-7smvc" Oct 30 19:05:53.268804 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:53.268564 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-869b55-b745cd4f5-7smvc" event={"ID":"0709501c-2bbd-4926-8679-0e65a78e09ac","Type":"ContainerDied","Data":"908cb5d05e08bba2a38762ed63e2737dea0985078888c26d9f7fbe407bb312e7"} Oct 30 19:05:53.268804 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:53.268661 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-869b55-b745cd4f5-7smvc" event={"ID":"0709501c-2bbd-4926-8679-0e65a78e09ac","Type":"ContainerDied","Data":"f05e2872002277c537b2dc811f1fa329a662680521aa27428edb20a473382b5a"} Oct 30 19:05:53.268804 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:53.268708 4442 scope.go:122] "RemoveContainer" containerID="908cb5d05e08bba2a38762ed63e2737dea0985078888c26d9f7fbe407bb312e7" Oct 30 19:05:53.294210 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:53.279285 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0709501c-2bbd-4926-8679-0e65a78e09ac-kube-api-access-756zj\" (UniqueName: \"kubernetes.io/projected/0709501c-2bbd-4926-8679-0e65a78e09ac-kube-api-access-756zj\") pod \"0709501c-2bbd-4926-8679-0e65a78e09ac\" (UID: \"0709501c-2bbd-4926-8679-0e65a78e09ac\") " Oct 30 19:05:53.294210 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:53.279507 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0709501c-2bbd-4926-8679-0e65a78e09ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0709501c-2bbd-4926-8679-0e65a78e09ac-plugins-config-volume\") pod \"0709501c-2bbd-4926-8679-0e65a78e09ac\" (UID: \"0709501c-2bbd-4926-8679-0e65a78e09ac\") " Oct 30 19:05:53.294210 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:53.288569 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0709501c-2bbd-4926-8679-0e65a78e09ac-plugins-config-volume" pod "0709501c-2bbd-4926-8679-0e65a78e09ac" (UID: "0709501c-2bbd-4926-8679-0e65a78e09ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:05:53.297598 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:53.297567 4442 scope.go:122] "RemoveContainer" containerID="908cb5d05e08bba2a38762ed63e2737dea0985078888c26d9f7fbe407bb312e7" Oct 30 19:05:53.298877 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:05:53.298065 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"908cb5d05e08bba2a38762ed63e2737dea0985078888c26d9f7fbe407bb312e7\": container with ID starting with 908cb5d05e08bba2a38762ed63e2737dea0985078888c26d9f7fbe407bb312e7 not found: ID does not exist" containerID="908cb5d05e08bba2a38762ed63e2737dea0985078888c26d9f7fbe407bb312e7" Oct 30 19:05:53.298877 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:53.298124 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"908cb5d05e08bba2a38762ed63e2737dea0985078888c26d9f7fbe407bb312e7"} err="failed to get container status \"908cb5d05e08bba2a38762ed63e2737dea0985078888c26d9f7fbe407bb312e7\": rpc error: code = NotFound desc = could not find container \"908cb5d05e08bba2a38762ed63e2737dea0985078888c26d9f7fbe407bb312e7\": container with ID starting with 908cb5d05e08bba2a38762ed63e2737dea0985078888c26d9f7fbe407bb312e7 not found: ID does not exist" Oct 30 19:05:53.323512 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:53.323451 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0709501c-2bbd-4926-8679-0e65a78e09ac-kube-api-access-756zj" pod "0709501c-2bbd-4926-8679-0e65a78e09ac" (UID: "0709501c-2bbd-4926-8679-0e65a78e09ac"). InnerVolumeSpecName "kube-api-access-756zj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:05:53.383052 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:53.383003 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-756zj\" (UniqueName: \"kubernetes.io/projected/0709501c-2bbd-4926-8679-0e65a78e09ac-kube-api-access-756zj\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:05:53.383052 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:53.383053 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0709501c-2bbd-4926-8679-0e65a78e09ac-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:05:53.630220 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:53.626481 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-869b55-b745cd4f5-7smvc"] Oct 30 19:05:53.636148 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:53.636084 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-869b55-b745cd4f5-7smvc"] Oct 30 19:05:55.454498 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:55.454451 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0709501c-2bbd-4926-8679-0e65a78e09ac" path="/var/lib/kubelet/pods/0709501c-2bbd-4926-8679-0e65a78e09ac/volumes" Oct 30 19:05:55.520862 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:55.520813 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-x2rr2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:05:55.521187 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:05:55.521150 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" podUID="e03523d4-1b6f-4a80-9006-26b0345d809d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:06:17.562481 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:06:17.562439 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 19:06:28.735681 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:06:28.735649 4442 scope.go:122] "RemoveContainer" containerID="7a4cb71c99c5c8ec3367c8db46813b7c9015a6dc8185e3dd57e85a93fdeab7d1" Oct 30 19:06:28.750027 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:06:28.750004 4442 scope.go:122] "RemoveContainer" containerID="63e206950b02387b2cceae2d0b8e44941be2f6a8b45932b0878d4ff8963d64dd" Oct 30 19:06:28.761257 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:06:28.761233 4442 scope.go:122] "RemoveContainer" containerID="c00c75efa7a49f95d797d6838c588bb6a83e3e266586f75834fca09c9d8a2832" Oct 30 19:06:47.566583 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:06:47.566535 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 19:07:03.286192 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:03.285527 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t"] Oct 30 19:07:03.292393 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:03.292305 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" podUID="5ba882f0-916a-4651-8ff9-1e8fe8d9e021" containerName="istio-proxy" containerID="cri-o://bfd655fb9d40b44edbaf88bc1903e789af11b5e5d7f0b773783f48e8b364fe72" gracePeriod=30 Oct 30 19:07:07.036834 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:07.036798 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eed264df-wdcbd"] Oct 30 19:07:07.043196 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:07.043169 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eed264df-wdcbd"] Oct 30 19:07:07.069101 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:07.069061 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764" path="/var/lib/kubelet/pods/3b7c6bd3-a04b-44c0-bd5c-24e3dcea6764/volumes" Oct 30 19:07:08.621518 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:08.621491 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" Oct 30 19:07:08.678769 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:08.678733 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-kube-api-access-mmfm5\" (UniqueName: \"kubernetes.io/projected/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-kube-api-access-mmfm5\") pod \"5ba882f0-916a-4651-8ff9-1e8fe8d9e021\" (UID: \"5ba882f0-916a-4651-8ff9-1e8fe8d9e021\") " Oct 30 19:07:08.678969 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:08.678862 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-istio-podinfo\") pod \"5ba882f0-916a-4651-8ff9-1e8fe8d9e021\" (UID: \"5ba882f0-916a-4651-8ff9-1e8fe8d9e021\") " Oct 30 19:07:08.679041 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:08.679013 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-istio-data\") pod \"5ba882f0-916a-4651-8ff9-1e8fe8d9e021\" (UID: \"5ba882f0-916a-4651-8ff9-1e8fe8d9e021\") " Oct 30 19:07:08.679096 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:08.679057 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-workload-socket\") pod \"5ba882f0-916a-4651-8ff9-1e8fe8d9e021\" (UID: \"5ba882f0-916a-4651-8ff9-1e8fe8d9e021\") " Oct 30 19:07:08.679192 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:08.679095 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-istio-envoy\") pod \"5ba882f0-916a-4651-8ff9-1e8fe8d9e021\" (UID: \"5ba882f0-916a-4651-8ff9-1e8fe8d9e021\") " Oct 30 19:07:08.679192 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:08.679155 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-workload-certs\") pod \"5ba882f0-916a-4651-8ff9-1e8fe8d9e021\" (UID: \"5ba882f0-916a-4651-8ff9-1e8fe8d9e021\") " Oct 30 19:07:08.679319 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:08.679227 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-istiod-ca-cert\") pod \"5ba882f0-916a-4651-8ff9-1e8fe8d9e021\" (UID: \"5ba882f0-916a-4651-8ff9-1e8fe8d9e021\") " Oct 30 19:07:08.679319 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:08.679282 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-istio-token\" (UniqueName: \"kubernetes.io/projected/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-istio-token\") pod \"5ba882f0-916a-4651-8ff9-1e8fe8d9e021\" (UID: \"5ba882f0-916a-4651-8ff9-1e8fe8d9e021\") " Oct 30 19:07:08.679428 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:08.679330 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-credential-socket\") pod \"5ba882f0-916a-4651-8ff9-1e8fe8d9e021\" (UID: \"5ba882f0-916a-4651-8ff9-1e8fe8d9e021\") " Oct 30 19:07:08.679522 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:08.679476 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-workload-certs" pod "5ba882f0-916a-4651-8ff9-1e8fe8d9e021" (UID: "5ba882f0-916a-4651-8ff9-1e8fe8d9e021"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:07:08.679654 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:08.679633 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-istio-data" pod "5ba882f0-916a-4651-8ff9-1e8fe8d9e021" (UID: "5ba882f0-916a-4651-8ff9-1e8fe8d9e021"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:07:08.679767 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:08.679653 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-workload-socket" pod "5ba882f0-916a-4651-8ff9-1e8fe8d9e021" (UID: "5ba882f0-916a-4651-8ff9-1e8fe8d9e021"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:07:08.680015 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:08.679986 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-credential-socket" pod "5ba882f0-916a-4651-8ff9-1e8fe8d9e021" (UID: "5ba882f0-916a-4651-8ff9-1e8fe8d9e021"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:07:08.680015 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:08.679984 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-istiod-ca-cert" pod "5ba882f0-916a-4651-8ff9-1e8fe8d9e021" (UID: "5ba882f0-916a-4651-8ff9-1e8fe8d9e021"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:07:08.681518 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:08.681491 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:07:08.681661 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:08.681523 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:07:08.681661 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:08.681539 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:07:08.681661 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:08.681555 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:07:08.681661 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:08.681572 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:07:08.683634 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:08.683605 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-istio-podinfo" pod "5ba882f0-916a-4651-8ff9-1e8fe8d9e021" (UID: "5ba882f0-916a-4651-8ff9-1e8fe8d9e021"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:07:08.683634 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:08.683619 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-istio-envoy" pod "5ba882f0-916a-4651-8ff9-1e8fe8d9e021" (UID: "5ba882f0-916a-4651-8ff9-1e8fe8d9e021"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:07:08.683783 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:08.683677 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-kube-api-access-mmfm5" pod "5ba882f0-916a-4651-8ff9-1e8fe8d9e021" (UID: "5ba882f0-916a-4651-8ff9-1e8fe8d9e021"). InnerVolumeSpecName "kube-api-access-mmfm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:07:08.684321 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:08.684293 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-istio-token" pod "5ba882f0-916a-4651-8ff9-1e8fe8d9e021" (UID: "5ba882f0-916a-4651-8ff9-1e8fe8d9e021"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:07:08.806155 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:08.797489 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmfm5\" (UniqueName: \"kubernetes.io/projected/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-kube-api-access-mmfm5\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:07:08.806155 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:08.797530 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:07:08.806155 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:08.797545 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:07:08.806155 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:08.797556 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ba882f0-916a-4651-8ff9-1e8fe8d9e021-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:07:08.827231 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:08.817547 4442 generic.go:356] "Generic (PLEG): container finished" podID="5ba882f0-916a-4651-8ff9-1e8fe8d9e021" containerID="bfd655fb9d40b44edbaf88bc1903e789af11b5e5d7f0b773783f48e8b364fe72" exitCode=0 Oct 30 19:07:08.827231 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:08.817668 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" event={"ID":"5ba882f0-916a-4651-8ff9-1e8fe8d9e021","Type":"ContainerDied","Data":"bfd655fb9d40b44edbaf88bc1903e789af11b5e5d7f0b773783f48e8b364fe72"} Oct 30 19:07:08.827231 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:08.817719 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" event={"ID":"5ba882f0-916a-4651-8ff9-1e8fe8d9e021","Type":"ContainerDied","Data":"49df48a5e5b6a9f1bb5119df516589e65afda5cce8d091d59d3e836201aa6296"} Oct 30 19:07:08.827231 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:08.817748 4442 scope.go:122] "RemoveContainer" containerID="bfd655fb9d40b44edbaf88bc1903e789af11b5e5d7f0b773783f48e8b364fe72" Oct 30 19:07:08.827231 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:08.817926 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t" Oct 30 19:07:08.865470 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:08.865440 4442 scope.go:122] "RemoveContainer" containerID="701d8ba2128526b0148de4a9c69ecf43f6126e34c7665e54f82af7f51f3e8fd1" Oct 30 19:07:08.880292 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:08.880231 4442 scope.go:122] "RemoveContainer" containerID="bfd655fb9d40b44edbaf88bc1903e789af11b5e5d7f0b773783f48e8b364fe72" Oct 30 19:07:08.880591 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:07:08.880568 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfd655fb9d40b44edbaf88bc1903e789af11b5e5d7f0b773783f48e8b364fe72\": container with ID starting with bfd655fb9d40b44edbaf88bc1903e789af11b5e5d7f0b773783f48e8b364fe72 not found: ID does not exist" containerID="bfd655fb9d40b44edbaf88bc1903e789af11b5e5d7f0b773783f48e8b364fe72" Oct 30 19:07:08.880663 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:08.880601 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfd655fb9d40b44edbaf88bc1903e789af11b5e5d7f0b773783f48e8b364fe72"} err="failed to get container status \"bfd655fb9d40b44edbaf88bc1903e789af11b5e5d7f0b773783f48e8b364fe72\": rpc error: code = NotFound desc = could not find container \"bfd655fb9d40b44edbaf88bc1903e789af11b5e5d7f0b773783f48e8b364fe72\": container with ID starting with bfd655fb9d40b44edbaf88bc1903e789af11b5e5d7f0b773783f48e8b364fe72 not found: ID does not exist" Oct 30 19:07:08.880663 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:08.880622 4442 scope.go:122] "RemoveContainer" containerID="701d8ba2128526b0148de4a9c69ecf43f6126e34c7665e54f82af7f51f3e8fd1" Oct 30 19:07:08.880869 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:07:08.880848 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"701d8ba2128526b0148de4a9c69ecf43f6126e34c7665e54f82af7f51f3e8fd1\": container with ID starting with 701d8ba2128526b0148de4a9c69ecf43f6126e34c7665e54f82af7f51f3e8fd1 not found: ID does not exist" containerID="701d8ba2128526b0148de4a9c69ecf43f6126e34c7665e54f82af7f51f3e8fd1" Oct 30 19:07:08.880926 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:08.880879 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"701d8ba2128526b0148de4a9c69ecf43f6126e34c7665e54f82af7f51f3e8fd1"} err="failed to get container status \"701d8ba2128526b0148de4a9c69ecf43f6126e34c7665e54f82af7f51f3e8fd1\": rpc error: code = NotFound desc = could not find container \"701d8ba2128526b0148de4a9c69ecf43f6126e34c7665e54f82af7f51f3e8fd1\": container with ID starting with 701d8ba2128526b0148de4a9c69ecf43f6126e34c7665e54f82af7f51f3e8fd1 not found: ID does not exist" Oct 30 19:07:08.893403 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:08.893372 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t"] Oct 30 19:07:08.898005 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:08.897974 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9f6b7c-openshift-default-98b5bc999-tcq4t"] Oct 30 19:07:09.069013 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:09.068968 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ba882f0-916a-4651-8ff9-1e8fe8d9e021" path="/var/lib/kubelet/pods/5ba882f0-916a-4651-8ff9-1e8fe8d9e021/volumes" Oct 30 19:07:09.834122 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:09.834091 4442 generic.go:356] "Generic (PLEG): container finished" podID="0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf" containerID="0c2ef475dfa96e7018d41cac96a0a2285ed2f01cbf2d8bbc0a64d2aa63155315" exitCode=0 Oct 30 19:07:09.834567 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:09.834168 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" event={"ID":"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf","Type":"ContainerDied","Data":"0c2ef475dfa96e7018d41cac96a0a2285ed2f01cbf2d8bbc0a64d2aa63155315"} Oct 30 19:07:09.834711 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:09.834694 4442 scope.go:122] "RemoveContainer" containerID="0c2ef475dfa96e7018d41cac96a0a2285ed2f01cbf2d8bbc0a64d2aa63155315" Oct 30 19:07:09.837069 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:09.837031 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:07:09.837235 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:09.837107 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:07:09.837235 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:09.837173 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:07:10.867581 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:10.867543 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" event={"ID":"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf","Type":"ContainerStarted","Data":"bba6d2ac3f713e864fd4ea7804176077f7cb63bce3cd5377ce40545a1a54f3dd"} Oct 30 19:07:11.483986 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:11.483940 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" Oct 30 19:07:11.484258 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:11.484222 4442 patch_prober.go:29] interesting pod/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused" start-of-body= Oct 30 19:07:11.484417 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:11.484287 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" podUID="0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused" Oct 30 19:07:12.484591 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:12.484555 4442 patch_prober.go:29] interesting pod/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused" start-of-body= Oct 30 19:07:12.485020 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:12.484628 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" podUID="0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused" Oct 30 19:07:13.486592 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:13.486557 4442 patch_prober.go:29] interesting pod/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused" start-of-body= Oct 30 19:07:13.487250 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:13.487214 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" podUID="0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused" Oct 30 19:07:14.488643 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:14.488608 4442 patch_prober.go:29] interesting pod/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused" start-of-body= Oct 30 19:07:14.489312 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:14.489282 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" podUID="0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused" Oct 30 19:07:15.486397 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:15.486346 4442 patch_prober.go:29] interesting pod/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused" start-of-body= Oct 30 19:07:15.486607 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:15.486426 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" podUID="0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused" Oct 30 19:07:16.238228 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:16.231232 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4" podUID="cf7d5f54-e091-4032-ab11-af2687e453c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:07:16.487000 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:16.486533 4442 patch_prober.go:29] interesting pod/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused" start-of-body= Oct 30 19:07:16.487000 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:16.486593 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" podUID="0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused" Oct 30 19:07:16.515673 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:16.515580 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" Oct 30 19:07:17.488914 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:17.488872 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" Oct 30 19:07:17.490168 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:17.490121 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" Oct 30 19:07:17.567100 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:07:17.567058 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 19:07:21.173617 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:21.173580 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2107d4-7bf5b8d588-hz7nk"] Oct 30 19:07:21.206870 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:21.206846 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2107d4-7588cd98bc-nt76m"] Oct 30 19:07:26.174954 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:26.174906 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-2107d4-7bf5b8d588-hz7nk" podUID="017e9c94-87b9-4143-bd73-311bb6edbed9" containerName="payload-processing" containerID="cri-o://8d4456cf1413d30fb074e5d95ebc5e0dad05102ad7f3e9eae250d3ffadcb86c0" gracePeriod=25 Oct 30 19:07:26.207645 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:26.207603 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2107d4-7588cd98bc-nt76m" podUID="d147fcbd-d2ce-4a51-a8f1-531db7bcf8fc" containerName="payload-pre-processing" containerID="cri-o://7b49cf6f38e1fa31410e1fcc6ed89a4499a6a0c27b0b4473a619bdfe99821767" gracePeriod=25 Oct 30 19:07:28.849292 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:28.849259 4442 scope.go:122] "RemoveContainer" containerID="846b422026467ec817b01e85b3fcce1e3c195afd96f8dd4e453ffb4d7e053cba" Oct 30 19:07:28.891451 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:28.891420 4442 scope.go:122] "RemoveContainer" containerID="cfd677f5f650f0e85f1feda81a61c5eaedab811ea6803a782fcd5876ae0299c7" Oct 30 19:07:28.904717 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:28.904692 4442 scope.go:122] "RemoveContainer" containerID="1496ab5158b4a1badc27f63ce11e4f85d8e105be0d66ccc1619c5120585cb232" Oct 30 19:07:28.915770 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:28.915748 4442 scope.go:122] "RemoveContainer" containerID="331ea596a4f2dd27567ddc312b77651908996c90d8ce9738e6227576409b806d" Oct 30 19:07:35.591427 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:35.591391 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-2107d4-7bf5b8d588-hz7nk" podUID="017e9c94-87b9-4143-bd73-311bb6edbed9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:07:35.591937 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:35.591392 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2107d4-7588cd98bc-nt76m" podUID="d147fcbd-d2ce-4a51-a8f1-531db7bcf8fc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:07:38.521247 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:38.521210 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg"] Oct 30 19:07:38.521698 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:38.521468 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" podUID="3ae44a8a-6af3-482e-9407-33e148efded7" containerName="istio-proxy" containerID="cri-o://06a1e4039003f2f75279d7af0e79d0f43fe502b537a67af0009539f1b5fb6be6" gracePeriod=30 Oct 30 19:07:40.982613 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:40.982574 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:07:40.983024 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:40.982647 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" podUID="3ae44a8a-6af3-482e-9407-33e148efded7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:07:43.798245 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:43.798054 4442 generic.go:356] "Generic (PLEG): container finished" podID="3ae44a8a-6af3-482e-9407-33e148efded7" containerID="06a1e4039003f2f75279d7af0e79d0f43fe502b537a67af0009539f1b5fb6be6" exitCode=0 Oct 30 19:07:43.798245 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:43.798122 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" event={"ID":"3ae44a8a-6af3-482e-9407-33e148efded7","Type":"ContainerDied","Data":"06a1e4039003f2f75279d7af0e79d0f43fe502b537a67af0009539f1b5fb6be6"} Oct 30 19:07:43.798245 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:43.798215 4442 scope.go:122] "RemoveContainer" containerID="c8d3693599f4d919d5892252c2fd81271195ac69684137b5d32182fb27471a81" Oct 30 19:07:43.832793 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:07:43.832762 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8d3693599f4d919d5892252c2fd81271195ac69684137b5d32182fb27471a81\": container with ID starting with c8d3693599f4d919d5892252c2fd81271195ac69684137b5d32182fb27471a81 not found: ID does not exist" containerID="c8d3693599f4d919d5892252c2fd81271195ac69684137b5d32182fb27471a81" Oct 30 19:07:43.834987 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:43.834963 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" Oct 30 19:07:43.938656 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:43.938625 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ae44a8a-6af3-482e-9407-33e148efded7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae44a8a-6af3-482e-9407-33e148efded7-workload-socket\") pod \"3ae44a8a-6af3-482e-9407-33e148efded7\" (UID: \"3ae44a8a-6af3-482e-9407-33e148efded7\") " Oct 30 19:07:43.938883 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:43.938667 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ae44a8a-6af3-482e-9407-33e148efded7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ae44a8a-6af3-482e-9407-33e148efded7-istiod-ca-cert\") pod \"3ae44a8a-6af3-482e-9407-33e148efded7\" (UID: \"3ae44a8a-6af3-482e-9407-33e148efded7\") " Oct 30 19:07:43.938883 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:43.938856 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ae44a8a-6af3-482e-9407-33e148efded7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ae44a8a-6af3-482e-9407-33e148efded7-istio-data\") pod \"3ae44a8a-6af3-482e-9407-33e148efded7\" (UID: \"3ae44a8a-6af3-482e-9407-33e148efded7\") " Oct 30 19:07:43.939021 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:43.938903 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ae44a8a-6af3-482e-9407-33e148efded7-istio-token\" (UniqueName: \"kubernetes.io/projected/3ae44a8a-6af3-482e-9407-33e148efded7-istio-token\") pod \"3ae44a8a-6af3-482e-9407-33e148efded7\" (UID: \"3ae44a8a-6af3-482e-9407-33e148efded7\") " Oct 30 19:07:43.939021 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:43.938934 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ae44a8a-6af3-482e-9407-33e148efded7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae44a8a-6af3-482e-9407-33e148efded7-credential-socket\") pod \"3ae44a8a-6af3-482e-9407-33e148efded7\" (UID: \"3ae44a8a-6af3-482e-9407-33e148efded7\") " Oct 30 19:07:43.939021 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:43.938988 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ae44a8a-6af3-482e-9407-33e148efded7-workload-socket" pod "3ae44a8a-6af3-482e-9407-33e148efded7" (UID: "3ae44a8a-6af3-482e-9407-33e148efded7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:07:43.939241 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:43.939026 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ae44a8a-6af3-482e-9407-33e148efded7-kube-api-access-96vsx\" (UniqueName: \"kubernetes.io/projected/3ae44a8a-6af3-482e-9407-33e148efded7-kube-api-access-96vsx\") pod \"3ae44a8a-6af3-482e-9407-33e148efded7\" (UID: \"3ae44a8a-6af3-482e-9407-33e148efded7\") " Oct 30 19:07:43.939241 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:43.939104 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ae44a8a-6af3-482e-9407-33e148efded7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ae44a8a-6af3-482e-9407-33e148efded7-istio-podinfo\") pod \"3ae44a8a-6af3-482e-9407-33e148efded7\" (UID: \"3ae44a8a-6af3-482e-9407-33e148efded7\") " Oct 30 19:07:43.939241 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:43.939188 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ae44a8a-6af3-482e-9407-33e148efded7-istio-data" pod "3ae44a8a-6af3-482e-9407-33e148efded7" (UID: "3ae44a8a-6af3-482e-9407-33e148efded7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:07:43.939241 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:43.939189 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ae44a8a-6af3-482e-9407-33e148efded7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ae44a8a-6af3-482e-9407-33e148efded7-workload-certs\") pod \"3ae44a8a-6af3-482e-9407-33e148efded7\" (UID: \"3ae44a8a-6af3-482e-9407-33e148efded7\") " Oct 30 19:07:43.939241 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:43.939241 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ae44a8a-6af3-482e-9407-33e148efded7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ae44a8a-6af3-482e-9407-33e148efded7-istio-envoy\") pod \"3ae44a8a-6af3-482e-9407-33e148efded7\" (UID: \"3ae44a8a-6af3-482e-9407-33e148efded7\") " Oct 30 19:07:43.939526 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:43.939394 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ae44a8a-6af3-482e-9407-33e148efded7-workload-certs" pod "3ae44a8a-6af3-482e-9407-33e148efded7" (UID: "3ae44a8a-6af3-482e-9407-33e148efded7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:07:43.940289 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:43.940263 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ae44a8a-6af3-482e-9407-33e148efded7-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:07:43.940429 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:43.940292 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae44a8a-6af3-482e-9407-33e148efded7-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:07:43.940429 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:43.940310 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ae44a8a-6af3-482e-9407-33e148efded7-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:07:43.940815 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:43.940790 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ae44a8a-6af3-482e-9407-33e148efded7-credential-socket" pod "3ae44a8a-6af3-482e-9407-33e148efded7" (UID: "3ae44a8a-6af3-482e-9407-33e148efded7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:07:43.941404 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:43.941375 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ae44a8a-6af3-482e-9407-33e148efded7-istiod-ca-cert" pod "3ae44a8a-6af3-482e-9407-33e148efded7" (UID: "3ae44a8a-6af3-482e-9407-33e148efded7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:07:43.943093 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:43.943056 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ae44a8a-6af3-482e-9407-33e148efded7-kube-api-access-96vsx" pod "3ae44a8a-6af3-482e-9407-33e148efded7" (UID: "3ae44a8a-6af3-482e-9407-33e148efded7"). InnerVolumeSpecName "kube-api-access-96vsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:07:43.943704 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:43.943667 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ae44a8a-6af3-482e-9407-33e148efded7-istio-token" pod "3ae44a8a-6af3-482e-9407-33e148efded7" (UID: "3ae44a8a-6af3-482e-9407-33e148efded7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:07:43.945168 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:43.945097 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ae44a8a-6af3-482e-9407-33e148efded7-istio-envoy" pod "3ae44a8a-6af3-482e-9407-33e148efded7" (UID: "3ae44a8a-6af3-482e-9407-33e148efded7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:07:43.945272 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:43.945118 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ae44a8a-6af3-482e-9407-33e148efded7-istio-podinfo" pod "3ae44a8a-6af3-482e-9407-33e148efded7" (UID: "3ae44a8a-6af3-482e-9407-33e148efded7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:07:44.042311 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:44.042231 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96vsx\" (UniqueName: \"kubernetes.io/projected/3ae44a8a-6af3-482e-9407-33e148efded7-kube-api-access-96vsx\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:07:44.042311 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:44.042269 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ae44a8a-6af3-482e-9407-33e148efded7-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:07:44.042311 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:44.042284 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ae44a8a-6af3-482e-9407-33e148efded7-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:07:44.042311 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:44.042297 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ae44a8a-6af3-482e-9407-33e148efded7-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:07:44.042311 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:44.042310 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ae44a8a-6af3-482e-9407-33e148efded7-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:07:44.042655 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:44.042322 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae44a8a-6af3-482e-9407-33e148efded7-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:07:44.807243 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:44.807216 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" Oct 30 19:07:44.807716 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:44.807256 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg" event={"ID":"3ae44a8a-6af3-482e-9407-33e148efded7","Type":"ContainerDied","Data":"19b67b3d10907e77ed1d5cefc78848a726c9f3bb42794da75e358e05dadf01dd"} Oct 30 19:07:44.807716 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:44.807315 4442 scope.go:122] "RemoveContainer" containerID="06a1e4039003f2f75279d7af0e79d0f43fe502b537a67af0009539f1b5fb6be6" Oct 30 19:07:44.877563 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:44.877518 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg"] Oct 30 19:07:44.883090 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:44.883066 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-xqplg"] Oct 30 19:07:45.067078 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:45.066994 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ae44a8a-6af3-482e-9407-33e148efded7" path="/var/lib/kubelet/pods/3ae44a8a-6af3-482e-9407-33e148efded7/volumes" Oct 30 19:07:45.591692 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:45.591654 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2107d4-7588cd98bc-nt76m" podUID="d147fcbd-d2ce-4a51-a8f1-531db7bcf8fc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:07:45.591875 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:45.591654 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-2107d4-7bf5b8d588-hz7nk" podUID="017e9c94-87b9-4143-bd73-311bb6edbed9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:07:46.523485 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:46.523452 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2107d4-7bf5b8d588-hz7nk" Oct 30 19:07:46.594562 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:46.594533 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2107d4-7588cd98bc-nt76m" Oct 30 19:07:46.693318 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:46.693287 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d147fcbd-d2ce-4a51-a8f1-531db7bcf8fc-kube-api-access-vjvrz\" (UniqueName: \"kubernetes.io/projected/d147fcbd-d2ce-4a51-a8f1-531db7bcf8fc-kube-api-access-vjvrz\") pod \"d147fcbd-d2ce-4a51-a8f1-531db7bcf8fc\" (UID: \"d147fcbd-d2ce-4a51-a8f1-531db7bcf8fc\") " Oct 30 19:07:46.693514 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:46.693417 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/017e9c94-87b9-4143-bd73-311bb6edbed9-kube-api-access-bzc5p\" (UniqueName: \"kubernetes.io/projected/017e9c94-87b9-4143-bd73-311bb6edbed9-kube-api-access-bzc5p\") pod \"017e9c94-87b9-4143-bd73-311bb6edbed9\" (UID: \"017e9c94-87b9-4143-bd73-311bb6edbed9\") " Oct 30 19:07:46.693591 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:46.693567 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/017e9c94-87b9-4143-bd73-311bb6edbed9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/017e9c94-87b9-4143-bd73-311bb6edbed9-plugins-config-volume\") pod \"017e9c94-87b9-4143-bd73-311bb6edbed9\" (UID: \"017e9c94-87b9-4143-bd73-311bb6edbed9\") " Oct 30 19:07:46.693654 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:46.693627 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d147fcbd-d2ce-4a51-a8f1-531db7bcf8fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d147fcbd-d2ce-4a51-a8f1-531db7bcf8fc-plugins-config-volume\") pod \"d147fcbd-d2ce-4a51-a8f1-531db7bcf8fc\" (UID: \"d147fcbd-d2ce-4a51-a8f1-531db7bcf8fc\") " Oct 30 19:07:46.694034 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:46.694001 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/017e9c94-87b9-4143-bd73-311bb6edbed9-plugins-config-volume" pod "017e9c94-87b9-4143-bd73-311bb6edbed9" (UID: "017e9c94-87b9-4143-bd73-311bb6edbed9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:07:46.694356 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:46.694214 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d147fcbd-d2ce-4a51-a8f1-531db7bcf8fc-plugins-config-volume" pod "d147fcbd-d2ce-4a51-a8f1-531db7bcf8fc" (UID: "d147fcbd-d2ce-4a51-a8f1-531db7bcf8fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:07:46.694356 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:46.694231 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/017e9c94-87b9-4143-bd73-311bb6edbed9-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:07:46.695712 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:46.695680 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d147fcbd-d2ce-4a51-a8f1-531db7bcf8fc-kube-api-access-vjvrz" pod "d147fcbd-d2ce-4a51-a8f1-531db7bcf8fc" (UID: "d147fcbd-d2ce-4a51-a8f1-531db7bcf8fc"). InnerVolumeSpecName "kube-api-access-vjvrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:07:46.695823 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:46.695758 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/017e9c94-87b9-4143-bd73-311bb6edbed9-kube-api-access-bzc5p" pod "017e9c94-87b9-4143-bd73-311bb6edbed9" (UID: "017e9c94-87b9-4143-bd73-311bb6edbed9"). InnerVolumeSpecName "kube-api-access-bzc5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:07:46.796204 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:46.796095 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjvrz\" (UniqueName: \"kubernetes.io/projected/d147fcbd-d2ce-4a51-a8f1-531db7bcf8fc-kube-api-access-vjvrz\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:07:46.796204 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:46.796125 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzc5p\" (UniqueName: \"kubernetes.io/projected/017e9c94-87b9-4143-bd73-311bb6edbed9-kube-api-access-bzc5p\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:07:46.796204 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:46.796170 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d147fcbd-d2ce-4a51-a8f1-531db7bcf8fc-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:07:46.826620 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:46.826590 4442 generic.go:356] "Generic (PLEG): container finished" podID="017e9c94-87b9-4143-bd73-311bb6edbed9" containerID="8d4456cf1413d30fb074e5d95ebc5e0dad05102ad7f3e9eae250d3ffadcb86c0" exitCode=0 Oct 30 19:07:46.826821 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:46.826643 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2107d4-7bf5b8d588-hz7nk" Oct 30 19:07:46.826821 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:46.826680 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2107d4-7bf5b8d588-hz7nk" event={"ID":"017e9c94-87b9-4143-bd73-311bb6edbed9","Type":"ContainerDied","Data":"8d4456cf1413d30fb074e5d95ebc5e0dad05102ad7f3e9eae250d3ffadcb86c0"} Oct 30 19:07:46.826821 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:46.826750 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2107d4-7bf5b8d588-hz7nk" event={"ID":"017e9c94-87b9-4143-bd73-311bb6edbed9","Type":"ContainerDied","Data":"9a22a225a7c094903a78c9b13e550146070e46f658cb76b14b72305384cc80fa"} Oct 30 19:07:46.826821 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:46.826781 4442 scope.go:122] "RemoveContainer" containerID="8d4456cf1413d30fb074e5d95ebc5e0dad05102ad7f3e9eae250d3ffadcb86c0" Oct 30 19:07:46.828877 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:46.828845 4442 generic.go:356] "Generic (PLEG): container finished" podID="d147fcbd-d2ce-4a51-a8f1-531db7bcf8fc" containerID="7b49cf6f38e1fa31410e1fcc6ed89a4499a6a0c27b0b4473a619bdfe99821767" exitCode=0 Oct 30 19:07:46.829021 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:46.828910 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2107d4-7588cd98bc-nt76m" Oct 30 19:07:46.829021 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:46.828933 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2107d4-7588cd98bc-nt76m" event={"ID":"d147fcbd-d2ce-4a51-a8f1-531db7bcf8fc","Type":"ContainerDied","Data":"7b49cf6f38e1fa31410e1fcc6ed89a4499a6a0c27b0b4473a619bdfe99821767"} Oct 30 19:07:46.829021 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:46.828998 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2107d4-7588cd98bc-nt76m" event={"ID":"d147fcbd-d2ce-4a51-a8f1-531db7bcf8fc","Type":"ContainerDied","Data":"c6bf26dbfaac263e38ee2ceae53091f40135eb106a703154a8e41179c174e37a"} Oct 30 19:07:46.840862 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:46.840805 4442 scope.go:122] "RemoveContainer" containerID="8d4456cf1413d30fb074e5d95ebc5e0dad05102ad7f3e9eae250d3ffadcb86c0" Oct 30 19:07:46.841536 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:07:46.841502 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d4456cf1413d30fb074e5d95ebc5e0dad05102ad7f3e9eae250d3ffadcb86c0\": container with ID starting with 8d4456cf1413d30fb074e5d95ebc5e0dad05102ad7f3e9eae250d3ffadcb86c0 not found: ID does not exist" containerID="8d4456cf1413d30fb074e5d95ebc5e0dad05102ad7f3e9eae250d3ffadcb86c0" Oct 30 19:07:46.841647 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:46.841535 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d4456cf1413d30fb074e5d95ebc5e0dad05102ad7f3e9eae250d3ffadcb86c0"} err="failed to get container status \"8d4456cf1413d30fb074e5d95ebc5e0dad05102ad7f3e9eae250d3ffadcb86c0\": rpc error: code = NotFound desc = could not find container \"8d4456cf1413d30fb074e5d95ebc5e0dad05102ad7f3e9eae250d3ffadcb86c0\": container with ID starting with 8d4456cf1413d30fb074e5d95ebc5e0dad05102ad7f3e9eae250d3ffadcb86c0 not found: ID does not exist" Oct 30 19:07:46.841647 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:46.841557 4442 scope.go:122] "RemoveContainer" containerID="7b49cf6f38e1fa31410e1fcc6ed89a4499a6a0c27b0b4473a619bdfe99821767" Oct 30 19:07:46.854830 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:46.854809 4442 scope.go:122] "RemoveContainer" containerID="7b49cf6f38e1fa31410e1fcc6ed89a4499a6a0c27b0b4473a619bdfe99821767" Oct 30 19:07:46.855109 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:07:46.855087 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b49cf6f38e1fa31410e1fcc6ed89a4499a6a0c27b0b4473a619bdfe99821767\": container with ID starting with 7b49cf6f38e1fa31410e1fcc6ed89a4499a6a0c27b0b4473a619bdfe99821767 not found: ID does not exist" containerID="7b49cf6f38e1fa31410e1fcc6ed89a4499a6a0c27b0b4473a619bdfe99821767" Oct 30 19:07:46.855191 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:46.855117 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b49cf6f38e1fa31410e1fcc6ed89a4499a6a0c27b0b4473a619bdfe99821767"} err="failed to get container status \"7b49cf6f38e1fa31410e1fcc6ed89a4499a6a0c27b0b4473a619bdfe99821767\": rpc error: code = NotFound desc = could not find container \"7b49cf6f38e1fa31410e1fcc6ed89a4499a6a0c27b0b4473a619bdfe99821767\": container with ID starting with 7b49cf6f38e1fa31410e1fcc6ed89a4499a6a0c27b0b4473a619bdfe99821767 not found: ID does not exist" Oct 30 19:07:46.879445 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:46.879419 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2107d4-7bf5b8d588-hz7nk"] Oct 30 19:07:46.883787 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:46.883766 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2107d4-7bf5b8d588-hz7nk"] Oct 30 19:07:46.915633 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:46.915607 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2107d4-7588cd98bc-nt76m"] Oct 30 19:07:46.920927 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:46.920898 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2107d4-7588cd98bc-nt76m"] Oct 30 19:07:47.070320 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:47.070223 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="017e9c94-87b9-4143-bd73-311bb6edbed9" path="/var/lib/kubelet/pods/017e9c94-87b9-4143-bd73-311bb6edbed9/volumes" Oct 30 19:07:47.071103 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:47.071079 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d147fcbd-d2ce-4a51-a8f1-531db7bcf8fc" path="/var/lib/kubelet/pods/d147fcbd-d2ce-4a51-a8f1-531db7bcf8fc/volumes" Oct 30 19:07:47.563144 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:07:47.563086 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 19:07:49.639696 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:49.639657 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6018b4-6868d8bb99-r5frb"] Oct 30 19:07:54.640895 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:54.640857 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6018b4-6868d8bb99-r5frb" podUID="d6601257-a86e-414c-9b71-9dc6042b6818" containerName="payload-pre-processing" containerID="cri-o://a2a65a256d273f1db4cbce27ba4f6d5953e44622ebe1cec1c7c05cb6e4374102" gracePeriod=25 Oct 30 19:07:56.878568 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:07:56.878535 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6018b4-6868d8bb99-r5frb" podUID="d6601257-a86e-414c-9b71-9dc6042b6818" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:08:04.501192 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:04.501080 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2107d4-84dfc46fbf-wz2h2"] Oct 30 19:08:04.501662 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:04.501341 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2107d4-84dfc46fbf-wz2h2" podUID="83d795bd-5ce3-4648-8823-be54dddc5580" containerName="maas-api" containerID="cri-o://fd9e8d8fc7b6e887d23f7d65009a58aff70607e056ddf705848e94d6b25190ac" gracePeriod=30 Oct 30 19:08:05.034370 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:05.034331 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2107d4-84dfc46fbf-wz2h2" Oct 30 19:08:05.168165 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:05.167851 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83d795bd-5ce3-4648-8823-be54dddc5580-metrics-certs\" (UniqueName: \"kubernetes.io/secret/83d795bd-5ce3-4648-8823-be54dddc5580-metrics-certs\") pod \"83d795bd-5ce3-4648-8823-be54dddc5580\" (UID: \"83d795bd-5ce3-4648-8823-be54dddc5580\") " Oct 30 19:08:05.168165 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:05.168009 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83d795bd-5ce3-4648-8823-be54dddc5580-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83d795bd-5ce3-4648-8823-be54dddc5580-openshift-service-ca\") pod \"83d795bd-5ce3-4648-8823-be54dddc5580\" (UID: \"83d795bd-5ce3-4648-8823-be54dddc5580\") " Oct 30 19:08:05.168165 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:05.168039 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83d795bd-5ce3-4648-8823-be54dddc5580-kube-api-access-zvd8q\" (UniqueName: \"kubernetes.io/projected/83d795bd-5ce3-4648-8823-be54dddc5580-kube-api-access-zvd8q\") pod \"83d795bd-5ce3-4648-8823-be54dddc5580\" (UID: \"83d795bd-5ce3-4648-8823-be54dddc5580\") " Oct 30 19:08:05.168497 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:05.168231 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83d795bd-5ce3-4648-8823-be54dddc5580-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83d795bd-5ce3-4648-8823-be54dddc5580-maas-api-tls\") pod \"83d795bd-5ce3-4648-8823-be54dddc5580\" (UID: \"83d795bd-5ce3-4648-8823-be54dddc5580\") " Oct 30 19:08:05.217897 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:05.185532 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83d795bd-5ce3-4648-8823-be54dddc5580-openshift-service-ca" pod "83d795bd-5ce3-4648-8823-be54dddc5580" (UID: "83d795bd-5ce3-4648-8823-be54dddc5580"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:08:05.273753 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:05.273597 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83d795bd-5ce3-4648-8823-be54dddc5580-metrics-certs" pod "83d795bd-5ce3-4648-8823-be54dddc5580" (UID: "83d795bd-5ce3-4648-8823-be54dddc5580"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:08:05.291549 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:05.291522 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83d795bd-5ce3-4648-8823-be54dddc5580-openshift-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:08:05.298092 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:05.298067 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83d795bd-5ce3-4648-8823-be54dddc5580-metrics-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:08:05.302372 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:05.302339 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83d795bd-5ce3-4648-8823-be54dddc5580-kube-api-access-zvd8q" pod "83d795bd-5ce3-4648-8823-be54dddc5580" (UID: "83d795bd-5ce3-4648-8823-be54dddc5580"). InnerVolumeSpecName "kube-api-access-zvd8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:08:05.302811 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:05.302783 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83d795bd-5ce3-4648-8823-be54dddc5580-maas-api-tls" pod "83d795bd-5ce3-4648-8823-be54dddc5580" (UID: "83d795bd-5ce3-4648-8823-be54dddc5580"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:08:05.313431 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:05.313399 4442 generic.go:356] "Generic (PLEG): container finished" podID="83d795bd-5ce3-4648-8823-be54dddc5580" containerID="fd9e8d8fc7b6e887d23f7d65009a58aff70607e056ddf705848e94d6b25190ac" exitCode=0 Oct 30 19:08:05.313799 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:05.313783 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2107d4-84dfc46fbf-wz2h2" Oct 30 19:08:05.317807 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:05.317774 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2107d4-84dfc46fbf-wz2h2" event={"ID":"83d795bd-5ce3-4648-8823-be54dddc5580","Type":"ContainerDied","Data":"fd9e8d8fc7b6e887d23f7d65009a58aff70607e056ddf705848e94d6b25190ac"} Oct 30 19:08:05.318010 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:05.317990 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2107d4-84dfc46fbf-wz2h2" event={"ID":"83d795bd-5ce3-4648-8823-be54dddc5580","Type":"ContainerDied","Data":"91350d52495f1ff61da61ca51a678ad5829a972b559dea0a43ba5885d9682873"} Oct 30 19:08:05.318175 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:05.318159 4442 scope.go:122] "RemoveContainer" containerID="fd9e8d8fc7b6e887d23f7d65009a58aff70607e056ddf705848e94d6b25190ac" Oct 30 19:08:05.337714 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:05.337607 4442 scope.go:122] "RemoveContainer" containerID="fd9e8d8fc7b6e887d23f7d65009a58aff70607e056ddf705848e94d6b25190ac" Oct 30 19:08:05.338613 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:08:05.338115 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd9e8d8fc7b6e887d23f7d65009a58aff70607e056ddf705848e94d6b25190ac\": container with ID starting with fd9e8d8fc7b6e887d23f7d65009a58aff70607e056ddf705848e94d6b25190ac not found: ID does not exist" containerID="fd9e8d8fc7b6e887d23f7d65009a58aff70607e056ddf705848e94d6b25190ac" Oct 30 19:08:05.338613 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:05.338585 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd9e8d8fc7b6e887d23f7d65009a58aff70607e056ddf705848e94d6b25190ac"} err="failed to get container status \"fd9e8d8fc7b6e887d23f7d65009a58aff70607e056ddf705848e94d6b25190ac\": rpc error: code = NotFound desc = could not find container \"fd9e8d8fc7b6e887d23f7d65009a58aff70607e056ddf705848e94d6b25190ac\": container with ID starting with fd9e8d8fc7b6e887d23f7d65009a58aff70607e056ddf705848e94d6b25190ac not found: ID does not exist" Oct 30 19:08:05.371069 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:05.371035 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2107d4-84dfc46fbf-wz2h2"] Oct 30 19:08:05.375293 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:05.375260 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2107d4-84dfc46fbf-wz2h2"] Oct 30 19:08:05.403841 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:05.403806 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvd8q\" (UniqueName: \"kubernetes.io/projected/83d795bd-5ce3-4648-8823-be54dddc5580-kube-api-access-zvd8q\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:08:05.403841 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:05.403836 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83d795bd-5ce3-4648-8823-be54dddc5580-maas-api-tls\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:08:06.878578 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:06.878544 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6018b4-6868d8bb99-r5frb" podUID="d6601257-a86e-414c-9b71-9dc6042b6818" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:08:07.066687 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:07.066648 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83d795bd-5ce3-4648-8823-be54dddc5580" path="/var/lib/kubelet/pods/83d795bd-5ce3-4648-8823-be54dddc5580/volumes" Oct 30 19:08:14.065807 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:14.065751 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk"] Oct 30 19:08:14.066498 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:14.066402 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" podUID="c2be17bb-1852-432e-a24d-24d762b27661" containerName="istio-proxy" containerID="cri-o://0aa40967c7cfb14bbce349d6d9cc1d59e670313aedc729a0e64390ea456b8ee8" gracePeriod=30 Oct 30 19:08:15.021921 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:15.021885 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6018b4-6868d8bb99-r5frb" Oct 30 19:08:15.103293 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:15.103258 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6601257-a86e-414c-9b71-9dc6042b6818-kube-api-access-b56jm\" (UniqueName: \"kubernetes.io/projected/d6601257-a86e-414c-9b71-9dc6042b6818-kube-api-access-b56jm\") pod \"d6601257-a86e-414c-9b71-9dc6042b6818\" (UID: \"d6601257-a86e-414c-9b71-9dc6042b6818\") " Oct 30 19:08:15.103740 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:15.103506 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6601257-a86e-414c-9b71-9dc6042b6818-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6601257-a86e-414c-9b71-9dc6042b6818-plugins-config-volume\") pod \"d6601257-a86e-414c-9b71-9dc6042b6818\" (UID: \"d6601257-a86e-414c-9b71-9dc6042b6818\") " Oct 30 19:08:15.104016 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:15.103983 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6601257-a86e-414c-9b71-9dc6042b6818-plugins-config-volume" pod "d6601257-a86e-414c-9b71-9dc6042b6818" (UID: "d6601257-a86e-414c-9b71-9dc6042b6818"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:08:15.104190 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:15.104159 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6601257-a86e-414c-9b71-9dc6042b6818-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:08:15.106012 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:15.105991 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6601257-a86e-414c-9b71-9dc6042b6818-kube-api-access-b56jm" pod "d6601257-a86e-414c-9b71-9dc6042b6818" (UID: "d6601257-a86e-414c-9b71-9dc6042b6818"). InnerVolumeSpecName "kube-api-access-b56jm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:08:15.205773 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:15.205733 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b56jm\" (UniqueName: \"kubernetes.io/projected/d6601257-a86e-414c-9b71-9dc6042b6818-kube-api-access-b56jm\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:08:15.437288 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:15.437248 4442 generic.go:356] "Generic (PLEG): container finished" podID="d6601257-a86e-414c-9b71-9dc6042b6818" containerID="a2a65a256d273f1db4cbce27ba4f6d5953e44622ebe1cec1c7c05cb6e4374102" exitCode=0 Oct 30 19:08:15.437492 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:15.437332 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6018b4-6868d8bb99-r5frb" Oct 30 19:08:15.437492 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:15.437341 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6018b4-6868d8bb99-r5frb" event={"ID":"d6601257-a86e-414c-9b71-9dc6042b6818","Type":"ContainerDied","Data":"a2a65a256d273f1db4cbce27ba4f6d5953e44622ebe1cec1c7c05cb6e4374102"} Oct 30 19:08:15.437492 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:15.437403 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6018b4-6868d8bb99-r5frb" event={"ID":"d6601257-a86e-414c-9b71-9dc6042b6818","Type":"ContainerDied","Data":"11773a5d1610bee4365d9e3b10d1a258d535823d6d34ad1ab0fd88a5ff3333a3"} Oct 30 19:08:15.437492 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:15.437429 4442 scope.go:122] "RemoveContainer" containerID="a2a65a256d273f1db4cbce27ba4f6d5953e44622ebe1cec1c7c05cb6e4374102" Oct 30 19:08:15.451298 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:15.451274 4442 scope.go:122] "RemoveContainer" containerID="a2a65a256d273f1db4cbce27ba4f6d5953e44622ebe1cec1c7c05cb6e4374102" Oct 30 19:08:15.451659 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:08:15.451612 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2a65a256d273f1db4cbce27ba4f6d5953e44622ebe1cec1c7c05cb6e4374102\": container with ID starting with a2a65a256d273f1db4cbce27ba4f6d5953e44622ebe1cec1c7c05cb6e4374102 not found: ID does not exist" containerID="a2a65a256d273f1db4cbce27ba4f6d5953e44622ebe1cec1c7c05cb6e4374102" Oct 30 19:08:15.451753 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:15.451653 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2a65a256d273f1db4cbce27ba4f6d5953e44622ebe1cec1c7c05cb6e4374102"} err="failed to get container status \"a2a65a256d273f1db4cbce27ba4f6d5953e44622ebe1cec1c7c05cb6e4374102\": rpc error: code = NotFound desc = could not find container \"a2a65a256d273f1db4cbce27ba4f6d5953e44622ebe1cec1c7c05cb6e4374102\": container with ID starting with a2a65a256d273f1db4cbce27ba4f6d5953e44622ebe1cec1c7c05cb6e4374102 not found: ID does not exist" Oct 30 19:08:15.491112 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:15.491076 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6018b4-6868d8bb99-r5frb"] Oct 30 19:08:15.502730 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:15.502695 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6018b4-6868d8bb99-r5frb"] Oct 30 19:08:17.071438 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:17.071399 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6601257-a86e-414c-9b71-9dc6042b6818" path="/var/lib/kubelet/pods/d6601257-a86e-414c-9b71-9dc6042b6818/volumes" Oct 30 19:08:17.567345 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:08:17.567310 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 19:08:19.370328 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:19.370300 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" Oct 30 19:08:19.472636 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:19.472597 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2be17bb-1852-432e-a24d-24d762b27661-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2be17bb-1852-432e-a24d-24d762b27661-istiod-ca-cert\") pod \"c2be17bb-1852-432e-a24d-24d762b27661\" (UID: \"c2be17bb-1852-432e-a24d-24d762b27661\") " Oct 30 19:08:19.472900 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:19.472668 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2be17bb-1852-432e-a24d-24d762b27661-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2be17bb-1852-432e-a24d-24d762b27661-istio-data\") pod \"c2be17bb-1852-432e-a24d-24d762b27661\" (UID: \"c2be17bb-1852-432e-a24d-24d762b27661\") " Oct 30 19:08:19.472900 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:19.472706 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2be17bb-1852-432e-a24d-24d762b27661-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2be17bb-1852-432e-a24d-24d762b27661-istio-envoy\") pod \"c2be17bb-1852-432e-a24d-24d762b27661\" (UID: \"c2be17bb-1852-432e-a24d-24d762b27661\") " Oct 30 19:08:19.472900 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:19.472845 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2be17bb-1852-432e-a24d-24d762b27661-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2be17bb-1852-432e-a24d-24d762b27661-istio-podinfo\") pod \"c2be17bb-1852-432e-a24d-24d762b27661\" (UID: \"c2be17bb-1852-432e-a24d-24d762b27661\") " Oct 30 19:08:19.473116 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:19.472968 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2be17bb-1852-432e-a24d-24d762b27661-kube-api-access-8mrw8\" (UniqueName: \"kubernetes.io/projected/c2be17bb-1852-432e-a24d-24d762b27661-kube-api-access-8mrw8\") pod \"c2be17bb-1852-432e-a24d-24d762b27661\" (UID: \"c2be17bb-1852-432e-a24d-24d762b27661\") " Oct 30 19:08:19.473116 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:19.473020 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2be17bb-1852-432e-a24d-24d762b27661-istio-token\" (UniqueName: \"kubernetes.io/projected/c2be17bb-1852-432e-a24d-24d762b27661-istio-token\") pod \"c2be17bb-1852-432e-a24d-24d762b27661\" (UID: \"c2be17bb-1852-432e-a24d-24d762b27661\") " Oct 30 19:08:19.473260 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:19.473209 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2be17bb-1852-432e-a24d-24d762b27661-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2be17bb-1852-432e-a24d-24d762b27661-credential-socket\") pod \"c2be17bb-1852-432e-a24d-24d762b27661\" (UID: \"c2be17bb-1852-432e-a24d-24d762b27661\") " Oct 30 19:08:19.473260 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:19.473244 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2be17bb-1852-432e-a24d-24d762b27661-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2be17bb-1852-432e-a24d-24d762b27661-workload-socket\") pod \"c2be17bb-1852-432e-a24d-24d762b27661\" (UID: \"c2be17bb-1852-432e-a24d-24d762b27661\") " Oct 30 19:08:19.473383 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:19.473323 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2be17bb-1852-432e-a24d-24d762b27661-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2be17bb-1852-432e-a24d-24d762b27661-workload-certs\") pod \"c2be17bb-1852-432e-a24d-24d762b27661\" (UID: \"c2be17bb-1852-432e-a24d-24d762b27661\") " Oct 30 19:08:19.474232 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:19.474206 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2be17bb-1852-432e-a24d-24d762b27661-workload-certs" pod "c2be17bb-1852-432e-a24d-24d762b27661" (UID: "c2be17bb-1852-432e-a24d-24d762b27661"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:19.474607 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:19.474581 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2be17bb-1852-432e-a24d-24d762b27661-credential-socket" pod "c2be17bb-1852-432e-a24d-24d762b27661" (UID: "c2be17bb-1852-432e-a24d-24d762b27661"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:19.474832 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:19.474800 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2be17bb-1852-432e-a24d-24d762b27661-istio-data" pod "c2be17bb-1852-432e-a24d-24d762b27661" (UID: "c2be17bb-1852-432e-a24d-24d762b27661"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:19.474939 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:19.474820 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2be17bb-1852-432e-a24d-24d762b27661-workload-socket" pod "c2be17bb-1852-432e-a24d-24d762b27661" (UID: "c2be17bb-1852-432e-a24d-24d762b27661"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:19.475234 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:19.475208 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2be17bb-1852-432e-a24d-24d762b27661-istiod-ca-cert" pod "c2be17bb-1852-432e-a24d-24d762b27661" (UID: "c2be17bb-1852-432e-a24d-24d762b27661"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:08:19.478463 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:19.478422 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2be17bb-1852-432e-a24d-24d762b27661-istio-podinfo" pod "c2be17bb-1852-432e-a24d-24d762b27661" (UID: "c2be17bb-1852-432e-a24d-24d762b27661"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:08:19.483023 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:19.482753 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2be17bb-1852-432e-a24d-24d762b27661-istio-envoy" pod "c2be17bb-1852-432e-a24d-24d762b27661" (UID: "c2be17bb-1852-432e-a24d-24d762b27661"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:19.483023 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:19.482831 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2be17bb-1852-432e-a24d-24d762b27661-kube-api-access-8mrw8" pod "c2be17bb-1852-432e-a24d-24d762b27661" (UID: "c2be17bb-1852-432e-a24d-24d762b27661"). InnerVolumeSpecName "kube-api-access-8mrw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:08:19.483023 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:19.482973 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2be17bb-1852-432e-a24d-24d762b27661-istio-token" pod "c2be17bb-1852-432e-a24d-24d762b27661" (UID: "c2be17bb-1852-432e-a24d-24d762b27661"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:08:19.483480 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:19.483448 4442 generic.go:356] "Generic (PLEG): container finished" podID="c2be17bb-1852-432e-a24d-24d762b27661" containerID="0aa40967c7cfb14bbce349d6d9cc1d59e670313aedc729a0e64390ea456b8ee8" exitCode=0 Oct 30 19:08:19.483556 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:19.483515 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" event={"ID":"c2be17bb-1852-432e-a24d-24d762b27661","Type":"ContainerDied","Data":"0aa40967c7cfb14bbce349d6d9cc1d59e670313aedc729a0e64390ea456b8ee8"} Oct 30 19:08:19.483656 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:19.483610 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" event={"ID":"c2be17bb-1852-432e-a24d-24d762b27661","Type":"ContainerDied","Data":"69e0e9d4bd0a42a95bfc128355b3f10c16beb93f3de33cdd16242d75b8957235"} Oct 30 19:08:19.483656 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:19.483644 4442 scope.go:122] "RemoveContainer" containerID="0aa40967c7cfb14bbce349d6d9cc1d59e670313aedc729a0e64390ea456b8ee8" Oct 30 19:08:19.483814 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:19.483798 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk" Oct 30 19:08:19.510531 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:19.510513 4442 scope.go:122] "RemoveContainer" containerID="5898fae1e077c4b36892e8caf35c10bcb3ca6c3a9ffe4ff6b724a2c19ca0413b" Oct 30 19:08:19.520967 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:19.520948 4442 scope.go:122] "RemoveContainer" containerID="0aa40967c7cfb14bbce349d6d9cc1d59e670313aedc729a0e64390ea456b8ee8" Oct 30 19:08:19.521287 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:08:19.521267 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aa40967c7cfb14bbce349d6d9cc1d59e670313aedc729a0e64390ea456b8ee8\": container with ID starting with 0aa40967c7cfb14bbce349d6d9cc1d59e670313aedc729a0e64390ea456b8ee8 not found: ID does not exist" containerID="0aa40967c7cfb14bbce349d6d9cc1d59e670313aedc729a0e64390ea456b8ee8" Oct 30 19:08:19.521351 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:19.521299 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aa40967c7cfb14bbce349d6d9cc1d59e670313aedc729a0e64390ea456b8ee8"} err="failed to get container status \"0aa40967c7cfb14bbce349d6d9cc1d59e670313aedc729a0e64390ea456b8ee8\": rpc error: code = NotFound desc = could not find container \"0aa40967c7cfb14bbce349d6d9cc1d59e670313aedc729a0e64390ea456b8ee8\": container with ID starting with 0aa40967c7cfb14bbce349d6d9cc1d59e670313aedc729a0e64390ea456b8ee8 not found: ID does not exist" Oct 30 19:08:19.521351 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:19.521316 4442 scope.go:122] "RemoveContainer" containerID="5898fae1e077c4b36892e8caf35c10bcb3ca6c3a9ffe4ff6b724a2c19ca0413b" Oct 30 19:08:19.521621 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:08:19.521602 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5898fae1e077c4b36892e8caf35c10bcb3ca6c3a9ffe4ff6b724a2c19ca0413b\": container with ID starting with 5898fae1e077c4b36892e8caf35c10bcb3ca6c3a9ffe4ff6b724a2c19ca0413b not found: ID does not exist" containerID="5898fae1e077c4b36892e8caf35c10bcb3ca6c3a9ffe4ff6b724a2c19ca0413b" Oct 30 19:08:19.521680 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:19.521627 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5898fae1e077c4b36892e8caf35c10bcb3ca6c3a9ffe4ff6b724a2c19ca0413b"} err="failed to get container status \"5898fae1e077c4b36892e8caf35c10bcb3ca6c3a9ffe4ff6b724a2c19ca0413b\": rpc error: code = NotFound desc = could not find container \"5898fae1e077c4b36892e8caf35c10bcb3ca6c3a9ffe4ff6b724a2c19ca0413b\": container with ID starting with 5898fae1e077c4b36892e8caf35c10bcb3ca6c3a9ffe4ff6b724a2c19ca0413b not found: ID does not exist" Oct 30 19:08:19.548603 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:19.548562 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk"] Oct 30 19:08:19.555037 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:19.555004 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-2107d4-openshift-default-659d97db85-l4lxk"] Oct 30 19:08:19.575139 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:19.575110 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2be17bb-1852-432e-a24d-24d762b27661-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:08:19.575319 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:19.575154 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2be17bb-1852-432e-a24d-24d762b27661-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:08:19.575319 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:19.575166 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2be17bb-1852-432e-a24d-24d762b27661-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:08:19.575319 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:19.575176 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2be17bb-1852-432e-a24d-24d762b27661-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:08:19.575319 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:19.575184 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2be17bb-1852-432e-a24d-24d762b27661-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:08:19.575319 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:19.575191 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2be17bb-1852-432e-a24d-24d762b27661-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:08:19.575319 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:19.575199 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2be17bb-1852-432e-a24d-24d762b27661-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:08:19.575319 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:19.575206 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mrw8\" (UniqueName: \"kubernetes.io/projected/c2be17bb-1852-432e-a24d-24d762b27661-kube-api-access-8mrw8\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:08:19.575319 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:19.575215 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2be17bb-1852-432e-a24d-24d762b27661-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:08:21.068644 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:21.068611 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2be17bb-1852-432e-a24d-24d762b27661" path="/var/lib/kubelet/pods/c2be17bb-1852-432e-a24d-24d762b27661/volumes" Oct 30 19:08:24.261362 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:24.261331 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52rn6_516c6b6c-fc40-49cf-93b5-1c4471b02cf0/ovn-acl-logging/0.log" Oct 30 19:08:24.264414 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:24.264395 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52rn6_516c6b6c-fc40-49cf-93b5-1c4471b02cf0/ovn-acl-logging/0.log" Oct 30 19:08:25.211044 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:25.211006 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-38e0db-675d6bc647-sjkjz"] Oct 30 19:08:28.973461 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:28.973427 4442 scope.go:122] "RemoveContainer" containerID="d4f902c2ec0659eddeccc6c758cbac1304222baffd6508ab829a4f374f5b7111" Oct 30 19:08:28.984982 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:28.984962 4442 scope.go:122] "RemoveContainer" containerID="b9aa1eae205c029255b60967dd88f6f54a52a1f4822ae6a7b894d53c15b30c12" Oct 30 19:08:30.211732 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:30.211693 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-38e0db-675d6bc647-sjkjz" podUID="a2e19b96-f72b-4735-a1e8-a0058dc45470" containerName="payload-pre-processing" containerID="cri-o://18cb2c09ca4d7d17c1662ddb1e62e690b6bf6c8d4c0192b68eab987aa360554f" gracePeriod=25 Oct 30 19:08:33.799729 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:33.799687 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-38e0db-675d6bc647-sjkjz" podUID="a2e19b96-f72b-4735-a1e8-a0058dc45470" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:08:39.912462 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:39.912430 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm"] Oct 30 19:08:39.912968 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:39.912696 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" podUID="01d882ef-fc94-45cb-af2e-3debf4b418cd" containerName="istio-proxy" containerID="cri-o://03d19c61cf3557a6f3f873e1089d17461ff4270865d804cdded18e8df6030485" gracePeriod=30 Oct 30 19:08:43.799704 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:43.799664 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-38e0db-675d6bc647-sjkjz" podUID="a2e19b96-f72b-4735-a1e8-a0058dc45470" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:08:45.326772 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:45.326746 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" Oct 30 19:08:45.409388 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:45.409351 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01d882ef-fc94-45cb-af2e-3debf4b418cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01d882ef-fc94-45cb-af2e-3debf4b418cd-istio-envoy\") pod \"01d882ef-fc94-45cb-af2e-3debf4b418cd\" (UID: \"01d882ef-fc94-45cb-af2e-3debf4b418cd\") " Oct 30 19:08:45.409609 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:45.409411 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01d882ef-fc94-45cb-af2e-3debf4b418cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01d882ef-fc94-45cb-af2e-3debf4b418cd-istio-data\") pod \"01d882ef-fc94-45cb-af2e-3debf4b418cd\" (UID: \"01d882ef-fc94-45cb-af2e-3debf4b418cd\") " Oct 30 19:08:45.409609 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:45.409566 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01d882ef-fc94-45cb-af2e-3debf4b418cd-istio-token\" (UniqueName: \"kubernetes.io/projected/01d882ef-fc94-45cb-af2e-3debf4b418cd-istio-token\") pod \"01d882ef-fc94-45cb-af2e-3debf4b418cd\" (UID: \"01d882ef-fc94-45cb-af2e-3debf4b418cd\") " Oct 30 19:08:45.409747 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:45.409675 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01d882ef-fc94-45cb-af2e-3debf4b418cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01d882ef-fc94-45cb-af2e-3debf4b418cd-workload-certs\") pod \"01d882ef-fc94-45cb-af2e-3debf4b418cd\" (UID: \"01d882ef-fc94-45cb-af2e-3debf4b418cd\") " Oct 30 19:08:45.409810 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:45.409784 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01d882ef-fc94-45cb-af2e-3debf4b418cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01d882ef-fc94-45cb-af2e-3debf4b418cd-credential-socket\") pod \"01d882ef-fc94-45cb-af2e-3debf4b418cd\" (UID: \"01d882ef-fc94-45cb-af2e-3debf4b418cd\") " Oct 30 19:08:45.409869 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:45.409847 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01d882ef-fc94-45cb-af2e-3debf4b418cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01d882ef-fc94-45cb-af2e-3debf4b418cd-istio-podinfo\") pod \"01d882ef-fc94-45cb-af2e-3debf4b418cd\" (UID: \"01d882ef-fc94-45cb-af2e-3debf4b418cd\") " Oct 30 19:08:45.409928 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:45.409880 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01d882ef-fc94-45cb-af2e-3debf4b418cd-kube-api-access-jvpmj\" (UniqueName: \"kubernetes.io/projected/01d882ef-fc94-45cb-af2e-3debf4b418cd-kube-api-access-jvpmj\") pod \"01d882ef-fc94-45cb-af2e-3debf4b418cd\" (UID: \"01d882ef-fc94-45cb-af2e-3debf4b418cd\") " Oct 30 19:08:45.409980 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:45.409923 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01d882ef-fc94-45cb-af2e-3debf4b418cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01d882ef-fc94-45cb-af2e-3debf4b418cd-workload-socket\") pod \"01d882ef-fc94-45cb-af2e-3debf4b418cd\" (UID: \"01d882ef-fc94-45cb-af2e-3debf4b418cd\") " Oct 30 19:08:45.409980 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:45.409962 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01d882ef-fc94-45cb-af2e-3debf4b418cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01d882ef-fc94-45cb-af2e-3debf4b418cd-istiod-ca-cert\") pod \"01d882ef-fc94-45cb-af2e-3debf4b418cd\" (UID: \"01d882ef-fc94-45cb-af2e-3debf4b418cd\") " Oct 30 19:08:45.410952 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:45.410836 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01d882ef-fc94-45cb-af2e-3debf4b418cd-istiod-ca-cert" pod "01d882ef-fc94-45cb-af2e-3debf4b418cd" (UID: "01d882ef-fc94-45cb-af2e-3debf4b418cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:08:45.410952 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:45.410910 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01d882ef-fc94-45cb-af2e-3debf4b418cd-credential-socket" pod "01d882ef-fc94-45cb-af2e-3debf4b418cd" (UID: "01d882ef-fc94-45cb-af2e-3debf4b418cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:45.411227 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:45.411076 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01d882ef-fc94-45cb-af2e-3debf4b418cd-istio-data" pod "01d882ef-fc94-45cb-af2e-3debf4b418cd" (UID: "01d882ef-fc94-45cb-af2e-3debf4b418cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:45.411667 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:45.411603 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01d882ef-fc94-45cb-af2e-3debf4b418cd-workload-socket" pod "01d882ef-fc94-45cb-af2e-3debf4b418cd" (UID: "01d882ef-fc94-45cb-af2e-3debf4b418cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:45.411858 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:45.411667 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01d882ef-fc94-45cb-af2e-3debf4b418cd-workload-certs" pod "01d882ef-fc94-45cb-af2e-3debf4b418cd" (UID: "01d882ef-fc94-45cb-af2e-3debf4b418cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:45.413816 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:45.413780 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01d882ef-fc94-45cb-af2e-3debf4b418cd-istio-envoy" pod "01d882ef-fc94-45cb-af2e-3debf4b418cd" (UID: "01d882ef-fc94-45cb-af2e-3debf4b418cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:45.416380 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:45.416318 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01d882ef-fc94-45cb-af2e-3debf4b418cd-istio-token" pod "01d882ef-fc94-45cb-af2e-3debf4b418cd" (UID: "01d882ef-fc94-45cb-af2e-3debf4b418cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:08:45.416380 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:45.416358 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01d882ef-fc94-45cb-af2e-3debf4b418cd-kube-api-access-jvpmj" pod "01d882ef-fc94-45cb-af2e-3debf4b418cd" (UID: "01d882ef-fc94-45cb-af2e-3debf4b418cd"). InnerVolumeSpecName "kube-api-access-jvpmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:08:45.416920 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:45.416885 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01d882ef-fc94-45cb-af2e-3debf4b418cd-istio-podinfo" pod "01d882ef-fc94-45cb-af2e-3debf4b418cd" (UID: "01d882ef-fc94-45cb-af2e-3debf4b418cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:08:45.511986 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:45.511956 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01d882ef-fc94-45cb-af2e-3debf4b418cd-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:08:45.511986 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:45.511985 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01d882ef-fc94-45cb-af2e-3debf4b418cd-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:08:45.511986 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:45.511995 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01d882ef-fc94-45cb-af2e-3debf4b418cd-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:08:45.512263 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:45.512005 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01d882ef-fc94-45cb-af2e-3debf4b418cd-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:08:45.512263 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:45.512014 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvpmj\" (UniqueName: \"kubernetes.io/projected/01d882ef-fc94-45cb-af2e-3debf4b418cd-kube-api-access-jvpmj\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:08:45.512263 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:45.512022 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01d882ef-fc94-45cb-af2e-3debf4b418cd-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:08:45.512263 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:45.512031 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01d882ef-fc94-45cb-af2e-3debf4b418cd-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:08:45.512263 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:45.512040 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01d882ef-fc94-45cb-af2e-3debf4b418cd-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:08:45.512263 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:45.512048 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01d882ef-fc94-45cb-af2e-3debf4b418cd-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:08:45.744327 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:45.744290 4442 generic.go:356] "Generic (PLEG): container finished" podID="01d882ef-fc94-45cb-af2e-3debf4b418cd" containerID="03d19c61cf3557a6f3f873e1089d17461ff4270865d804cdded18e8df6030485" exitCode=0 Oct 30 19:08:45.744544 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:45.744367 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" Oct 30 19:08:45.744544 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:45.744378 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" event={"ID":"01d882ef-fc94-45cb-af2e-3debf4b418cd","Type":"ContainerDied","Data":"03d19c61cf3557a6f3f873e1089d17461ff4270865d804cdded18e8df6030485"} Oct 30 19:08:45.744544 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:45.744454 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm" event={"ID":"01d882ef-fc94-45cb-af2e-3debf4b418cd","Type":"ContainerDied","Data":"b3514419bf5716b45ef43d17ad06338bac35c07fb4bf25c0b81274c752ceab5a"} Oct 30 19:08:45.744544 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:45.744488 4442 scope.go:122] "RemoveContainer" containerID="03d19c61cf3557a6f3f873e1089d17461ff4270865d804cdded18e8df6030485" Oct 30 19:08:45.765749 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:45.765726 4442 scope.go:122] "RemoveContainer" containerID="ae5bbef007da2c1f37049e61f54a0811e527ad4d84d8c4557a21382b25156174" Oct 30 19:08:45.779629 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:45.779608 4442 scope.go:122] "RemoveContainer" containerID="03d19c61cf3557a6f3f873e1089d17461ff4270865d804cdded18e8df6030485" Oct 30 19:08:45.779935 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:08:45.779913 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03d19c61cf3557a6f3f873e1089d17461ff4270865d804cdded18e8df6030485\": container with ID starting with 03d19c61cf3557a6f3f873e1089d17461ff4270865d804cdded18e8df6030485 not found: ID does not exist" containerID="03d19c61cf3557a6f3f873e1089d17461ff4270865d804cdded18e8df6030485" Oct 30 19:08:45.779987 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:45.779945 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03d19c61cf3557a6f3f873e1089d17461ff4270865d804cdded18e8df6030485"} err="failed to get container status \"03d19c61cf3557a6f3f873e1089d17461ff4270865d804cdded18e8df6030485\": rpc error: code = NotFound desc = could not find container \"03d19c61cf3557a6f3f873e1089d17461ff4270865d804cdded18e8df6030485\": container with ID starting with 03d19c61cf3557a6f3f873e1089d17461ff4270865d804cdded18e8df6030485 not found: ID does not exist" Oct 30 19:08:45.779987 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:45.779962 4442 scope.go:122] "RemoveContainer" containerID="ae5bbef007da2c1f37049e61f54a0811e527ad4d84d8c4557a21382b25156174" Oct 30 19:08:45.780247 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:08:45.780227 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae5bbef007da2c1f37049e61f54a0811e527ad4d84d8c4557a21382b25156174\": container with ID starting with ae5bbef007da2c1f37049e61f54a0811e527ad4d84d8c4557a21382b25156174 not found: ID does not exist" containerID="ae5bbef007da2c1f37049e61f54a0811e527ad4d84d8c4557a21382b25156174" Oct 30 19:08:45.780306 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:45.780253 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae5bbef007da2c1f37049e61f54a0811e527ad4d84d8c4557a21382b25156174"} err="failed to get container status \"ae5bbef007da2c1f37049e61f54a0811e527ad4d84d8c4557a21382b25156174\": rpc error: code = NotFound desc = could not find container \"ae5bbef007da2c1f37049e61f54a0811e527ad4d84d8c4557a21382b25156174\": container with ID starting with ae5bbef007da2c1f37049e61f54a0811e527ad4d84d8c4557a21382b25156174 not found: ID does not exist" Oct 30 19:08:45.801690 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:45.801658 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm"] Oct 30 19:08:45.813461 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:45.813434 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-38e0db-openshift-default-559cb7d8df-6jhrm"] Oct 30 19:08:47.068801 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:47.068769 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01d882ef-fc94-45cb-af2e-3debf4b418cd" path="/var/lib/kubelet/pods/01d882ef-fc94-45cb-af2e-3debf4b418cd/volumes" Oct 30 19:08:47.560690 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:08:47.560656 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 19:08:50.535535 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:50.535501 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-38e0db-675d6bc647-sjkjz" Oct 30 19:08:50.584588 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:50.584558 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2e19b96-f72b-4735-a1e8-a0058dc45470-kube-api-access-cfhxm\" (UniqueName: \"kubernetes.io/projected/a2e19b96-f72b-4735-a1e8-a0058dc45470-kube-api-access-cfhxm\") pod \"a2e19b96-f72b-4735-a1e8-a0058dc45470\" (UID: \"a2e19b96-f72b-4735-a1e8-a0058dc45470\") " Oct 30 19:08:50.584797 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:50.584619 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2e19b96-f72b-4735-a1e8-a0058dc45470-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2e19b96-f72b-4735-a1e8-a0058dc45470-plugins-config-volume\") pod \"a2e19b96-f72b-4735-a1e8-a0058dc45470\" (UID: \"a2e19b96-f72b-4735-a1e8-a0058dc45470\") " Oct 30 19:08:50.585613 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:50.585584 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2e19b96-f72b-4735-a1e8-a0058dc45470-plugins-config-volume" pod "a2e19b96-f72b-4735-a1e8-a0058dc45470" (UID: "a2e19b96-f72b-4735-a1e8-a0058dc45470"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:08:50.587826 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:50.587799 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2e19b96-f72b-4735-a1e8-a0058dc45470-kube-api-access-cfhxm" pod "a2e19b96-f72b-4735-a1e8-a0058dc45470" (UID: "a2e19b96-f72b-4735-a1e8-a0058dc45470"). InnerVolumeSpecName "kube-api-access-cfhxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:08:50.687053 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:50.687012 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfhxm\" (UniqueName: \"kubernetes.io/projected/a2e19b96-f72b-4735-a1e8-a0058dc45470-kube-api-access-cfhxm\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:08:50.687053 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:50.687052 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2e19b96-f72b-4735-a1e8-a0058dc45470-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:08:50.791248 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:50.791126 4442 generic.go:356] "Generic (PLEG): container finished" podID="a2e19b96-f72b-4735-a1e8-a0058dc45470" containerID="18cb2c09ca4d7d17c1662ddb1e62e690b6bf6c8d4c0192b68eab987aa360554f" exitCode=0 Oct 30 19:08:50.791248 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:50.791208 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-38e0db-675d6bc647-sjkjz" Oct 30 19:08:50.791433 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:50.791248 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-38e0db-675d6bc647-sjkjz" event={"ID":"a2e19b96-f72b-4735-a1e8-a0058dc45470","Type":"ContainerDied","Data":"18cb2c09ca4d7d17c1662ddb1e62e690b6bf6c8d4c0192b68eab987aa360554f"} Oct 30 19:08:50.791433 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:50.791307 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-38e0db-675d6bc647-sjkjz" event={"ID":"a2e19b96-f72b-4735-a1e8-a0058dc45470","Type":"ContainerDied","Data":"dd10316c5ab967b0785b1cde308f94ec7c03bdb6c1d6efb829ac45ba202d17ad"} Oct 30 19:08:50.791433 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:50.791337 4442 scope.go:122] "RemoveContainer" containerID="18cb2c09ca4d7d17c1662ddb1e62e690b6bf6c8d4c0192b68eab987aa360554f" Oct 30 19:08:50.804628 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:50.804599 4442 scope.go:122] "RemoveContainer" containerID="18cb2c09ca4d7d17c1662ddb1e62e690b6bf6c8d4c0192b68eab987aa360554f" Oct 30 19:08:50.804990 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:08:50.804962 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18cb2c09ca4d7d17c1662ddb1e62e690b6bf6c8d4c0192b68eab987aa360554f\": container with ID starting with 18cb2c09ca4d7d17c1662ddb1e62e690b6bf6c8d4c0192b68eab987aa360554f not found: ID does not exist" containerID="18cb2c09ca4d7d17c1662ddb1e62e690b6bf6c8d4c0192b68eab987aa360554f" Oct 30 19:08:50.805091 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:50.805003 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18cb2c09ca4d7d17c1662ddb1e62e690b6bf6c8d4c0192b68eab987aa360554f"} err="failed to get container status \"18cb2c09ca4d7d17c1662ddb1e62e690b6bf6c8d4c0192b68eab987aa360554f\": rpc error: code = NotFound desc = could not find container \"18cb2c09ca4d7d17c1662ddb1e62e690b6bf6c8d4c0192b68eab987aa360554f\": container with ID starting with 18cb2c09ca4d7d17c1662ddb1e62e690b6bf6c8d4c0192b68eab987aa360554f not found: ID does not exist" Oct 30 19:08:50.844874 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:50.844823 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-38e0db-675d6bc647-sjkjz"] Oct 30 19:08:50.850490 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:50.850459 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-38e0db-675d6bc647-sjkjz"] Oct 30 19:08:51.115027 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:51.111909 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2e19b96-f72b-4735-a1e8-a0058dc45470" path="/var/lib/kubelet/pods/a2e19b96-f72b-4735-a1e8-a0058dc45470/volumes" Oct 30 19:08:51.859838 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:51.859800 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab244c-7659db958c-jbckm"] Oct 30 19:08:53.867725 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:53.867694 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq"] Oct 30 19:08:53.869206 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:53.869175 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" podUID="eb94bb4c-aa2a-4c61-b986-03dd313ab6e0" containerName="istio-proxy" containerID="cri-o://3d90cdd69073ad22d74141faf68bbc51694ef320f274e2d726548b4d4d29dd16" gracePeriod=30 Oct 30 19:08:56.860224 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:56.860184 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab244c-7659db958c-jbckm" podUID="a1352e66-acc1-499e-8344-4cab5734f78a" containerName="payload-pre-processing" containerID="cri-o://c9e6bda211e4c3e7ae4b6d0e8d2cf75080afa9d1c0858c5baa595dfaae447bf9" gracePeriod=25 Oct 30 19:08:59.232598 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:59.232571 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" Oct 30 19:08:59.359911 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:59.359875 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-credential-socket\") pod \"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0\" (UID: \"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0\") " Oct 30 19:08:59.360115 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:59.359938 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-workload-certs\") pod \"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0\" (UID: \"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0\") " Oct 30 19:08:59.360115 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:59.359982 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-istio-envoy\") pod \"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0\" (UID: \"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0\") " Oct 30 19:08:59.360115 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:59.360003 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-istio-podinfo\") pod \"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0\" (UID: \"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0\") " Oct 30 19:08:59.360115 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:59.360045 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-kube-api-access-cfl4f\" (UniqueName: \"kubernetes.io/projected/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-kube-api-access-cfl4f\") pod \"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0\" (UID: \"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0\") " Oct 30 19:08:59.360390 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:59.360124 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-istio-data\") pod \"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0\" (UID: \"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0\") " Oct 30 19:08:59.360390 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:59.360198 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-workload-socket\") pod \"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0\" (UID: \"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0\") " Oct 30 19:08:59.360390 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:59.360229 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-istio-token\" (UniqueName: \"kubernetes.io/projected/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-istio-token\") pod \"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0\" (UID: \"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0\") " Oct 30 19:08:59.360390 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:59.360244 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-credential-socket" pod "eb94bb4c-aa2a-4c61-b986-03dd313ab6e0" (UID: "eb94bb4c-aa2a-4c61-b986-03dd313ab6e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:59.360390 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:59.360351 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-workload-certs" pod "eb94bb4c-aa2a-4c61-b986-03dd313ab6e0" (UID: "eb94bb4c-aa2a-4c61-b986-03dd313ab6e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:59.360390 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:59.360374 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-istiod-ca-cert\") pod \"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0\" (UID: \"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0\") " Oct 30 19:08:59.360758 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:59.360737 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-istiod-ca-cert" pod "eb94bb4c-aa2a-4c61-b986-03dd313ab6e0" (UID: "eb94bb4c-aa2a-4c61-b986-03dd313ab6e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:08:59.360849 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:59.360820 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-istio-data" pod "eb94bb4c-aa2a-4c61-b986-03dd313ab6e0" (UID: "eb94bb4c-aa2a-4c61-b986-03dd313ab6e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:59.360973 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:59.360956 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:08:59.361034 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:59.360976 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:08:59.361034 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:59.360985 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:08:59.361034 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:59.360996 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:08:59.361262 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:59.361036 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-workload-socket" pod "eb94bb4c-aa2a-4c61-b986-03dd313ab6e0" (UID: "eb94bb4c-aa2a-4c61-b986-03dd313ab6e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:59.363044 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:59.363015 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-istio-podinfo" pod "eb94bb4c-aa2a-4c61-b986-03dd313ab6e0" (UID: "eb94bb4c-aa2a-4c61-b986-03dd313ab6e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:08:59.363234 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:59.363087 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-istio-token" pod "eb94bb4c-aa2a-4c61-b986-03dd313ab6e0" (UID: "eb94bb4c-aa2a-4c61-b986-03dd313ab6e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:08:59.363495 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:59.363474 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-istio-envoy" pod "eb94bb4c-aa2a-4c61-b986-03dd313ab6e0" (UID: "eb94bb4c-aa2a-4c61-b986-03dd313ab6e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:59.363495 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:59.363484 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-kube-api-access-cfl4f" pod "eb94bb4c-aa2a-4c61-b986-03dd313ab6e0" (UID: "eb94bb4c-aa2a-4c61-b986-03dd313ab6e0"). InnerVolumeSpecName "kube-api-access-cfl4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:08:59.462187 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:59.462072 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:08:59.462187 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:59.462101 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:08:59.462187 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:59.462109 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:08:59.462187 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:59.462117 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:08:59.462187 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:59.462126 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfl4f\" (UniqueName: \"kubernetes.io/projected/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0-kube-api-access-cfl4f\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:08:59.876881 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:59.876848 4442 generic.go:356] "Generic (PLEG): container finished" podID="eb94bb4c-aa2a-4c61-b986-03dd313ab6e0" containerID="3d90cdd69073ad22d74141faf68bbc51694ef320f274e2d726548b4d4d29dd16" exitCode=0 Oct 30 19:08:59.877079 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:59.876907 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" Oct 30 19:08:59.877079 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:59.876929 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" event={"ID":"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0","Type":"ContainerDied","Data":"3d90cdd69073ad22d74141faf68bbc51694ef320f274e2d726548b4d4d29dd16"} Oct 30 19:08:59.877079 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:59.877006 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq" event={"ID":"eb94bb4c-aa2a-4c61-b986-03dd313ab6e0","Type":"ContainerDied","Data":"203ad1dabde5cdacd87c19d825fdd58ef77fffd02038b0d264cc6ab1bab58f2f"} Oct 30 19:08:59.877079 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:59.877035 4442 scope.go:122] "RemoveContainer" containerID="3d90cdd69073ad22d74141faf68bbc51694ef320f274e2d726548b4d4d29dd16" Oct 30 19:08:59.888881 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:59.888861 4442 scope.go:122] "RemoveContainer" containerID="bda4e615fb2dbd2e4c3d093e91f3fcc329e846cc637cecd17211dbb72f73e2de" Oct 30 19:08:59.899082 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:59.899066 4442 scope.go:122] "RemoveContainer" containerID="3d90cdd69073ad22d74141faf68bbc51694ef320f274e2d726548b4d4d29dd16" Oct 30 19:08:59.899369 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:08:59.899352 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d90cdd69073ad22d74141faf68bbc51694ef320f274e2d726548b4d4d29dd16\": container with ID starting with 3d90cdd69073ad22d74141faf68bbc51694ef320f274e2d726548b4d4d29dd16 not found: ID does not exist" containerID="3d90cdd69073ad22d74141faf68bbc51694ef320f274e2d726548b4d4d29dd16" Oct 30 19:08:59.899443 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:59.899378 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d90cdd69073ad22d74141faf68bbc51694ef320f274e2d726548b4d4d29dd16"} err="failed to get container status \"3d90cdd69073ad22d74141faf68bbc51694ef320f274e2d726548b4d4d29dd16\": rpc error: code = NotFound desc = could not find container \"3d90cdd69073ad22d74141faf68bbc51694ef320f274e2d726548b4d4d29dd16\": container with ID starting with 3d90cdd69073ad22d74141faf68bbc51694ef320f274e2d726548b4d4d29dd16 not found: ID does not exist" Oct 30 19:08:59.899443 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:59.899395 4442 scope.go:122] "RemoveContainer" containerID="bda4e615fb2dbd2e4c3d093e91f3fcc329e846cc637cecd17211dbb72f73e2de" Oct 30 19:08:59.899600 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:08:59.899585 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bda4e615fb2dbd2e4c3d093e91f3fcc329e846cc637cecd17211dbb72f73e2de\": container with ID starting with bda4e615fb2dbd2e4c3d093e91f3fcc329e846cc637cecd17211dbb72f73e2de not found: ID does not exist" containerID="bda4e615fb2dbd2e4c3d093e91f3fcc329e846cc637cecd17211dbb72f73e2de" Oct 30 19:08:59.899646 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:59.899604 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bda4e615fb2dbd2e4c3d093e91f3fcc329e846cc637cecd17211dbb72f73e2de"} err="failed to get container status \"bda4e615fb2dbd2e4c3d093e91f3fcc329e846cc637cecd17211dbb72f73e2de\": rpc error: code = NotFound desc = could not find container \"bda4e615fb2dbd2e4c3d093e91f3fcc329e846cc637cecd17211dbb72f73e2de\": container with ID starting with bda4e615fb2dbd2e4c3d093e91f3fcc329e846cc637cecd17211dbb72f73e2de not found: ID does not exist" Oct 30 19:08:59.941723 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:59.941662 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq"] Oct 30 19:08:59.946730 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:08:59.946701 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e95c51-openshift-default-7958f47665-x9mbq"] Oct 30 19:09:01.067723 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:01.067689 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb94bb4c-aa2a-4c61-b986-03dd313ab6e0" path="/var/lib/kubelet/pods/eb94bb4c-aa2a-4c61-b986-03dd313ab6e0/volumes" Oct 30 19:09:04.948225 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:04.948187 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-d82e82-fdf98fb94-dtq2s"] Oct 30 19:09:04.981884 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:04.981844 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d82e82-7d456c8bbc-pmqhf"] Oct 30 19:09:06.616427 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:06.616390 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab244c-7659db958c-jbckm" podUID="a1352e66-acc1-499e-8344-4cab5734f78a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:09:09.949744 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:09.949661 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d82e82-fdf98fb94-dtq2s" podUID="bc6d66de-c4d6-4d9f-bfbf-4cfee09dcbda" containerName="payload-processing" containerID="cri-o://dba1d4bd8ac27bc3040bc6cd612dd127443e6b946c04a7fb6e4cc0c56c602d53" gracePeriod=25 Oct 30 19:09:09.982673 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:09.982618 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d82e82-7d456c8bbc-pmqhf" podUID="f367afff-a23b-47d2-944e-0ab238f13bd0" containerName="payload-pre-processing" containerID="cri-o://fa3f2cf66434fa744997c2f2786b72c0333f1e2638b2e2485e6d2c1b276b7214" gracePeriod=25 Oct 30 19:09:11.222222 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.222187 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8eee7a8-v9kd6"] Oct 30 19:09:11.222913 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.222897 4442 cpu_manager.go:414] "containerMap: removing container" podUID="5ba882f0-916a-4651-8ff9-1e8fe8d9e021" containerName="istio-proxy" Oct 30 19:09:11.222976 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.222914 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ba882f0-916a-4651-8ff9-1e8fe8d9e021" containerName="istio-proxy" Oct 30 19:09:11.222976 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.222926 4442 cpu_manager.go:414] "containerMap: removing container" podUID="83d795bd-5ce3-4648-8823-be54dddc5580" containerName="maas-api" Oct 30 19:09:11.222976 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.222931 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83d795bd-5ce3-4648-8823-be54dddc5580" containerName="maas-api" Oct 30 19:09:11.222976 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.222940 4442 cpu_manager.go:414] "containerMap: removing container" podUID="d147fcbd-d2ce-4a51-a8f1-531db7bcf8fc" containerName="payload-pre-processing" Oct 30 19:09:11.222976 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.222945 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d147fcbd-d2ce-4a51-a8f1-531db7bcf8fc" containerName="payload-pre-processing" Oct 30 19:09:11.222976 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.222957 4442 cpu_manager.go:414] "containerMap: removing container" podUID="3ae44a8a-6af3-482e-9407-33e148efded7" containerName="istio-proxy" Oct 30 19:09:11.222976 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.222961 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ae44a8a-6af3-482e-9407-33e148efded7" containerName="istio-proxy" Oct 30 19:09:11.222976 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.222968 4442 cpu_manager.go:414] "containerMap: removing container" podUID="017e9c94-87b9-4143-bd73-311bb6edbed9" containerName="payload-processing" Oct 30 19:09:11.222976 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.222973 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="017e9c94-87b9-4143-bd73-311bb6edbed9" containerName="payload-processing" Oct 30 19:09:11.223415 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.222984 4442 cpu_manager.go:414] "containerMap: removing container" podUID="c2be17bb-1852-432e-a24d-24d762b27661" containerName="istio-proxy" Oct 30 19:09:11.223415 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.222988 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2be17bb-1852-432e-a24d-24d762b27661" containerName="istio-proxy" Oct 30 19:09:11.223415 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.222997 4442 cpu_manager.go:414] "containerMap: removing container" podUID="eb94bb4c-aa2a-4c61-b986-03dd313ab6e0" containerName="istio-proxy" Oct 30 19:09:11.223415 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.223001 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb94bb4c-aa2a-4c61-b986-03dd313ab6e0" containerName="istio-proxy" Oct 30 19:09:11.223415 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.223007 4442 cpu_manager.go:414] "containerMap: removing container" podUID="01d882ef-fc94-45cb-af2e-3debf4b418cd" containerName="istio-proxy" Oct 30 19:09:11.223415 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.223011 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01d882ef-fc94-45cb-af2e-3debf4b418cd" containerName="istio-proxy" Oct 30 19:09:11.223415 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.223020 4442 cpu_manager.go:414] "containerMap: removing container" podUID="0709501c-2bbd-4926-8679-0e65a78e09ac" containerName="payload-pre-processing" Oct 30 19:09:11.223415 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.223024 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0709501c-2bbd-4926-8679-0e65a78e09ac" containerName="payload-pre-processing" Oct 30 19:09:11.223415 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.223034 4442 cpu_manager.go:414] "containerMap: removing container" podUID="69d3fb60-519d-4101-a848-17c74270fca7" containerName="main" Oct 30 19:09:11.223415 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.223038 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69d3fb60-519d-4101-a848-17c74270fca7" containerName="main" Oct 30 19:09:11.223415 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.223048 4442 cpu_manager.go:414] "containerMap: removing container" podUID="6d8d84cf-1334-4038-8b24-8cbec03e6882" containerName="istio-proxy" Oct 30 19:09:11.223415 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.223052 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d8d84cf-1334-4038-8b24-8cbec03e6882" containerName="istio-proxy" Oct 30 19:09:11.223415 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.223061 4442 cpu_manager.go:414] "containerMap: removing container" podUID="c2be17bb-1852-432e-a24d-24d762b27661" containerName="istio-proxy" Oct 30 19:09:11.223415 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.223065 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2be17bb-1852-432e-a24d-24d762b27661" containerName="istio-proxy" Oct 30 19:09:11.223415 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.223076 4442 cpu_manager.go:414] "containerMap: removing container" podUID="d6601257-a86e-414c-9b71-9dc6042b6818" containerName="payload-pre-processing" Oct 30 19:09:11.223415 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.223081 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6601257-a86e-414c-9b71-9dc6042b6818" containerName="payload-pre-processing" Oct 30 19:09:11.223415 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.223089 4442 cpu_manager.go:414] "containerMap: removing container" podUID="01d882ef-fc94-45cb-af2e-3debf4b418cd" containerName="istio-proxy" Oct 30 19:09:11.223415 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.223093 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01d882ef-fc94-45cb-af2e-3debf4b418cd" containerName="istio-proxy" Oct 30 19:09:11.223415 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.223100 4442 cpu_manager.go:414] "containerMap: removing container" podUID="eb94bb4c-aa2a-4c61-b986-03dd313ab6e0" containerName="istio-proxy" Oct 30 19:09:11.223415 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.223105 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb94bb4c-aa2a-4c61-b986-03dd313ab6e0" containerName="istio-proxy" Oct 30 19:09:11.223415 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.223113 4442 cpu_manager.go:414] "containerMap: removing container" podUID="29ce0e02-4041-4ecb-bb5f-c885099d00b7" containerName="istio-proxy" Oct 30 19:09:11.223415 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.223117 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29ce0e02-4041-4ecb-bb5f-c885099d00b7" containerName="istio-proxy" Oct 30 19:09:11.223415 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.223124 4442 cpu_manager.go:414] "containerMap: removing container" podUID="a2e19b96-f72b-4735-a1e8-a0058dc45470" containerName="payload-pre-processing" Oct 30 19:09:11.223415 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.223145 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2e19b96-f72b-4735-a1e8-a0058dc45470" containerName="payload-pre-processing" Oct 30 19:09:11.223415 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.223158 4442 cpu_manager.go:414] "containerMap: removing container" podUID="29ce0e02-4041-4ecb-bb5f-c885099d00b7" containerName="istio-proxy" Oct 30 19:09:11.223415 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.223166 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29ce0e02-4041-4ecb-bb5f-c885099d00b7" containerName="istio-proxy" Oct 30 19:09:11.223415 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.223179 4442 cpu_manager.go:414] "containerMap: removing container" podUID="3ae44a8a-6af3-482e-9407-33e148efded7" containerName="istio-proxy" Oct 30 19:09:11.223415 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.223185 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ae44a8a-6af3-482e-9407-33e148efded7" containerName="istio-proxy" Oct 30 19:09:11.223415 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.223334 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ba882f0-916a-4651-8ff9-1e8fe8d9e021" containerName="istio-proxy" Oct 30 19:09:11.223415 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.223343 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="83d795bd-5ce3-4648-8823-be54dddc5580" containerName="maas-api" Oct 30 19:09:11.223415 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.223351 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="d147fcbd-d2ce-4a51-a8f1-531db7bcf8fc" containerName="payload-pre-processing" Oct 30 19:09:11.223415 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.223362 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ae44a8a-6af3-482e-9407-33e148efded7" containerName="istio-proxy" Oct 30 19:09:11.223415 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.223372 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="017e9c94-87b9-4143-bd73-311bb6edbed9" containerName="payload-processing" Oct 30 19:09:11.223415 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.223382 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2be17bb-1852-432e-a24d-24d762b27661" containerName="istio-proxy" Oct 30 19:09:11.223415 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.223391 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb94bb4c-aa2a-4c61-b986-03dd313ab6e0" containerName="istio-proxy" Oct 30 19:09:11.223415 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.223400 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="01d882ef-fc94-45cb-af2e-3debf4b418cd" containerName="istio-proxy" Oct 30 19:09:11.223415 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.223408 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="0709501c-2bbd-4926-8679-0e65a78e09ac" containerName="payload-pre-processing" Oct 30 19:09:11.223415 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.223418 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="69d3fb60-519d-4101-a848-17c74270fca7" containerName="main" Oct 30 19:09:11.223415 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.223429 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2be17bb-1852-432e-a24d-24d762b27661" containerName="istio-proxy" Oct 30 19:09:11.223415 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.223441 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6601257-a86e-414c-9b71-9dc6042b6818" containerName="payload-pre-processing" Oct 30 19:09:11.225559 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.223448 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="01d882ef-fc94-45cb-af2e-3debf4b418cd" containerName="istio-proxy" Oct 30 19:09:11.225559 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.223458 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb94bb4c-aa2a-4c61-b986-03dd313ab6e0" containerName="istio-proxy" Oct 30 19:09:11.225559 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.223467 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="29ce0e02-4041-4ecb-bb5f-c885099d00b7" containerName="istio-proxy" Oct 30 19:09:11.225559 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.223476 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2e19b96-f72b-4735-a1e8-a0058dc45470" containerName="payload-pre-processing" Oct 30 19:09:11.225559 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.223486 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="29ce0e02-4041-4ecb-bb5f-c885099d00b7" containerName="istio-proxy" Oct 30 19:09:11.227918 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.227894 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8eee7a8-v9kd6" Oct 30 19:09:11.247436 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.246309 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8eee7a8-v9kd6"] Oct 30 19:09:11.255280 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.254467 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:09:11.255280 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.254566 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:09:11.311633 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.311589 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck8xc\" (UniqueName: \"kubernetes.io/projected/855d59cb-e130-4f86-aba2-2f77b4e9fd2a-kube-api-access-ck8xc\") pod \"maas-api-revoke-sub-d8eee7a8-v9kd6\" (UID: \"855d59cb-e130-4f86-aba2-2f77b4e9fd2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8eee7a8-v9kd6" Oct 30 19:09:11.311818 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.311766 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/855d59cb-e130-4f86-aba2-2f77b4e9fd2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8eee7a8-v9kd6\" (UID: \"855d59cb-e130-4f86-aba2-2f77b4e9fd2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8eee7a8-v9kd6" Oct 30 19:09:11.311818 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.311802 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/855d59cb-e130-4f86-aba2-2f77b4e9fd2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8eee7a8-v9kd6\" (UID: \"855d59cb-e130-4f86-aba2-2f77b4e9fd2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8eee7a8-v9kd6" Oct 30 19:09:11.413915 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.413876 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck8xc\" (UniqueName: \"kubernetes.io/projected/855d59cb-e130-4f86-aba2-2f77b4e9fd2a-kube-api-access-ck8xc\") pod \"maas-api-revoke-sub-d8eee7a8-v9kd6\" (UID: \"855d59cb-e130-4f86-aba2-2f77b4e9fd2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8eee7a8-v9kd6" Oct 30 19:09:11.414099 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.413966 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/855d59cb-e130-4f86-aba2-2f77b4e9fd2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8eee7a8-v9kd6\" (UID: \"855d59cb-e130-4f86-aba2-2f77b4e9fd2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8eee7a8-v9kd6" Oct 30 19:09:11.414099 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.413984 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/855d59cb-e130-4f86-aba2-2f77b4e9fd2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8eee7a8-v9kd6\" (UID: \"855d59cb-e130-4f86-aba2-2f77b4e9fd2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8eee7a8-v9kd6" Oct 30 19:09:11.414645 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.414616 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/855d59cb-e130-4f86-aba2-2f77b4e9fd2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8eee7a8-v9kd6\" (UID: \"855d59cb-e130-4f86-aba2-2f77b4e9fd2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8eee7a8-v9kd6" Oct 30 19:09:11.416723 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.416703 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/855d59cb-e130-4f86-aba2-2f77b4e9fd2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8eee7a8-v9kd6\" (UID: \"855d59cb-e130-4f86-aba2-2f77b4e9fd2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8eee7a8-v9kd6" Oct 30 19:09:11.433090 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.433064 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck8xc\" (UniqueName: \"kubernetes.io/projected/855d59cb-e130-4f86-aba2-2f77b4e9fd2a-kube-api-access-ck8xc\") pod \"maas-api-revoke-sub-d8eee7a8-v9kd6\" (UID: \"855d59cb-e130-4f86-aba2-2f77b4e9fd2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8eee7a8-v9kd6" Oct 30 19:09:11.557863 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.557767 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8eee7a8-v9kd6" Oct 30 19:09:11.861826 ip-10-0-138-176 kubenswrapper[4442]: W1030 19:09:11.861785 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod855d59cb_e130_4f86_aba2_2f77b4e9fd2a.slice/crio-3b07310fca72578b4f1146a0a80fdc679d5cd764057db2ab9e0b1522c0a108fd WatchSource:0}: Error finding container 3b07310fca72578b4f1146a0a80fdc679d5cd764057db2ab9e0b1522c0a108fd: Status 404 returned error can't find the container with id 3b07310fca72578b4f1146a0a80fdc679d5cd764057db2ab9e0b1522c0a108fd Oct 30 19:09:11.862568 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.862475 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8eee7a8-v9kd6"] Oct 30 19:09:11.996614 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:11.996580 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8eee7a8-v9kd6" event={"ID":"855d59cb-e130-4f86-aba2-2f77b4e9fd2a","Type":"ContainerStarted","Data":"3b07310fca72578b4f1146a0a80fdc679d5cd764057db2ab9e0b1522c0a108fd"} Oct 30 19:09:13.014831 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:13.014738 4442 generic.go:356] "Generic (PLEG): container finished" podID="855d59cb-e130-4f86-aba2-2f77b4e9fd2a" containerID="25c81b4c4da19c1e4e250313672fd83a2493e200ea23b6a1863a4e40fcb74970" exitCode=0 Oct 30 19:09:13.015338 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:13.014987 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8eee7a8-v9kd6" event={"ID":"855d59cb-e130-4f86-aba2-2f77b4e9fd2a","Type":"ContainerDied","Data":"25c81b4c4da19c1e4e250313672fd83a2493e200ea23b6a1863a4e40fcb74970"} Oct 30 19:09:14.190995 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:14.190966 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8eee7a8-v9kd6" Oct 30 19:09:14.198993 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:14.198969 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/855d59cb-e130-4f86-aba2-2f77b4e9fd2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/855d59cb-e130-4f86-aba2-2f77b4e9fd2a-maas-api-openshift-ca\") pod \"855d59cb-e130-4f86-aba2-2f77b4e9fd2a\" (UID: \"855d59cb-e130-4f86-aba2-2f77b4e9fd2a\") " Oct 30 19:09:14.199154 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:14.199100 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/855d59cb-e130-4f86-aba2-2f77b4e9fd2a-kube-api-access-ck8xc\" (UniqueName: \"kubernetes.io/projected/855d59cb-e130-4f86-aba2-2f77b4e9fd2a-kube-api-access-ck8xc\") pod \"855d59cb-e130-4f86-aba2-2f77b4e9fd2a\" (UID: \"855d59cb-e130-4f86-aba2-2f77b4e9fd2a\") " Oct 30 19:09:14.199260 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:14.199244 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/855d59cb-e130-4f86-aba2-2f77b4e9fd2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/855d59cb-e130-4f86-aba2-2f77b4e9fd2a-maas-api-xks-ca\") pod \"855d59cb-e130-4f86-aba2-2f77b4e9fd2a\" (UID: \"855d59cb-e130-4f86-aba2-2f77b4e9fd2a\") " Oct 30 19:09:14.199341 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:14.199318 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/855d59cb-e130-4f86-aba2-2f77b4e9fd2a-maas-api-openshift-ca" pod "855d59cb-e130-4f86-aba2-2f77b4e9fd2a" (UID: "855d59cb-e130-4f86-aba2-2f77b4e9fd2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:09:14.199755 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:14.199736 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/855d59cb-e130-4f86-aba2-2f77b4e9fd2a-maas-api-openshift-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:09:14.201412 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:14.201393 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/855d59cb-e130-4f86-aba2-2f77b4e9fd2a-maas-api-xks-ca" pod "855d59cb-e130-4f86-aba2-2f77b4e9fd2a" (UID: "855d59cb-e130-4f86-aba2-2f77b4e9fd2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:09:14.201488 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:14.201393 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/855d59cb-e130-4f86-aba2-2f77b4e9fd2a-kube-api-access-ck8xc" pod "855d59cb-e130-4f86-aba2-2f77b4e9fd2a" (UID: "855d59cb-e130-4f86-aba2-2f77b4e9fd2a"). InnerVolumeSpecName "kube-api-access-ck8xc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:09:14.301277 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:14.301198 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck8xc\" (UniqueName: \"kubernetes.io/projected/855d59cb-e130-4f86-aba2-2f77b4e9fd2a-kube-api-access-ck8xc\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:09:14.301277 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:14.301228 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/855d59cb-e130-4f86-aba2-2f77b4e9fd2a-maas-api-xks-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:09:15.041171 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:15.041109 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8eee7a8-v9kd6" event={"ID":"855d59cb-e130-4f86-aba2-2f77b4e9fd2a","Type":"ContainerDied","Data":"3b07310fca72578b4f1146a0a80fdc679d5cd764057db2ab9e0b1522c0a108fd"} Oct 30 19:09:15.041411 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:15.041177 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8eee7a8-v9kd6" Oct 30 19:09:15.041411 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:15.041196 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b07310fca72578b4f1146a0a80fdc679d5cd764057db2ab9e0b1522c0a108fd" Oct 30 19:09:16.617244 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:16.617200 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab244c-7659db958c-jbckm" podUID="a1352e66-acc1-499e-8344-4cab5734f78a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:09:16.709081 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:16.709042 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d82e82-7d456c8bbc-pmqhf" podUID="f367afff-a23b-47d2-944e-0ab238f13bd0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:09:16.940553 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:09:16.940519 4442 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1352e66_acc1_499e_8344_4cab5734f78a.slice/crio-conmon-c9e6bda211e4c3e7ae4b6d0e8d2cf75080afa9d1c0858c5baa595dfaae447bf9.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:09:17.080169 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:17.080095 4442 generic.go:356] "Generic (PLEG): container finished" podID="a1352e66-acc1-499e-8344-4cab5734f78a" containerID="c9e6bda211e4c3e7ae4b6d0e8d2cf75080afa9d1c0858c5baa595dfaae447bf9" exitCode=0 Oct 30 19:09:17.088887 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:17.088849 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab244c-7659db958c-jbckm" event={"ID":"a1352e66-acc1-499e-8344-4cab5734f78a","Type":"ContainerDied","Data":"c9e6bda211e4c3e7ae4b6d0e8d2cf75080afa9d1c0858c5baa595dfaae447bf9"} Oct 30 19:09:17.162083 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:17.162058 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab244c-7659db958c-jbckm" Oct 30 19:09:17.245619 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:17.245579 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1352e66-acc1-499e-8344-4cab5734f78a-kube-api-access-tr7m4\" (UniqueName: \"kubernetes.io/projected/a1352e66-acc1-499e-8344-4cab5734f78a-kube-api-access-tr7m4\") pod \"a1352e66-acc1-499e-8344-4cab5734f78a\" (UID: \"a1352e66-acc1-499e-8344-4cab5734f78a\") " Oct 30 19:09:17.246091 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:17.245693 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1352e66-acc1-499e-8344-4cab5734f78a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1352e66-acc1-499e-8344-4cab5734f78a-plugins-config-volume\") pod \"a1352e66-acc1-499e-8344-4cab5734f78a\" (UID: \"a1352e66-acc1-499e-8344-4cab5734f78a\") " Oct 30 19:09:17.246831 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:17.246789 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1352e66-acc1-499e-8344-4cab5734f78a-plugins-config-volume" pod "a1352e66-acc1-499e-8344-4cab5734f78a" (UID: "a1352e66-acc1-499e-8344-4cab5734f78a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:09:17.261269 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:17.261227 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1352e66-acc1-499e-8344-4cab5734f78a-kube-api-access-tr7m4" pod "a1352e66-acc1-499e-8344-4cab5734f78a" (UID: "a1352e66-acc1-499e-8344-4cab5734f78a"). InnerVolumeSpecName "kube-api-access-tr7m4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:09:17.348339 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:17.348305 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr7m4\" (UniqueName: \"kubernetes.io/projected/a1352e66-acc1-499e-8344-4cab5734f78a-kube-api-access-tr7m4\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:09:17.348339 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:17.348336 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1352e66-acc1-499e-8344-4cab5734f78a-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:09:17.616562 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:09:17.616527 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 19:09:17.751695 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:17.751653 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d82e82-fdf98fb94-dtq2s" podUID="bc6d66de-c4d6-4d9f-bfbf-4cfee09dcbda" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:09:18.090099 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:18.090014 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab244c-7659db958c-jbckm" Oct 30 19:09:18.090099 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:18.090031 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab244c-7659db958c-jbckm" event={"ID":"a1352e66-acc1-499e-8344-4cab5734f78a","Type":"ContainerDied","Data":"f2318c14e16bb8f6ca10815d326258d939ba5100976968933a4ff21c2444abd3"} Oct 30 19:09:18.090099 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:18.090099 4442 scope.go:122] "RemoveContainer" containerID="c9e6bda211e4c3e7ae4b6d0e8d2cf75080afa9d1c0858c5baa595dfaae447bf9" Oct 30 19:09:18.146779 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:18.146746 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab244c-7659db958c-jbckm"] Oct 30 19:09:18.155708 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:18.155673 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ab244c-7659db958c-jbckm"] Oct 30 19:09:19.066699 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:19.066668 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1352e66-acc1-499e-8344-4cab5734f78a" path="/var/lib/kubelet/pods/a1352e66-acc1-499e-8344-4cab5734f78a/volumes" Oct 30 19:09:20.733505 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:20.733473 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg"] Oct 30 19:09:20.733959 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:20.733735 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" podUID="0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf" containerName="istio-proxy" containerID="cri-o://bba6d2ac3f713e864fd4ea7804176077f7cb63bce3cd5377ce40545a1a54f3dd" gracePeriod=30 Oct 30 19:09:21.643708 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:21.643668 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39a799a2-qmhd2"] Oct 30 19:09:21.644425 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:21.644408 4442 cpu_manager.go:414] "containerMap: removing container" podUID="a1352e66-acc1-499e-8344-4cab5734f78a" containerName="payload-pre-processing" Oct 30 19:09:21.644425 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:21.644425 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1352e66-acc1-499e-8344-4cab5734f78a" containerName="payload-pre-processing" Oct 30 19:09:21.644536 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:21.644455 4442 cpu_manager.go:414] "containerMap: removing container" podUID="5ba882f0-916a-4651-8ff9-1e8fe8d9e021" containerName="istio-proxy" Oct 30 19:09:21.644536 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:21.644460 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ba882f0-916a-4651-8ff9-1e8fe8d9e021" containerName="istio-proxy" Oct 30 19:09:21.644536 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:21.644470 4442 cpu_manager.go:414] "containerMap: removing container" podUID="855d59cb-e130-4f86-aba2-2f77b4e9fd2a" containerName="revoke-keys" Oct 30 19:09:21.644536 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:21.644474 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="855d59cb-e130-4f86-aba2-2f77b4e9fd2a" containerName="revoke-keys" Oct 30 19:09:21.644664 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:21.644563 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1352e66-acc1-499e-8344-4cab5734f78a" containerName="payload-pre-processing" Oct 30 19:09:21.644664 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:21.644575 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ba882f0-916a-4651-8ff9-1e8fe8d9e021" containerName="istio-proxy" Oct 30 19:09:21.644664 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:21.644581 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="855d59cb-e130-4f86-aba2-2f77b4e9fd2a" containerName="revoke-keys" Oct 30 19:09:21.644664 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:21.644589 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ae44a8a-6af3-482e-9407-33e148efded7" containerName="istio-proxy" Oct 30 19:09:21.649617 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:21.649595 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a799a2-qmhd2" Oct 30 19:09:21.665681 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:21.665650 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39a799a2-qmhd2"] Oct 30 19:09:21.672060 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:21.672033 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:09:21.672229 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:21.672073 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:09:21.719443 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:21.719404 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef052ec7-f935-4c53-bf21-6d1e2e9c90fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39a799a2-qmhd2\" (UID: \"ef052ec7-f935-4c53-bf21-6d1e2e9c90fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a799a2-qmhd2" Oct 30 19:09:21.719443 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:21.719442 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2d74\" (UniqueName: \"kubernetes.io/projected/ef052ec7-f935-4c53-bf21-6d1e2e9c90fc-kube-api-access-j2d74\") pod \"maas-api-revoke-sub-39a799a2-qmhd2\" (UID: \"ef052ec7-f935-4c53-bf21-6d1e2e9c90fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a799a2-qmhd2" Oct 30 19:09:21.719679 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:21.719604 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef052ec7-f935-4c53-bf21-6d1e2e9c90fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39a799a2-qmhd2\" (UID: \"ef052ec7-f935-4c53-bf21-6d1e2e9c90fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a799a2-qmhd2" Oct 30 19:09:21.823780 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:21.821289 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef052ec7-f935-4c53-bf21-6d1e2e9c90fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39a799a2-qmhd2\" (UID: \"ef052ec7-f935-4c53-bf21-6d1e2e9c90fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a799a2-qmhd2" Oct 30 19:09:21.824431 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:21.824400 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2d74\" (UniqueName: \"kubernetes.io/projected/ef052ec7-f935-4c53-bf21-6d1e2e9c90fc-kube-api-access-j2d74\") pod \"maas-api-revoke-sub-39a799a2-qmhd2\" (UID: \"ef052ec7-f935-4c53-bf21-6d1e2e9c90fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a799a2-qmhd2" Oct 30 19:09:21.824573 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:21.824512 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef052ec7-f935-4c53-bf21-6d1e2e9c90fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39a799a2-qmhd2\" (UID: \"ef052ec7-f935-4c53-bf21-6d1e2e9c90fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a799a2-qmhd2" Oct 30 19:09:21.824573 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:21.824523 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef052ec7-f935-4c53-bf21-6d1e2e9c90fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39a799a2-qmhd2\" (UID: \"ef052ec7-f935-4c53-bf21-6d1e2e9c90fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a799a2-qmhd2" Oct 30 19:09:21.827266 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:21.827244 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef052ec7-f935-4c53-bf21-6d1e2e9c90fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39a799a2-qmhd2\" (UID: \"ef052ec7-f935-4c53-bf21-6d1e2e9c90fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a799a2-qmhd2" Oct 30 19:09:21.846573 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:21.846547 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2d74\" (UniqueName: \"kubernetes.io/projected/ef052ec7-f935-4c53-bf21-6d1e2e9c90fc-kube-api-access-j2d74\") pod \"maas-api-revoke-sub-39a799a2-qmhd2\" (UID: \"ef052ec7-f935-4c53-bf21-6d1e2e9c90fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a799a2-qmhd2" Oct 30 19:09:21.966930 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:21.966826 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a799a2-qmhd2" Oct 30 19:09:22.382113 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:22.381063 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39a799a2-qmhd2"] Oct 30 19:09:23.156702 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:23.156662 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a799a2-qmhd2" event={"ID":"ef052ec7-f935-4c53-bf21-6d1e2e9c90fc","Type":"ContainerStarted","Data":"733abb0344a8b76e9a54e07a79abe990dd59ad56da02401951b833e172bbe4d4"} Oct 30 19:09:23.157250 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:23.156731 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a799a2-qmhd2" event={"ID":"ef052ec7-f935-4c53-bf21-6d1e2e9c90fc","Type":"ContainerStarted","Data":"ac9711d894adc9548ce2b9eef9d70bc128fcb311285d1a102ce30c33b8d64e95"} Oct 30 19:09:23.219634 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:23.219512 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a799a2-qmhd2" podStartSLOduration=2.219498315 podStartE2EDuration="2.219498315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:23.216340016 +0000 UTC m=+4260.306402354" watchObservedRunningTime="2026-10-30 19:09:23.219498315 +0000 UTC m=+4260.309560685" Oct 30 19:09:24.167865 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:24.167832 4442 generic.go:356] "Generic (PLEG): container finished" podID="ef052ec7-f935-4c53-bf21-6d1e2e9c90fc" containerID="733abb0344a8b76e9a54e07a79abe990dd59ad56da02401951b833e172bbe4d4" exitCode=0 Oct 30 19:09:24.168401 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:24.167923 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a799a2-qmhd2" event={"ID":"ef052ec7-f935-4c53-bf21-6d1e2e9c90fc","Type":"ContainerDied","Data":"733abb0344a8b76e9a54e07a79abe990dd59ad56da02401951b833e172bbe4d4"} Oct 30 19:09:25.355961 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:25.355932 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a799a2-qmhd2" Oct 30 19:09:25.524118 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:25.524031 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef052ec7-f935-4c53-bf21-6d1e2e9c90fc-kube-api-access-j2d74\" (UniqueName: \"kubernetes.io/projected/ef052ec7-f935-4c53-bf21-6d1e2e9c90fc-kube-api-access-j2d74\") pod \"ef052ec7-f935-4c53-bf21-6d1e2e9c90fc\" (UID: \"ef052ec7-f935-4c53-bf21-6d1e2e9c90fc\") " Oct 30 19:09:25.524311 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:25.524235 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef052ec7-f935-4c53-bf21-6d1e2e9c90fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef052ec7-f935-4c53-bf21-6d1e2e9c90fc-maas-api-openshift-ca\") pod \"ef052ec7-f935-4c53-bf21-6d1e2e9c90fc\" (UID: \"ef052ec7-f935-4c53-bf21-6d1e2e9c90fc\") " Oct 30 19:09:25.524311 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:25.524286 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef052ec7-f935-4c53-bf21-6d1e2e9c90fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef052ec7-f935-4c53-bf21-6d1e2e9c90fc-maas-api-xks-ca\") pod \"ef052ec7-f935-4c53-bf21-6d1e2e9c90fc\" (UID: \"ef052ec7-f935-4c53-bf21-6d1e2e9c90fc\") " Oct 30 19:09:25.524614 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:25.524578 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef052ec7-f935-4c53-bf21-6d1e2e9c90fc-maas-api-openshift-ca" pod "ef052ec7-f935-4c53-bf21-6d1e2e9c90fc" (UID: "ef052ec7-f935-4c53-bf21-6d1e2e9c90fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:09:25.524861 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:25.524843 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef052ec7-f935-4c53-bf21-6d1e2e9c90fc-maas-api-openshift-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:09:25.526499 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:25.526477 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef052ec7-f935-4c53-bf21-6d1e2e9c90fc-maas-api-xks-ca" pod "ef052ec7-f935-4c53-bf21-6d1e2e9c90fc" (UID: "ef052ec7-f935-4c53-bf21-6d1e2e9c90fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:09:25.526585 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:25.526541 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef052ec7-f935-4c53-bf21-6d1e2e9c90fc-kube-api-access-j2d74" pod "ef052ec7-f935-4c53-bf21-6d1e2e9c90fc" (UID: "ef052ec7-f935-4c53-bf21-6d1e2e9c90fc"). InnerVolumeSpecName "kube-api-access-j2d74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:09:25.625942 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:25.625908 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef052ec7-f935-4c53-bf21-6d1e2e9c90fc-maas-api-xks-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:09:25.625942 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:25.625948 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2d74\" (UniqueName: \"kubernetes.io/projected/ef052ec7-f935-4c53-bf21-6d1e2e9c90fc-kube-api-access-j2d74\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:09:26.024508 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:26.024481 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" Oct 30 19:09:26.133583 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:26.133552 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-istio-envoy\") pod \"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf\" (UID: \"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf\") " Oct 30 19:09:26.133799 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:26.133611 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-istio-token\" (UniqueName: \"kubernetes.io/projected/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-istio-token\") pod \"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf\" (UID: \"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf\") " Oct 30 19:09:26.133799 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:26.133665 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-istio-podinfo\") pod \"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf\" (UID: \"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf\") " Oct 30 19:09:26.133799 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:26.133712 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-workload-certs\") pod \"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf\" (UID: \"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf\") " Oct 30 19:09:26.133799 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:26.133790 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-workload-socket\") pod \"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf\" (UID: \"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf\") " Oct 30 19:09:26.134043 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:26.133877 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-credential-socket\") pod \"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf\" (UID: \"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf\") " Oct 30 19:09:26.134043 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:26.133994 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-istio-data\") pod \"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf\" (UID: \"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf\") " Oct 30 19:09:26.134210 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:26.134077 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-istiod-ca-cert\") pod \"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf\" (UID: \"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf\") " Oct 30 19:09:26.134275 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:26.134247 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-kube-api-access-5ps2q\" (UniqueName: \"kubernetes.io/projected/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-kube-api-access-5ps2q\") pod \"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf\" (UID: \"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf\") " Oct 30 19:09:26.134349 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:26.134293 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-workload-certs" pod "0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf" (UID: "0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:09:26.134416 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:26.134369 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-credential-socket" pod "0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf" (UID: "0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:09:26.134574 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:26.134556 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-workload-socket" pod "0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf" (UID: "0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:09:26.135007 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:26.134981 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:09:26.135117 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:26.135012 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:09:26.135117 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:26.135028 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:09:26.135469 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:26.135446 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-istio-data" pod "0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf" (UID: "0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:09:26.135792 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:26.135770 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-istiod-ca-cert" pod "0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf" (UID: "0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:09:26.137094 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:26.137067 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-istio-envoy" pod "0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf" (UID: "0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:09:26.137238 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:26.137163 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-istio-podinfo" pod "0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf" (UID: "0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:09:26.137395 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:26.137374 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-kube-api-access-5ps2q" pod "0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf" (UID: "0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf"). InnerVolumeSpecName "kube-api-access-5ps2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:09:26.137821 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:26.137799 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-istio-token" pod "0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf" (UID: "0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:09:26.189732 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:26.189699 4442 generic.go:356] "Generic (PLEG): container finished" podID="0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf" containerID="bba6d2ac3f713e864fd4ea7804176077f7cb63bce3cd5377ce40545a1a54f3dd" exitCode=0 Oct 30 19:09:26.189912 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:26.189785 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" Oct 30 19:09:26.189912 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:26.189781 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" event={"ID":"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf","Type":"ContainerDied","Data":"bba6d2ac3f713e864fd4ea7804176077f7cb63bce3cd5377ce40545a1a54f3dd"} Oct 30 19:09:26.189912 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:26.189842 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg" event={"ID":"0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf","Type":"ContainerDied","Data":"6fe3a0ce8c3d6d4e7364470350a85b9ab7bac3e18b3f2553ec41c3e9dcaaceac"} Oct 30 19:09:26.189912 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:26.189872 4442 scope.go:122] "RemoveContainer" containerID="bba6d2ac3f713e864fd4ea7804176077f7cb63bce3cd5377ce40545a1a54f3dd" Oct 30 19:09:26.191727 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:26.191682 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a799a2-qmhd2" Oct 30 19:09:26.191841 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:26.191701 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a799a2-qmhd2" event={"ID":"ef052ec7-f935-4c53-bf21-6d1e2e9c90fc","Type":"ContainerDied","Data":"ac9711d894adc9548ce2b9eef9d70bc128fcb311285d1a102ce30c33b8d64e95"} Oct 30 19:09:26.191841 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:26.191835 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac9711d894adc9548ce2b9eef9d70bc128fcb311285d1a102ce30c33b8d64e95" Oct 30 19:09:26.203705 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:26.203677 4442 scope.go:122] "RemoveContainer" containerID="0c2ef475dfa96e7018d41cac96a0a2285ed2f01cbf2d8bbc0a64d2aa63155315" Oct 30 19:09:26.214332 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:26.214313 4442 scope.go:122] "RemoveContainer" containerID="bba6d2ac3f713e864fd4ea7804176077f7cb63bce3cd5377ce40545a1a54f3dd" Oct 30 19:09:26.214615 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:09:26.214584 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bba6d2ac3f713e864fd4ea7804176077f7cb63bce3cd5377ce40545a1a54f3dd\": container with ID starting with bba6d2ac3f713e864fd4ea7804176077f7cb63bce3cd5377ce40545a1a54f3dd not found: ID does not exist" containerID="bba6d2ac3f713e864fd4ea7804176077f7cb63bce3cd5377ce40545a1a54f3dd" Oct 30 19:09:26.214698 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:26.214613 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bba6d2ac3f713e864fd4ea7804176077f7cb63bce3cd5377ce40545a1a54f3dd"} err="failed to get container status \"bba6d2ac3f713e864fd4ea7804176077f7cb63bce3cd5377ce40545a1a54f3dd\": rpc error: code = NotFound desc = could not find container \"bba6d2ac3f713e864fd4ea7804176077f7cb63bce3cd5377ce40545a1a54f3dd\": container with ID starting with bba6d2ac3f713e864fd4ea7804176077f7cb63bce3cd5377ce40545a1a54f3dd not found: ID does not exist" Oct 30 19:09:26.214698 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:26.214634 4442 scope.go:122] "RemoveContainer" containerID="0c2ef475dfa96e7018d41cac96a0a2285ed2f01cbf2d8bbc0a64d2aa63155315" Oct 30 19:09:26.214911 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:09:26.214895 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c2ef475dfa96e7018d41cac96a0a2285ed2f01cbf2d8bbc0a64d2aa63155315\": container with ID starting with 0c2ef475dfa96e7018d41cac96a0a2285ed2f01cbf2d8bbc0a64d2aa63155315 not found: ID does not exist" containerID="0c2ef475dfa96e7018d41cac96a0a2285ed2f01cbf2d8bbc0a64d2aa63155315" Oct 30 19:09:26.214963 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:26.214919 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c2ef475dfa96e7018d41cac96a0a2285ed2f01cbf2d8bbc0a64d2aa63155315"} err="failed to get container status \"0c2ef475dfa96e7018d41cac96a0a2285ed2f01cbf2d8bbc0a64d2aa63155315\": rpc error: code = NotFound desc = could not find container \"0c2ef475dfa96e7018d41cac96a0a2285ed2f01cbf2d8bbc0a64d2aa63155315\": container with ID starting with 0c2ef475dfa96e7018d41cac96a0a2285ed2f01cbf2d8bbc0a64d2aa63155315 not found: ID does not exist" Oct 30 19:09:26.236701 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:26.236666 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ps2q\" (UniqueName: \"kubernetes.io/projected/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-kube-api-access-5ps2q\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:09:26.236701 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:26.236692 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:09:26.236701 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:26.236702 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:09:26.236963 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:26.236710 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:09:26.236963 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:26.236718 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:09:26.236963 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:26.236725 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:09:26.257940 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:26.257904 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg"] Oct 30 19:09:26.262047 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:26.262017 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-15e8827b-openshift-default-8cc99c85c-r7bgg"] Oct 30 19:09:26.709404 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:26.709359 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d82e82-7d456c8bbc-pmqhf" podUID="f367afff-a23b-47d2-944e-0ab238f13bd0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:09:27.067072 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:27.066998 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf" path="/var/lib/kubelet/pods/0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf/volumes" Oct 30 19:09:27.751415 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:27.751379 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d82e82-fdf98fb94-dtq2s" podUID="bc6d66de-c4d6-4d9f-bfbf-4cfee09dcbda" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:09:30.233807 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:30.233716 4442 generic.go:356] "Generic (PLEG): container finished" podID="bc6d66de-c4d6-4d9f-bfbf-4cfee09dcbda" containerID="dba1d4bd8ac27bc3040bc6cd612dd127443e6b946c04a7fb6e4cc0c56c602d53" exitCode=0 Oct 30 19:09:30.234692 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:30.234652 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d82e82-fdf98fb94-dtq2s" event={"ID":"bc6d66de-c4d6-4d9f-bfbf-4cfee09dcbda","Type":"ContainerDied","Data":"dba1d4bd8ac27bc3040bc6cd612dd127443e6b946c04a7fb6e4cc0c56c602d53"} Oct 30 19:09:30.237426 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:30.237390 4442 generic.go:356] "Generic (PLEG): container finished" podID="f367afff-a23b-47d2-944e-0ab238f13bd0" containerID="fa3f2cf66434fa744997c2f2786b72c0333f1e2638b2e2485e6d2c1b276b7214" exitCode=0 Oct 30 19:09:30.237651 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:30.237624 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d82e82-7d456c8bbc-pmqhf" event={"ID":"f367afff-a23b-47d2-944e-0ab238f13bd0","Type":"ContainerDied","Data":"fa3f2cf66434fa744997c2f2786b72c0333f1e2638b2e2485e6d2c1b276b7214"} Oct 30 19:09:30.355269 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:30.351220 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d82e82-fdf98fb94-dtq2s" Oct 30 19:09:30.371051 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:30.371022 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d82e82-7d456c8bbc-pmqhf" Oct 30 19:09:30.397902 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:30.397873 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f367afff-a23b-47d2-944e-0ab238f13bd0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f367afff-a23b-47d2-944e-0ab238f13bd0-plugins-config-volume\") pod \"f367afff-a23b-47d2-944e-0ab238f13bd0\" (UID: \"f367afff-a23b-47d2-944e-0ab238f13bd0\") " Oct 30 19:09:30.398083 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:30.397927 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc6d66de-c4d6-4d9f-bfbf-4cfee09dcbda-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc6d66de-c4d6-4d9f-bfbf-4cfee09dcbda-plugins-config-volume\") pod \"bc6d66de-c4d6-4d9f-bfbf-4cfee09dcbda\" (UID: \"bc6d66de-c4d6-4d9f-bfbf-4cfee09dcbda\") " Oct 30 19:09:30.398083 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:30.398004 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc6d66de-c4d6-4d9f-bfbf-4cfee09dcbda-kube-api-access-p8cl4\" (UniqueName: \"kubernetes.io/projected/bc6d66de-c4d6-4d9f-bfbf-4cfee09dcbda-kube-api-access-p8cl4\") pod \"bc6d66de-c4d6-4d9f-bfbf-4cfee09dcbda\" (UID: \"bc6d66de-c4d6-4d9f-bfbf-4cfee09dcbda\") " Oct 30 19:09:30.398387 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:30.398367 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f367afff-a23b-47d2-944e-0ab238f13bd0-plugins-config-volume" pod "f367afff-a23b-47d2-944e-0ab238f13bd0" (UID: "f367afff-a23b-47d2-944e-0ab238f13bd0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:09:30.398458 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:30.398436 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc6d66de-c4d6-4d9f-bfbf-4cfee09dcbda-plugins-config-volume" pod "bc6d66de-c4d6-4d9f-bfbf-4cfee09dcbda" (UID: "bc6d66de-c4d6-4d9f-bfbf-4cfee09dcbda"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:09:30.398594 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:30.398577 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f367afff-a23b-47d2-944e-0ab238f13bd0-kube-api-access-mpm6f\" (UniqueName: \"kubernetes.io/projected/f367afff-a23b-47d2-944e-0ab238f13bd0-kube-api-access-mpm6f\") pod \"f367afff-a23b-47d2-944e-0ab238f13bd0\" (UID: \"f367afff-a23b-47d2-944e-0ab238f13bd0\") " Oct 30 19:09:30.399242 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:30.399219 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f367afff-a23b-47d2-944e-0ab238f13bd0-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:09:30.399242 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:30.399241 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc6d66de-c4d6-4d9f-bfbf-4cfee09dcbda-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:09:30.400643 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:30.400621 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc6d66de-c4d6-4d9f-bfbf-4cfee09dcbda-kube-api-access-p8cl4" pod "bc6d66de-c4d6-4d9f-bfbf-4cfee09dcbda" (UID: "bc6d66de-c4d6-4d9f-bfbf-4cfee09dcbda"). InnerVolumeSpecName "kube-api-access-p8cl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:09:30.401230 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:30.401206 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f367afff-a23b-47d2-944e-0ab238f13bd0-kube-api-access-mpm6f" pod "f367afff-a23b-47d2-944e-0ab238f13bd0" (UID: "f367afff-a23b-47d2-944e-0ab238f13bd0"). InnerVolumeSpecName "kube-api-access-mpm6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:09:30.501093 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:30.501010 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8cl4\" (UniqueName: \"kubernetes.io/projected/bc6d66de-c4d6-4d9f-bfbf-4cfee09dcbda-kube-api-access-p8cl4\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:09:30.501093 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:30.501043 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpm6f\" (UniqueName: \"kubernetes.io/projected/f367afff-a23b-47d2-944e-0ab238f13bd0-kube-api-access-mpm6f\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:09:31.081519 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:31.081495 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm"] Oct 30 19:09:31.081750 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:31.081724 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" podUID="cd55e4f9-4d9d-4fba-a53e-fd3403134906" containerName="istio-proxy" containerID="cri-o://59c69f00f4e8e3d29ab16a527166616434def2b07649b87dae8f67261cef32ce" gracePeriod=30 Oct 30 19:09:31.293665 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:31.293556 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d82e82-fdf98fb94-dtq2s" event={"ID":"bc6d66de-c4d6-4d9f-bfbf-4cfee09dcbda","Type":"ContainerDied","Data":"326bcf1cf0282a725c99d4d76b8da4d32fd0e72f471bd618e990848ece078421"} Oct 30 19:09:31.294237 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:31.294217 4442 scope.go:122] "RemoveContainer" containerID="dba1d4bd8ac27bc3040bc6cd612dd127443e6b946c04a7fb6e4cc0c56c602d53" Oct 30 19:09:31.294609 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:31.294592 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d82e82-fdf98fb94-dtq2s" Oct 30 19:09:31.321536 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:31.321502 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d82e82-7d456c8bbc-pmqhf" event={"ID":"f367afff-a23b-47d2-944e-0ab238f13bd0","Type":"ContainerDied","Data":"afa603d097de270129f2e80a5994651c9351d04293c63875e302bb92c6d94db8"} Oct 30 19:09:31.321892 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:31.321875 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d82e82-7d456c8bbc-pmqhf" Oct 30 19:09:31.361386 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:31.361259 4442 scope.go:122] "RemoveContainer" containerID="fa3f2cf66434fa744997c2f2786b72c0333f1e2638b2e2485e6d2c1b276b7214" Oct 30 19:09:31.385897 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:31.385867 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-d82e82-fdf98fb94-dtq2s"] Oct 30 19:09:31.389818 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:31.389790 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-d82e82-fdf98fb94-dtq2s"] Oct 30 19:09:31.418394 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:31.418368 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d82e82-7d456c8bbc-pmqhf"] Oct 30 19:09:31.422175 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:31.422127 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-d82e82-7d456c8bbc-pmqhf"] Oct 30 19:09:31.612730 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:31.612649 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4a7508-s9c2j"] Oct 30 19:09:31.613324 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:31.613310 4442 cpu_manager.go:414] "containerMap: removing container" podUID="bc6d66de-c4d6-4d9f-bfbf-4cfee09dcbda" containerName="payload-processing" Oct 30 19:09:31.613324 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:31.613325 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc6d66de-c4d6-4d9f-bfbf-4cfee09dcbda" containerName="payload-processing" Oct 30 19:09:31.613437 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:31.613339 4442 cpu_manager.go:414] "containerMap: removing container" podUID="f367afff-a23b-47d2-944e-0ab238f13bd0" containerName="payload-pre-processing" Oct 30 19:09:31.613437 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:31.613345 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f367afff-a23b-47d2-944e-0ab238f13bd0" containerName="payload-pre-processing" Oct 30 19:09:31.613437 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:31.613360 4442 cpu_manager.go:414] "containerMap: removing container" podUID="0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf" containerName="istio-proxy" Oct 30 19:09:31.613437 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:31.613365 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf" containerName="istio-proxy" Oct 30 19:09:31.613437 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:31.613412 4442 cpu_manager.go:414] "containerMap: removing container" podUID="0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf" containerName="istio-proxy" Oct 30 19:09:31.613437 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:31.613418 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf" containerName="istio-proxy" Oct 30 19:09:31.613437 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:31.613432 4442 cpu_manager.go:414] "containerMap: removing container" podUID="ef052ec7-f935-4c53-bf21-6d1e2e9c90fc" containerName="revoke-keys" Oct 30 19:09:31.613437 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:31.613437 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef052ec7-f935-4c53-bf21-6d1e2e9c90fc" containerName="revoke-keys" Oct 30 19:09:31.613688 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:31.613519 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf" containerName="istio-proxy" Oct 30 19:09:31.613688 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:31.613528 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef052ec7-f935-4c53-bf21-6d1e2e9c90fc" containerName="revoke-keys" Oct 30 19:09:31.613688 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:31.613534 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc6d66de-c4d6-4d9f-bfbf-4cfee09dcbda" containerName="payload-processing" Oct 30 19:09:31.613688 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:31.613541 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="f367afff-a23b-47d2-944e-0ab238f13bd0" containerName="payload-pre-processing" Oct 30 19:09:31.613688 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:31.613547 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="0edbed5d-d9dd-4ccd-8ee5-7af32e7c74cf" containerName="istio-proxy" Oct 30 19:09:31.617259 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:31.617237 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4a7508-s9c2j" Oct 30 19:09:31.635992 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:31.635965 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4a7508-s9c2j"] Oct 30 19:09:31.653190 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:31.649387 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:09:31.653190 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:31.649519 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:09:31.675436 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:31.675399 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2df8e39c-34a7-4294-a546-bfdb150db74f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4a7508-s9c2j\" (UID: \"2df8e39c-34a7-4294-a546-bfdb150db74f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4a7508-s9c2j" Oct 30 19:09:31.675631 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:31.675479 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsd8q\" (UniqueName: \"kubernetes.io/projected/2df8e39c-34a7-4294-a546-bfdb150db74f-kube-api-access-fsd8q\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4a7508-s9c2j\" (UID: \"2df8e39c-34a7-4294-a546-bfdb150db74f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4a7508-s9c2j" Oct 30 19:09:31.675631 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:31.675590 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2df8e39c-34a7-4294-a546-bfdb150db74f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4a7508-s9c2j\" (UID: \"2df8e39c-34a7-4294-a546-bfdb150db74f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4a7508-s9c2j" Oct 30 19:09:31.776804 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:31.776773 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2df8e39c-34a7-4294-a546-bfdb150db74f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4a7508-s9c2j\" (UID: \"2df8e39c-34a7-4294-a546-bfdb150db74f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4a7508-s9c2j" Oct 30 19:09:31.777029 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:31.776839 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsd8q\" (UniqueName: \"kubernetes.io/projected/2df8e39c-34a7-4294-a546-bfdb150db74f-kube-api-access-fsd8q\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4a7508-s9c2j\" (UID: \"2df8e39c-34a7-4294-a546-bfdb150db74f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4a7508-s9c2j" Oct 30 19:09:31.777029 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:31.776957 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2df8e39c-34a7-4294-a546-bfdb150db74f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4a7508-s9c2j\" (UID: \"2df8e39c-34a7-4294-a546-bfdb150db74f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4a7508-s9c2j" Oct 30 19:09:31.777702 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:31.777625 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2df8e39c-34a7-4294-a546-bfdb150db74f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4a7508-s9c2j\" (UID: \"2df8e39c-34a7-4294-a546-bfdb150db74f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4a7508-s9c2j" Oct 30 19:09:31.779534 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:31.779511 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2df8e39c-34a7-4294-a546-bfdb150db74f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4a7508-s9c2j\" (UID: \"2df8e39c-34a7-4294-a546-bfdb150db74f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4a7508-s9c2j" Oct 30 19:09:31.796048 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:31.796027 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsd8q\" (UniqueName: \"kubernetes.io/projected/2df8e39c-34a7-4294-a546-bfdb150db74f-kube-api-access-fsd8q\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4a7508-s9c2j\" (UID: \"2df8e39c-34a7-4294-a546-bfdb150db74f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4a7508-s9c2j" Oct 30 19:09:31.933235 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:31.933190 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4a7508-s9c2j" Oct 30 19:09:32.241025 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:32.240986 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4a7508-s9c2j"] Oct 30 19:09:32.336645 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:32.336605 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4a7508-s9c2j" event={"ID":"2df8e39c-34a7-4294-a546-bfdb150db74f","Type":"ContainerStarted","Data":"4be26db4510ff98f925803c94dcc1f559b4642ba069902ebe4162d2fe4f87df5"} Oct 30 19:09:33.067632 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:33.067551 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc6d66de-c4d6-4d9f-bfbf-4cfee09dcbda" path="/var/lib/kubelet/pods/bc6d66de-c4d6-4d9f-bfbf-4cfee09dcbda/volumes" Oct 30 19:09:33.068043 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:33.068028 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f367afff-a23b-47d2-944e-0ab238f13bd0" path="/var/lib/kubelet/pods/f367afff-a23b-47d2-944e-0ab238f13bd0/volumes" Oct 30 19:09:33.352023 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:33.351926 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4a7508-s9c2j" event={"ID":"2df8e39c-34a7-4294-a546-bfdb150db74f","Type":"ContainerStarted","Data":"d84d5af7ae4a789f6973f93eeb805460fa02e6f2f904d3b026d2df8a51a02e96"} Oct 30 19:09:33.403888 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:33.403828 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4a7508-s9c2j" podStartSLOduration=2.403812022 podStartE2EDuration="2.403812022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:33.402803345 +0000 UTC m=+4270.492865685" watchObservedRunningTime="2026-10-30 19:09:33.403812022 +0000 UTC m=+4270.493874361" Oct 30 19:09:34.363345 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:34.363309 4442 generic.go:356] "Generic (PLEG): container finished" podID="2df8e39c-34a7-4294-a546-bfdb150db74f" containerID="d84d5af7ae4a789f6973f93eeb805460fa02e6f2f904d3b026d2df8a51a02e96" exitCode=0 Oct 30 19:09:34.363808 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:34.363404 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4a7508-s9c2j" event={"ID":"2df8e39c-34a7-4294-a546-bfdb150db74f","Type":"ContainerDied","Data":"d84d5af7ae4a789f6973f93eeb805460fa02e6f2f904d3b026d2df8a51a02e96"} Oct 30 19:09:35.560207 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:35.560180 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4a7508-s9c2j" Oct 30 19:09:35.638855 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:35.638704 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2df8e39c-34a7-4294-a546-bfdb150db74f-kube-api-access-fsd8q\" (UniqueName: \"kubernetes.io/projected/2df8e39c-34a7-4294-a546-bfdb150db74f-kube-api-access-fsd8q\") pod \"2df8e39c-34a7-4294-a546-bfdb150db74f\" (UID: \"2df8e39c-34a7-4294-a546-bfdb150db74f\") " Oct 30 19:09:35.639115 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:35.639091 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2df8e39c-34a7-4294-a546-bfdb150db74f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2df8e39c-34a7-4294-a546-bfdb150db74f-maas-api-xks-ca\") pod \"2df8e39c-34a7-4294-a546-bfdb150db74f\" (UID: \"2df8e39c-34a7-4294-a546-bfdb150db74f\") " Oct 30 19:09:35.639414 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:35.639394 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2df8e39c-34a7-4294-a546-bfdb150db74f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2df8e39c-34a7-4294-a546-bfdb150db74f-maas-api-openshift-ca\") pod \"2df8e39c-34a7-4294-a546-bfdb150db74f\" (UID: \"2df8e39c-34a7-4294-a546-bfdb150db74f\") " Oct 30 19:09:35.640029 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:35.639903 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2df8e39c-34a7-4294-a546-bfdb150db74f-maas-api-openshift-ca" pod "2df8e39c-34a7-4294-a546-bfdb150db74f" (UID: "2df8e39c-34a7-4294-a546-bfdb150db74f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:09:35.640737 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:35.640713 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2df8e39c-34a7-4294-a546-bfdb150db74f-maas-api-openshift-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:09:35.641739 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:35.641713 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2df8e39c-34a7-4294-a546-bfdb150db74f-kube-api-access-fsd8q" pod "2df8e39c-34a7-4294-a546-bfdb150db74f" (UID: "2df8e39c-34a7-4294-a546-bfdb150db74f"). InnerVolumeSpecName "kube-api-access-fsd8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:09:35.645862 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:35.645836 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2df8e39c-34a7-4294-a546-bfdb150db74f-maas-api-xks-ca" pod "2df8e39c-34a7-4294-a546-bfdb150db74f" (UID: "2df8e39c-34a7-4294-a546-bfdb150db74f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:09:35.742202 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:35.742122 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2df8e39c-34a7-4294-a546-bfdb150db74f-maas-api-xks-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:09:35.742202 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:35.742205 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsd8q\" (UniqueName: \"kubernetes.io/projected/2df8e39c-34a7-4294-a546-bfdb150db74f-kube-api-access-fsd8q\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:09:36.563201 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:36.557841 4442 generic.go:356] "Generic (PLEG): container finished" podID="cd55e4f9-4d9d-4fba-a53e-fd3403134906" containerID="59c69f00f4e8e3d29ab16a527166616434def2b07649b87dae8f67261cef32ce" exitCode=0 Oct 30 19:09:36.563201 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:36.557992 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" event={"ID":"cd55e4f9-4d9d-4fba-a53e-fd3403134906","Type":"ContainerDied","Data":"59c69f00f4e8e3d29ab16a527166616434def2b07649b87dae8f67261cef32ce"} Oct 30 19:09:36.563201 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:36.558047 4442 scope.go:122] "RemoveContainer" containerID="4b6fff8d24713f5365a2e0787679ac6779d2151c989a6708a3b1b6ce309d68bb" Oct 30 19:09:36.564501 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:36.564477 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4a7508-s9c2j" Oct 30 19:09:36.564604 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:36.564496 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4a7508-s9c2j" event={"ID":"2df8e39c-34a7-4294-a546-bfdb150db74f","Type":"ContainerDied","Data":"4be26db4510ff98f925803c94dcc1f559b4642ba069902ebe4162d2fe4f87df5"} Oct 30 19:09:36.564604 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:36.564555 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4be26db4510ff98f925803c94dcc1f559b4642ba069902ebe4162d2fe4f87df5" Oct 30 19:09:36.631826 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:36.631798 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" Oct 30 19:09:36.765777 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:36.765744 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd55e4f9-4d9d-4fba-a53e-fd3403134906-kube-api-access-vgck5\" (UniqueName: \"kubernetes.io/projected/cd55e4f9-4d9d-4fba-a53e-fd3403134906-kube-api-access-vgck5\") pod \"cd55e4f9-4d9d-4fba-a53e-fd3403134906\" (UID: \"cd55e4f9-4d9d-4fba-a53e-fd3403134906\") " Oct 30 19:09:36.765938 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:36.765825 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd55e4f9-4d9d-4fba-a53e-fd3403134906-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd55e4f9-4d9d-4fba-a53e-fd3403134906-istio-data\") pod \"cd55e4f9-4d9d-4fba-a53e-fd3403134906\" (UID: \"cd55e4f9-4d9d-4fba-a53e-fd3403134906\") " Oct 30 19:09:36.765938 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:36.765918 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd55e4f9-4d9d-4fba-a53e-fd3403134906-istio-token\" (UniqueName: \"kubernetes.io/projected/cd55e4f9-4d9d-4fba-a53e-fd3403134906-istio-token\") pod \"cd55e4f9-4d9d-4fba-a53e-fd3403134906\" (UID: \"cd55e4f9-4d9d-4fba-a53e-fd3403134906\") " Oct 30 19:09:36.766040 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:36.765951 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd55e4f9-4d9d-4fba-a53e-fd3403134906-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd55e4f9-4d9d-4fba-a53e-fd3403134906-workload-socket\") pod \"cd55e4f9-4d9d-4fba-a53e-fd3403134906\" (UID: \"cd55e4f9-4d9d-4fba-a53e-fd3403134906\") " Oct 30 19:09:36.766040 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:36.766004 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd55e4f9-4d9d-4fba-a53e-fd3403134906-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd55e4f9-4d9d-4fba-a53e-fd3403134906-istio-envoy\") pod \"cd55e4f9-4d9d-4fba-a53e-fd3403134906\" (UID: \"cd55e4f9-4d9d-4fba-a53e-fd3403134906\") " Oct 30 19:09:36.766223 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:36.766096 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd55e4f9-4d9d-4fba-a53e-fd3403134906-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd55e4f9-4d9d-4fba-a53e-fd3403134906-istiod-ca-cert\") pod \"cd55e4f9-4d9d-4fba-a53e-fd3403134906\" (UID: \"cd55e4f9-4d9d-4fba-a53e-fd3403134906\") " Oct 30 19:09:36.766223 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:36.766114 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd55e4f9-4d9d-4fba-a53e-fd3403134906-istio-data" pod "cd55e4f9-4d9d-4fba-a53e-fd3403134906" (UID: "cd55e4f9-4d9d-4fba-a53e-fd3403134906"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:09:36.766339 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:36.766316 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd55e4f9-4d9d-4fba-a53e-fd3403134906-workload-socket" pod "cd55e4f9-4d9d-4fba-a53e-fd3403134906" (UID: "cd55e4f9-4d9d-4fba-a53e-fd3403134906"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:09:36.766558 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:36.766536 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd55e4f9-4d9d-4fba-a53e-fd3403134906-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd55e4f9-4d9d-4fba-a53e-fd3403134906-credential-socket\") pod \"cd55e4f9-4d9d-4fba-a53e-fd3403134906\" (UID: \"cd55e4f9-4d9d-4fba-a53e-fd3403134906\") " Oct 30 19:09:36.766632 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:36.766590 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd55e4f9-4d9d-4fba-a53e-fd3403134906-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd55e4f9-4d9d-4fba-a53e-fd3403134906-istio-podinfo\") pod \"cd55e4f9-4d9d-4fba-a53e-fd3403134906\" (UID: \"cd55e4f9-4d9d-4fba-a53e-fd3403134906\") " Oct 30 19:09:36.766695 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:36.766681 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd55e4f9-4d9d-4fba-a53e-fd3403134906-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd55e4f9-4d9d-4fba-a53e-fd3403134906-workload-certs\") pod \"cd55e4f9-4d9d-4fba-a53e-fd3403134906\" (UID: \"cd55e4f9-4d9d-4fba-a53e-fd3403134906\") " Oct 30 19:09:36.766812 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:36.766753 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd55e4f9-4d9d-4fba-a53e-fd3403134906-istiod-ca-cert" pod "cd55e4f9-4d9d-4fba-a53e-fd3403134906" (UID: "cd55e4f9-4d9d-4fba-a53e-fd3403134906"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:09:36.767033 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:36.767013 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd55e4f9-4d9d-4fba-a53e-fd3403134906-credential-socket" pod "cd55e4f9-4d9d-4fba-a53e-fd3403134906" (UID: "cd55e4f9-4d9d-4fba-a53e-fd3403134906"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:09:36.767376 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:36.767352 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd55e4f9-4d9d-4fba-a53e-fd3403134906-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:09:36.767376 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:36.767378 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd55e4f9-4d9d-4fba-a53e-fd3403134906-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:09:36.767616 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:36.767393 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd55e4f9-4d9d-4fba-a53e-fd3403134906-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:09:36.767616 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:36.767407 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd55e4f9-4d9d-4fba-a53e-fd3403134906-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:09:36.767616 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:36.767582 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd55e4f9-4d9d-4fba-a53e-fd3403134906-workload-certs" pod "cd55e4f9-4d9d-4fba-a53e-fd3403134906" (UID: "cd55e4f9-4d9d-4fba-a53e-fd3403134906"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:09:36.768768 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:36.768742 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd55e4f9-4d9d-4fba-a53e-fd3403134906-kube-api-access-vgck5" pod "cd55e4f9-4d9d-4fba-a53e-fd3403134906" (UID: "cd55e4f9-4d9d-4fba-a53e-fd3403134906"). InnerVolumeSpecName "kube-api-access-vgck5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:09:36.768871 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:36.768838 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd55e4f9-4d9d-4fba-a53e-fd3403134906-istio-token" pod "cd55e4f9-4d9d-4fba-a53e-fd3403134906" (UID: "cd55e4f9-4d9d-4fba-a53e-fd3403134906"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:09:36.769516 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:36.769490 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd55e4f9-4d9d-4fba-a53e-fd3403134906-istio-envoy" pod "cd55e4f9-4d9d-4fba-a53e-fd3403134906" (UID: "cd55e4f9-4d9d-4fba-a53e-fd3403134906"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:09:36.769688 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:36.769667 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd55e4f9-4d9d-4fba-a53e-fd3403134906-istio-podinfo" pod "cd55e4f9-4d9d-4fba-a53e-fd3403134906" (UID: "cd55e4f9-4d9d-4fba-a53e-fd3403134906"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:09:36.885162 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:36.870449 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd55e4f9-4d9d-4fba-a53e-fd3403134906-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:09:36.885162 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:36.870483 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd55e4f9-4d9d-4fba-a53e-fd3403134906-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:09:36.885162 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:36.870501 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd55e4f9-4d9d-4fba-a53e-fd3403134906-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:09:36.885162 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:36.870515 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd55e4f9-4d9d-4fba-a53e-fd3403134906-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:09:36.885162 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:36.870529 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgck5\" (UniqueName: \"kubernetes.io/projected/cd55e4f9-4d9d-4fba-a53e-fd3403134906-kube-api-access-vgck5\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:09:37.575761 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:37.575730 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" Oct 30 19:09:37.576224 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:37.575728 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm" event={"ID":"cd55e4f9-4d9d-4fba-a53e-fd3403134906","Type":"ContainerDied","Data":"0f7749592ef66f80457589258fc9491b6527fb22177c2d718ce7e2aec24d2a3e"} Oct 30 19:09:37.576224 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:37.575874 4442 scope.go:122] "RemoveContainer" containerID="59c69f00f4e8e3d29ab16a527166616434def2b07649b87dae8f67261cef32ce" Oct 30 19:09:37.642329 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:37.642293 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm"] Oct 30 19:09:37.651937 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:37.651906 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-mrkjm"] Oct 30 19:09:39.067672 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:39.067631 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd55e4f9-4d9d-4fba-a53e-fd3403134906" path="/var/lib/kubelet/pods/cd55e4f9-4d9d-4fba-a53e-fd3403134906/volumes" Oct 30 19:09:43.176760 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:43.176726 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cecb19a3-wjgv7"] Oct 30 19:09:43.177464 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:43.177448 4442 cpu_manager.go:414] "containerMap: removing container" podUID="cd55e4f9-4d9d-4fba-a53e-fd3403134906" containerName="istio-proxy" Oct 30 19:09:43.177516 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:43.177465 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd55e4f9-4d9d-4fba-a53e-fd3403134906" containerName="istio-proxy" Oct 30 19:09:43.177516 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:43.177483 4442 cpu_manager.go:414] "containerMap: removing container" podUID="cd55e4f9-4d9d-4fba-a53e-fd3403134906" containerName="istio-proxy" Oct 30 19:09:43.177516 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:43.177489 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd55e4f9-4d9d-4fba-a53e-fd3403134906" containerName="istio-proxy" Oct 30 19:09:43.177516 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:43.177495 4442 cpu_manager.go:414] "containerMap: removing container" podUID="2df8e39c-34a7-4294-a546-bfdb150db74f" containerName="revoke-keys" Oct 30 19:09:43.177516 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:43.177499 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2df8e39c-34a7-4294-a546-bfdb150db74f" containerName="revoke-keys" Oct 30 19:09:43.177679 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:43.177590 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd55e4f9-4d9d-4fba-a53e-fd3403134906" containerName="istio-proxy" Oct 30 19:09:43.177679 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:43.177596 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="2df8e39c-34a7-4294-a546-bfdb150db74f" containerName="revoke-keys" Oct 30 19:09:43.177679 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:43.177613 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd55e4f9-4d9d-4fba-a53e-fd3403134906" containerName="istio-proxy" Oct 30 19:09:43.182737 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:43.182706 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cecb19a3-wjgv7" Oct 30 19:09:43.209807 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:43.209785 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:09:43.212949 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:43.212929 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:09:43.222519 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:43.222491 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cecb19a3-wjgv7"] Oct 30 19:09:43.299755 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:43.299720 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aff128e4-d366-47f4-bcf9-ba1c133122f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cecb19a3-wjgv7\" (UID: \"aff128e4-d366-47f4-bcf9-ba1c133122f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cecb19a3-wjgv7" Oct 30 19:09:43.299917 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:43.299843 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8hnr\" (UniqueName: \"kubernetes.io/projected/aff128e4-d366-47f4-bcf9-ba1c133122f4-kube-api-access-b8hnr\") pod \"maas-api-revoke-sub-cecb19a3-wjgv7\" (UID: \"aff128e4-d366-47f4-bcf9-ba1c133122f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cecb19a3-wjgv7" Oct 30 19:09:43.299979 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:43.299971 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aff128e4-d366-47f4-bcf9-ba1c133122f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cecb19a3-wjgv7\" (UID: \"aff128e4-d366-47f4-bcf9-ba1c133122f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cecb19a3-wjgv7" Oct 30 19:09:43.400989 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:43.400953 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8hnr\" (UniqueName: \"kubernetes.io/projected/aff128e4-d366-47f4-bcf9-ba1c133122f4-kube-api-access-b8hnr\") pod \"maas-api-revoke-sub-cecb19a3-wjgv7\" (UID: \"aff128e4-d366-47f4-bcf9-ba1c133122f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cecb19a3-wjgv7" Oct 30 19:09:43.401202 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:43.401076 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aff128e4-d366-47f4-bcf9-ba1c133122f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cecb19a3-wjgv7\" (UID: \"aff128e4-d366-47f4-bcf9-ba1c133122f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cecb19a3-wjgv7" Oct 30 19:09:43.401290 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:43.401215 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aff128e4-d366-47f4-bcf9-ba1c133122f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cecb19a3-wjgv7\" (UID: \"aff128e4-d366-47f4-bcf9-ba1c133122f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cecb19a3-wjgv7" Oct 30 19:09:43.401851 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:43.401824 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aff128e4-d366-47f4-bcf9-ba1c133122f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cecb19a3-wjgv7\" (UID: \"aff128e4-d366-47f4-bcf9-ba1c133122f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cecb19a3-wjgv7" Oct 30 19:09:43.403760 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:43.403737 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aff128e4-d366-47f4-bcf9-ba1c133122f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cecb19a3-wjgv7\" (UID: \"aff128e4-d366-47f4-bcf9-ba1c133122f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cecb19a3-wjgv7" Oct 30 19:09:43.421419 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:43.421397 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8hnr\" (UniqueName: \"kubernetes.io/projected/aff128e4-d366-47f4-bcf9-ba1c133122f4-kube-api-access-b8hnr\") pod \"maas-api-revoke-sub-cecb19a3-wjgv7\" (UID: \"aff128e4-d366-47f4-bcf9-ba1c133122f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cecb19a3-wjgv7" Oct 30 19:09:43.500587 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:43.500496 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cecb19a3-wjgv7" Oct 30 19:09:43.805422 ip-10-0-138-176 kubenswrapper[4442]: W1030 19:09:43.805391 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff128e4_d366_47f4_bcf9_ba1c133122f4.slice/crio-edac0dd1065c027f690e110dbcab29cb5c9f14b40e00b08ae5bb076dac4edb9d WatchSource:0}: Error finding container edac0dd1065c027f690e110dbcab29cb5c9f14b40e00b08ae5bb076dac4edb9d: Status 404 returned error can't find the container with id edac0dd1065c027f690e110dbcab29cb5c9f14b40e00b08ae5bb076dac4edb9d Oct 30 19:09:43.806459 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:43.806430 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cecb19a3-wjgv7"] Oct 30 19:09:44.662467 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:44.662431 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cecb19a3-wjgv7" event={"ID":"aff128e4-d366-47f4-bcf9-ba1c133122f4","Type":"ContainerStarted","Data":"8cb609731d517ef62f6d0aa166f2da14c5b91b6597eb4f0ef5ab2ddfbca166d3"} Oct 30 19:09:44.662994 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:44.662492 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cecb19a3-wjgv7" event={"ID":"aff128e4-d366-47f4-bcf9-ba1c133122f4","Type":"ContainerStarted","Data":"edac0dd1065c027f690e110dbcab29cb5c9f14b40e00b08ae5bb076dac4edb9d"} Oct 30 19:09:44.713543 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:44.713493 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cecb19a3-wjgv7" podStartSLOduration=1.713478141 podStartE2EDuration="1.713478141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:44.708420873 +0000 UTC m=+4281.798483214" watchObservedRunningTime="2026-10-30 19:09:44.713478141 +0000 UTC m=+4281.803540480" Oct 30 19:09:45.672075 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:45.672038 4442 generic.go:356] "Generic (PLEG): container finished" podID="aff128e4-d366-47f4-bcf9-ba1c133122f4" containerID="8cb609731d517ef62f6d0aa166f2da14c5b91b6597eb4f0ef5ab2ddfbca166d3" exitCode=0 Oct 30 19:09:45.672690 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:45.672125 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cecb19a3-wjgv7" event={"ID":"aff128e4-d366-47f4-bcf9-ba1c133122f4","Type":"ContainerDied","Data":"8cb609731d517ef62f6d0aa166f2da14c5b91b6597eb4f0ef5ab2ddfbca166d3"} Oct 30 19:09:46.903564 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:46.903539 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cecb19a3-wjgv7" Oct 30 19:09:46.999704 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:46.999615 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aff128e4-d366-47f4-bcf9-ba1c133122f4-kube-api-access-b8hnr\" (UniqueName: \"kubernetes.io/projected/aff128e4-d366-47f4-bcf9-ba1c133122f4-kube-api-access-b8hnr\") pod \"aff128e4-d366-47f4-bcf9-ba1c133122f4\" (UID: \"aff128e4-d366-47f4-bcf9-ba1c133122f4\") " Oct 30 19:09:46.999872 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:46.999779 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aff128e4-d366-47f4-bcf9-ba1c133122f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aff128e4-d366-47f4-bcf9-ba1c133122f4-maas-api-xks-ca\") pod \"aff128e4-d366-47f4-bcf9-ba1c133122f4\" (UID: \"aff128e4-d366-47f4-bcf9-ba1c133122f4\") " Oct 30 19:09:46.999872 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:46.999866 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aff128e4-d366-47f4-bcf9-ba1c133122f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aff128e4-d366-47f4-bcf9-ba1c133122f4-maas-api-openshift-ca\") pod \"aff128e4-d366-47f4-bcf9-ba1c133122f4\" (UID: \"aff128e4-d366-47f4-bcf9-ba1c133122f4\") " Oct 30 19:09:47.000334 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:47.000304 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aff128e4-d366-47f4-bcf9-ba1c133122f4-maas-api-openshift-ca" pod "aff128e4-d366-47f4-bcf9-ba1c133122f4" (UID: "aff128e4-d366-47f4-bcf9-ba1c133122f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:09:47.000641 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:47.000457 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aff128e4-d366-47f4-bcf9-ba1c133122f4-maas-api-openshift-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:09:47.002079 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:47.002043 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aff128e4-d366-47f4-bcf9-ba1c133122f4-maas-api-xks-ca" pod "aff128e4-d366-47f4-bcf9-ba1c133122f4" (UID: "aff128e4-d366-47f4-bcf9-ba1c133122f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:09:47.002227 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:47.002078 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff128e4-d366-47f4-bcf9-ba1c133122f4-kube-api-access-b8hnr" pod "aff128e4-d366-47f4-bcf9-ba1c133122f4" (UID: "aff128e4-d366-47f4-bcf9-ba1c133122f4"). InnerVolumeSpecName "kube-api-access-b8hnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:09:47.101676 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:47.101647 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aff128e4-d366-47f4-bcf9-ba1c133122f4-maas-api-xks-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:09:47.101676 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:47.101682 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8hnr\" (UniqueName: \"kubernetes.io/projected/aff128e4-d366-47f4-bcf9-ba1c133122f4-kube-api-access-b8hnr\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:09:47.114901 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:09:47.114875 4442 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff128e4_d366_47f4_bcf9_ba1c133122f4.slice\": RecentStats: unable to find data in memory cache]" Oct 30 19:09:47.562499 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:09:47.562463 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 19:09:47.689314 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:47.689279 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cecb19a3-wjgv7" Oct 30 19:09:47.689510 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:47.689283 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cecb19a3-wjgv7" event={"ID":"aff128e4-d366-47f4-bcf9-ba1c133122f4","Type":"ContainerDied","Data":"edac0dd1065c027f690e110dbcab29cb5c9f14b40e00b08ae5bb076dac4edb9d"} Oct 30 19:09:47.689510 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:47.689401 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edac0dd1065c027f690e110dbcab29cb5c9f14b40e00b08ae5bb076dac4edb9d" Oct 30 19:09:49.464880 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:49.464846 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-09ef1d/rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8"] Oct 30 19:09:49.465860 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:49.465840 4442 cpu_manager.go:414] "containerMap: removing container" podUID="aff128e4-d366-47f4-bcf9-ba1c133122f4" containerName="revoke-keys" Oct 30 19:09:49.465860 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:49.465861 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aff128e4-d366-47f4-bcf9-ba1c133122f4" containerName="revoke-keys" Oct 30 19:09:49.465991 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:49.465945 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="aff128e4-d366-47f4-bcf9-ba1c133122f4" containerName="revoke-keys" Oct 30 19:09:49.469930 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:49.469909 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-09ef1d/rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8" Oct 30 19:09:49.496536 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:49.496502 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-09ef1d/rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8"] Oct 30 19:09:49.501683 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:49.501647 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-09ef1d\"/\"kube-root-ca.crt\"" Oct 30 19:09:49.501683 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:49.501660 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-09ef1d\"/\"rate-test-model-09ef1d-kserve-self-signed-certs\"" Oct 30 19:09:49.501871 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:49.501647 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-09ef1d\"/\"openshift-service-ca.crt\"" Oct 30 19:09:49.501871 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:49.501696 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-09ef1d\"/\"default-dockercfg-b2jfh\"" Oct 30 19:09:49.534882 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:49.534851 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106-dshm\") pod \"rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8\" (UID: \"ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106\") " pod="ai-tenant-e2e-shared-b-w2-09ef1d/rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8" Oct 30 19:09:49.534882 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:49.534885 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106-model-cache\") pod \"rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8\" (UID: \"ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106\") " pod="ai-tenant-e2e-shared-b-w2-09ef1d/rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8" Oct 30 19:09:49.535123 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:49.535011 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106-tmp-dir\") pod \"rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8\" (UID: \"ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106\") " pod="ai-tenant-e2e-shared-b-w2-09ef1d/rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8" Oct 30 19:09:49.535123 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:49.535043 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmb7n\" (UniqueName: \"kubernetes.io/projected/ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106-kube-api-access-cmb7n\") pod \"rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8\" (UID: \"ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106\") " pod="ai-tenant-e2e-shared-b-w2-09ef1d/rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8" Oct 30 19:09:49.535280 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:49.535221 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106-tls-certs\") pod \"rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8\" (UID: \"ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106\") " pod="ai-tenant-e2e-shared-b-w2-09ef1d/rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8" Oct 30 19:09:49.535354 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:49.535336 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106-home\") pod \"rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8\" (UID: \"ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106\") " pod="ai-tenant-e2e-shared-b-w2-09ef1d/rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8" Oct 30 19:09:49.637035 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:49.637001 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106-home\") pod \"rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8\" (UID: \"ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106\") " pod="ai-tenant-e2e-shared-b-w2-09ef1d/rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8" Oct 30 19:09:49.637225 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:49.637113 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106-dshm\") pod \"rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8\" (UID: \"ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106\") " pod="ai-tenant-e2e-shared-b-w2-09ef1d/rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8" Oct 30 19:09:49.637225 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:49.637183 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106-model-cache\") pod \"rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8\" (UID: \"ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106\") " pod="ai-tenant-e2e-shared-b-w2-09ef1d/rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8" Oct 30 19:09:49.637328 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:49.637289 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106-tmp-dir\") pod \"rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8\" (UID: \"ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106\") " pod="ai-tenant-e2e-shared-b-w2-09ef1d/rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8" Oct 30 19:09:49.637374 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:49.637335 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmb7n\" (UniqueName: \"kubernetes.io/projected/ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106-kube-api-access-cmb7n\") pod \"rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8\" (UID: \"ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106\") " pod="ai-tenant-e2e-shared-b-w2-09ef1d/rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8" Oct 30 19:09:49.637460 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:49.637441 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106-tls-certs\") pod \"rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8\" (UID: \"ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106\") " pod="ai-tenant-e2e-shared-b-w2-09ef1d/rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8" Oct 30 19:09:49.637525 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:49.637502 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106-home\") pod \"rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8\" (UID: \"ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106\") " pod="ai-tenant-e2e-shared-b-w2-09ef1d/rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8" Oct 30 19:09:49.637701 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:49.637650 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106-tmp-dir\") pod \"rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8\" (UID: \"ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106\") " pod="ai-tenant-e2e-shared-b-w2-09ef1d/rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8" Oct 30 19:09:49.637915 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:49.637891 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106-model-cache\") pod \"rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8\" (UID: \"ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106\") " pod="ai-tenant-e2e-shared-b-w2-09ef1d/rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8" Oct 30 19:09:49.639638 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:49.639597 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106-dshm\") pod \"rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8\" (UID: \"ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106\") " pod="ai-tenant-e2e-shared-b-w2-09ef1d/rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8" Oct 30 19:09:49.640056 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:49.640037 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106-tls-certs\") pod \"rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8\" (UID: \"ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106\") " pod="ai-tenant-e2e-shared-b-w2-09ef1d/rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8" Oct 30 19:09:49.656591 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:49.656565 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmb7n\" (UniqueName: \"kubernetes.io/projected/ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106-kube-api-access-cmb7n\") pod \"rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8\" (UID: \"ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106\") " pod="ai-tenant-e2e-shared-b-w2-09ef1d/rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8" Oct 30 19:09:49.796305 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:49.796215 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-09ef1d/rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8" Oct 30 19:09:50.103083 ip-10-0-138-176 kubenswrapper[4442]: W1030 19:09:50.103053 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac7e3bc8_8a90_41ac_9ed9_1cad1ae82106.slice/crio-ca69e47baabbee41216bca33324e1a9459c8d2f98c2a69092aa98c9fb8324833 WatchSource:0}: Error finding container ca69e47baabbee41216bca33324e1a9459c8d2f98c2a69092aa98c9fb8324833: Status 404 returned error can't find the container with id ca69e47baabbee41216bca33324e1a9459c8d2f98c2a69092aa98c9fb8324833 Oct 30 19:09:50.103689 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:50.103665 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-09ef1d/rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8"] Oct 30 19:09:50.739151 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:50.739107 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-09ef1d/rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8" event={"ID":"ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106","Type":"ContainerStarted","Data":"b3a71c95aa9b6631e47e40fd84d8b7c66e5859f5b938056b4628c84041ee0ed7"} Oct 30 19:09:50.739586 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:50.739171 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-09ef1d/rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8" event={"ID":"ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106","Type":"ContainerStarted","Data":"ca69e47baabbee41216bca33324e1a9459c8d2f98c2a69092aa98c9fb8324833"} Oct 30 19:09:50.790335 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:50.790284 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-09ef1d/rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8" podStartSLOduration=1.790269738 podStartE2EDuration="1.790269738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:50.787611584 +0000 UTC m=+4287.877673925" watchObservedRunningTime="2026-10-30 19:09:50.790269738 +0000 UTC m=+4287.880332077" Oct 30 19:09:50.797188 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:50.797159 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-09ef1d/rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8" Oct 30 19:09:55.873362 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:55.873325 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-380c08-65f99966c6-8p2v9"] Oct 30 19:09:55.905012 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:55.904980 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-380c08-67478544f9-4pdqp"] Oct 30 19:09:57.991603 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:57.991564 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4a7508-s9c2j"] Oct 30 19:09:58.001309 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:58.001283 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4a7508-s9c2j"] Oct 30 19:09:59.068659 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:59.068625 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2df8e39c-34a7-4294-a546-bfdb150db74f" path="/var/lib/kubelet/pods/2df8e39c-34a7-4294-a546-bfdb150db74f/volumes" Oct 30 19:09:59.594618 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:59.594587 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b49e554a-btjhm"] Oct 30 19:09:59.599361 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:59.599314 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b49e554a-btjhm" Oct 30 19:09:59.619001 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:59.618588 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b49e554a-btjhm"] Oct 30 19:09:59.619525 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:59.619501 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs7nv\" (UniqueName: \"kubernetes.io/projected/f89327e4-347c-4656-89eb-5674c31f9ec8-kube-api-access-vs7nv\") pod \"maas-api-revoke-sub-b49e554a-btjhm\" (UID: \"f89327e4-347c-4656-89eb-5674c31f9ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b49e554a-btjhm" Oct 30 19:09:59.619726 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:59.619705 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f89327e4-347c-4656-89eb-5674c31f9ec8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b49e554a-btjhm\" (UID: \"f89327e4-347c-4656-89eb-5674c31f9ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b49e554a-btjhm" Oct 30 19:09:59.619792 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:59.619764 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f89327e4-347c-4656-89eb-5674c31f9ec8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b49e554a-btjhm\" (UID: \"f89327e4-347c-4656-89eb-5674c31f9ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b49e554a-btjhm" Oct 30 19:09:59.621479 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:59.621453 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:09:59.638461 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:59.638435 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:09:59.720572 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:59.720540 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f89327e4-347c-4656-89eb-5674c31f9ec8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b49e554a-btjhm\" (UID: \"f89327e4-347c-4656-89eb-5674c31f9ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b49e554a-btjhm" Oct 30 19:09:59.720782 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:59.720595 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f89327e4-347c-4656-89eb-5674c31f9ec8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b49e554a-btjhm\" (UID: \"f89327e4-347c-4656-89eb-5674c31f9ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b49e554a-btjhm" Oct 30 19:09:59.720782 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:59.720681 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs7nv\" (UniqueName: \"kubernetes.io/projected/f89327e4-347c-4656-89eb-5674c31f9ec8-kube-api-access-vs7nv\") pod \"maas-api-revoke-sub-b49e554a-btjhm\" (UID: \"f89327e4-347c-4656-89eb-5674c31f9ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b49e554a-btjhm" Oct 30 19:09:59.721411 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:59.721383 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f89327e4-347c-4656-89eb-5674c31f9ec8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b49e554a-btjhm\" (UID: \"f89327e4-347c-4656-89eb-5674c31f9ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b49e554a-btjhm" Oct 30 19:09:59.723079 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:59.723052 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f89327e4-347c-4656-89eb-5674c31f9ec8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b49e554a-btjhm\" (UID: \"f89327e4-347c-4656-89eb-5674c31f9ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b49e554a-btjhm" Oct 30 19:09:59.740986 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:59.740952 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs7nv\" (UniqueName: \"kubernetes.io/projected/f89327e4-347c-4656-89eb-5674c31f9ec8-kube-api-access-vs7nv\") pod \"maas-api-revoke-sub-b49e554a-btjhm\" (UID: \"f89327e4-347c-4656-89eb-5674c31f9ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b49e554a-btjhm" Oct 30 19:09:59.797160 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:59.797099 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-09ef1d/rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8" Oct 30 19:09:59.810989 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:59.810963 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-09ef1d/rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8" Oct 30 19:09:59.828348 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:59.828318 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-09ef1d/rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8" Oct 30 19:09:59.936174 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:09:59.936128 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b49e554a-btjhm" Oct 30 19:10:00.252341 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:00.252311 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b49e554a-btjhm"] Oct 30 19:10:00.252721 ip-10-0-138-176 kubenswrapper[4442]: W1030 19:10:00.252523 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf89327e4_347c_4656_89eb_5674c31f9ec8.slice/crio-b17f57f065b1a71f560edc83e3719a74eb1cac36e0ceba59dbbf5fe0fea189ae WatchSource:0}: Error finding container b17f57f065b1a71f560edc83e3719a74eb1cac36e0ceba59dbbf5fe0fea189ae: Status 404 returned error can't find the container with id b17f57f065b1a71f560edc83e3719a74eb1cac36e0ceba59dbbf5fe0fea189ae Oct 30 19:10:00.828657 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:00.828619 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b49e554a-btjhm" event={"ID":"f89327e4-347c-4656-89eb-5674c31f9ec8","Type":"ContainerStarted","Data":"fd6a2d78739f6a2627f8834dedd93ae3f253fa9084a2cb243dbbefde2d316184"} Oct 30 19:10:00.828852 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:00.828683 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b49e554a-btjhm" event={"ID":"f89327e4-347c-4656-89eb-5674c31f9ec8","Type":"ContainerStarted","Data":"b17f57f065b1a71f560edc83e3719a74eb1cac36e0ceba59dbbf5fe0fea189ae"} Oct 30 19:10:00.871636 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:00.871582 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b49e554a-btjhm" podStartSLOduration=1.871567481 podStartE2EDuration="1.871567481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:00.866261326 +0000 UTC m=+4297.956323666" watchObservedRunningTime="2026-10-30 19:10:00.871567481 +0000 UTC m=+4297.961629840" Oct 30 19:10:00.874183 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:00.874107 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-380c08-65f99966c6-8p2v9" podUID="a3b6ea07-7632-44a9-9343-20b7ac9526c5" containerName="payload-processing" containerID="cri-o://f98a7cabbe20754eb4b650e704ede40269e9fe27cd2a5df9413d1a2465b9cddd" gracePeriod=25 Oct 30 19:10:00.905629 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:00.905543 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-380c08-67478544f9-4pdqp" podUID="9b4cff9f-9eb5-4fc5-a125-0d92216168e6" containerName="payload-pre-processing" containerID="cri-o://7c0527787e5aa1965595b7875173efdb86ae30c58ab6612cbbc58e79582bdd6b" gracePeriod=25 Oct 30 19:10:01.880493 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:01.880016 4442 generic.go:356] "Generic (PLEG): container finished" podID="f89327e4-347c-4656-89eb-5674c31f9ec8" containerID="fd6a2d78739f6a2627f8834dedd93ae3f253fa9084a2cb243dbbefde2d316184" exitCode=0 Oct 30 19:10:01.880493 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:01.880222 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b49e554a-btjhm" event={"ID":"f89327e4-347c-4656-89eb-5674c31f9ec8","Type":"ContainerDied","Data":"fd6a2d78739f6a2627f8834dedd93ae3f253fa9084a2cb243dbbefde2d316184"} Oct 30 19:10:03.062413 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:03.062374 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b49e554a-btjhm" Oct 30 19:10:03.176386 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:03.176296 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f89327e4-347c-4656-89eb-5674c31f9ec8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f89327e4-347c-4656-89eb-5674c31f9ec8-maas-api-openshift-ca\") pod \"f89327e4-347c-4656-89eb-5674c31f9ec8\" (UID: \"f89327e4-347c-4656-89eb-5674c31f9ec8\") " Oct 30 19:10:03.176546 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:03.176407 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f89327e4-347c-4656-89eb-5674c31f9ec8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f89327e4-347c-4656-89eb-5674c31f9ec8-maas-api-xks-ca\") pod \"f89327e4-347c-4656-89eb-5674c31f9ec8\" (UID: \"f89327e4-347c-4656-89eb-5674c31f9ec8\") " Oct 30 19:10:03.176546 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:03.176534 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f89327e4-347c-4656-89eb-5674c31f9ec8-kube-api-access-vs7nv\" (UniqueName: \"kubernetes.io/projected/f89327e4-347c-4656-89eb-5674c31f9ec8-kube-api-access-vs7nv\") pod \"f89327e4-347c-4656-89eb-5674c31f9ec8\" (UID: \"f89327e4-347c-4656-89eb-5674c31f9ec8\") " Oct 30 19:10:03.176762 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:03.176735 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f89327e4-347c-4656-89eb-5674c31f9ec8-maas-api-openshift-ca" pod "f89327e4-347c-4656-89eb-5674c31f9ec8" (UID: "f89327e4-347c-4656-89eb-5674c31f9ec8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:10:03.177271 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:03.177178 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f89327e4-347c-4656-89eb-5674c31f9ec8-maas-api-openshift-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:10:03.178801 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:03.178772 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f89327e4-347c-4656-89eb-5674c31f9ec8-maas-api-xks-ca" pod "f89327e4-347c-4656-89eb-5674c31f9ec8" (UID: "f89327e4-347c-4656-89eb-5674c31f9ec8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:10:03.178978 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:03.178954 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f89327e4-347c-4656-89eb-5674c31f9ec8-kube-api-access-vs7nv" pod "f89327e4-347c-4656-89eb-5674c31f9ec8" (UID: "f89327e4-347c-4656-89eb-5674c31f9ec8"). InnerVolumeSpecName "kube-api-access-vs7nv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:10:03.279320 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:03.279288 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f89327e4-347c-4656-89eb-5674c31f9ec8-maas-api-xks-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:10:03.279320 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:03.279318 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs7nv\" (UniqueName: \"kubernetes.io/projected/f89327e4-347c-4656-89eb-5674c31f9ec8-kube-api-access-vs7nv\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:10:03.898060 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:03.898033 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b49e554a-btjhm" Oct 30 19:10:03.898296 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:03.898066 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b49e554a-btjhm" event={"ID":"f89327e4-347c-4656-89eb-5674c31f9ec8","Type":"ContainerDied","Data":"b17f57f065b1a71f560edc83e3719a74eb1cac36e0ceba59dbbf5fe0fea189ae"} Oct 30 19:10:03.898296 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:03.898127 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b17f57f065b1a71f560edc83e3719a74eb1cac36e0ceba59dbbf5fe0fea189ae" Oct 30 19:10:05.293174 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:05.290281 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-380c08-65f99966c6-8p2v9" podUID="a3b6ea07-7632-44a9-9343-20b7ac9526c5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:10:05.293174 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:05.290719 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-380c08-67478544f9-4pdqp" podUID="9b4cff9f-9eb5-4fc5-a125-0d92216168e6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:10:05.510796 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:05.510709 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr"] Oct 30 19:10:05.511336 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:05.511297 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" podUID="a85dd1f3-b655-4c52-b1ef-c5471365634a" containerName="istio-proxy" containerID="cri-o://2c0cdec945b34a8b050175d5d48fe7c3bd1d39781f53d2d0dd284dab4af85eab" gracePeriod=30 Oct 30 19:10:06.226825 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:06.226778 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-de7e3a-w5vxx"] Oct 30 19:10:06.227982 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:06.227959 4442 cpu_manager.go:414] "containerMap: removing container" podUID="f89327e4-347c-4656-89eb-5674c31f9ec8" containerName="revoke-keys" Oct 30 19:10:06.228046 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:06.227986 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f89327e4-347c-4656-89eb-5674c31f9ec8" containerName="revoke-keys" Oct 30 19:10:06.228194 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:06.228181 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="f89327e4-347c-4656-89eb-5674c31f9ec8" containerName="revoke-keys" Oct 30 19:10:06.232581 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:06.232559 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-de7e3a-w5vxx" Oct 30 19:10:06.251747 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:06.251635 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:10:06.251747 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:06.251738 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:10:06.254364 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:06.254328 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-de7e3a-w5vxx"] Oct 30 19:10:06.399187 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:06.399095 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdjcz\" (UniqueName: \"kubernetes.io/projected/de08df66-6bcc-44a5-bd23-1944c0a4eed1-kube-api-access-rdjcz\") pod \"maas-api-revoke-keys-e2e-worker-w3-de7e3a-w5vxx\" (UID: \"de08df66-6bcc-44a5-bd23-1944c0a4eed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-de7e3a-w5vxx" Oct 30 19:10:06.399708 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:06.399686 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de08df66-6bcc-44a5-bd23-1944c0a4eed1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-de7e3a-w5vxx\" (UID: \"de08df66-6bcc-44a5-bd23-1944c0a4eed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-de7e3a-w5vxx" Oct 30 19:10:06.399809 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:06.399797 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de08df66-6bcc-44a5-bd23-1944c0a4eed1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-de7e3a-w5vxx\" (UID: \"de08df66-6bcc-44a5-bd23-1944c0a4eed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-de7e3a-w5vxx" Oct 30 19:10:06.501770 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:06.501685 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de08df66-6bcc-44a5-bd23-1944c0a4eed1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-de7e3a-w5vxx\" (UID: \"de08df66-6bcc-44a5-bd23-1944c0a4eed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-de7e3a-w5vxx" Oct 30 19:10:06.501935 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:06.501912 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdjcz\" (UniqueName: \"kubernetes.io/projected/de08df66-6bcc-44a5-bd23-1944c0a4eed1-kube-api-access-rdjcz\") pod \"maas-api-revoke-keys-e2e-worker-w3-de7e3a-w5vxx\" (UID: \"de08df66-6bcc-44a5-bd23-1944c0a4eed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-de7e3a-w5vxx" Oct 30 19:10:06.502052 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:06.502036 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de08df66-6bcc-44a5-bd23-1944c0a4eed1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-de7e3a-w5vxx\" (UID: \"de08df66-6bcc-44a5-bd23-1944c0a4eed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-de7e3a-w5vxx" Oct 30 19:10:06.502529 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:06.502507 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de08df66-6bcc-44a5-bd23-1944c0a4eed1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-de7e3a-w5vxx\" (UID: \"de08df66-6bcc-44a5-bd23-1944c0a4eed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-de7e3a-w5vxx" Oct 30 19:10:06.504903 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:06.504872 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de08df66-6bcc-44a5-bd23-1944c0a4eed1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-de7e3a-w5vxx\" (UID: \"de08df66-6bcc-44a5-bd23-1944c0a4eed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-de7e3a-w5vxx" Oct 30 19:10:06.527055 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:06.527012 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdjcz\" (UniqueName: \"kubernetes.io/projected/de08df66-6bcc-44a5-bd23-1944c0a4eed1-kube-api-access-rdjcz\") pod \"maas-api-revoke-keys-e2e-worker-w3-de7e3a-w5vxx\" (UID: \"de08df66-6bcc-44a5-bd23-1944c0a4eed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-de7e3a-w5vxx" Oct 30 19:10:06.553005 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:06.552959 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-de7e3a-w5vxx" Oct 30 19:10:06.861286 ip-10-0-138-176 kubenswrapper[4442]: W1030 19:10:06.861245 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde08df66_6bcc_44a5_bd23_1944c0a4eed1.slice/crio-b9f93a484e148fb3166bc44e2b47df29a18292745ae19fa555b3b43a9b20eb16 WatchSource:0}: Error finding container b9f93a484e148fb3166bc44e2b47df29a18292745ae19fa555b3b43a9b20eb16: Status 404 returned error can't find the container with id b9f93a484e148fb3166bc44e2b47df29a18292745ae19fa555b3b43a9b20eb16 Oct 30 19:10:06.862511 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:06.862391 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-de7e3a-w5vxx"] Oct 30 19:10:07.020355 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:07.020290 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-de7e3a-w5vxx" event={"ID":"de08df66-6bcc-44a5-bd23-1944c0a4eed1","Type":"ContainerStarted","Data":"b9f93a484e148fb3166bc44e2b47df29a18292745ae19fa555b3b43a9b20eb16"} Oct 30 19:10:08.032382 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:08.032345 4442 generic.go:356] "Generic (PLEG): container finished" podID="de08df66-6bcc-44a5-bd23-1944c0a4eed1" containerID="1842bd253d4c09e7632a481163c766517ce78910d9cec408ebaba8b7370a584d" exitCode=0 Oct 30 19:10:08.032382 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:08.032386 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-de7e3a-w5vxx" event={"ID":"de08df66-6bcc-44a5-bd23-1944c0a4eed1","Type":"ContainerDied","Data":"1842bd253d4c09e7632a481163c766517ce78910d9cec408ebaba8b7370a584d"} Oct 30 19:10:09.212264 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:09.212236 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-de7e3a-w5vxx" Oct 30 19:10:09.257989 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:09.257954 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de08df66-6bcc-44a5-bd23-1944c0a4eed1-kube-api-access-rdjcz\" (UniqueName: \"kubernetes.io/projected/de08df66-6bcc-44a5-bd23-1944c0a4eed1-kube-api-access-rdjcz\") pod \"de08df66-6bcc-44a5-bd23-1944c0a4eed1\" (UID: \"de08df66-6bcc-44a5-bd23-1944c0a4eed1\") " Oct 30 19:10:09.258195 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:09.258071 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de08df66-6bcc-44a5-bd23-1944c0a4eed1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de08df66-6bcc-44a5-bd23-1944c0a4eed1-maas-api-xks-ca\") pod \"de08df66-6bcc-44a5-bd23-1944c0a4eed1\" (UID: \"de08df66-6bcc-44a5-bd23-1944c0a4eed1\") " Oct 30 19:10:09.258283 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:09.258244 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de08df66-6bcc-44a5-bd23-1944c0a4eed1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de08df66-6bcc-44a5-bd23-1944c0a4eed1-maas-api-openshift-ca\") pod \"de08df66-6bcc-44a5-bd23-1944c0a4eed1\" (UID: \"de08df66-6bcc-44a5-bd23-1944c0a4eed1\") " Oct 30 19:10:09.259283 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:09.259251 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de08df66-6bcc-44a5-bd23-1944c0a4eed1-maas-api-openshift-ca" pod "de08df66-6bcc-44a5-bd23-1944c0a4eed1" (UID: "de08df66-6bcc-44a5-bd23-1944c0a4eed1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:10:09.269436 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:09.262624 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de08df66-6bcc-44a5-bd23-1944c0a4eed1-kube-api-access-rdjcz" pod "de08df66-6bcc-44a5-bd23-1944c0a4eed1" (UID: "de08df66-6bcc-44a5-bd23-1944c0a4eed1"). InnerVolumeSpecName "kube-api-access-rdjcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:10:09.269436 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:09.266548 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de08df66-6bcc-44a5-bd23-1944c0a4eed1-maas-api-xks-ca" pod "de08df66-6bcc-44a5-bd23-1944c0a4eed1" (UID: "de08df66-6bcc-44a5-bd23-1944c0a4eed1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:10:09.361051 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:09.360962 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdjcz\" (UniqueName: \"kubernetes.io/projected/de08df66-6bcc-44a5-bd23-1944c0a4eed1-kube-api-access-rdjcz\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:10:09.361051 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:09.361003 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de08df66-6bcc-44a5-bd23-1944c0a4eed1-maas-api-xks-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:10:09.361051 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:09.361018 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de08df66-6bcc-44a5-bd23-1944c0a4eed1-maas-api-openshift-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:10:10.051596 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:10.051566 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-de7e3a-w5vxx" Oct 30 19:10:10.051800 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:10.051601 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-de7e3a-w5vxx" event={"ID":"de08df66-6bcc-44a5-bd23-1944c0a4eed1","Type":"ContainerDied","Data":"b9f93a484e148fb3166bc44e2b47df29a18292745ae19fa555b3b43a9b20eb16"} Oct 30 19:10:10.051800 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:10.051686 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9f93a484e148fb3166bc44e2b47df29a18292745ae19fa555b3b43a9b20eb16" Oct 30 19:10:10.927779 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:10.927742 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8"] Oct 30 19:10:11.075716 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:11.075684 4442 generic.go:356] "Generic (PLEG): container finished" podID="a85dd1f3-b655-4c52-b1ef-c5471365634a" containerID="2c0cdec945b34a8b050175d5d48fe7c3bd1d39781f53d2d0dd284dab4af85eab" exitCode=0 Oct 30 19:10:11.077430 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:11.077408 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" event={"ID":"a85dd1f3-b655-4c52-b1ef-c5471365634a","Type":"ContainerDied","Data":"2c0cdec945b34a8b050175d5d48fe7c3bd1d39781f53d2d0dd284dab4af85eab"} Oct 30 19:10:11.077501 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:11.077451 4442 scope.go:122] "RemoveContainer" containerID="9c571179f15191af5b7055f6a9a40b007a0b1635912253adc928269ba5044e04" Oct 30 19:10:11.253799 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:11.253775 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" Oct 30 19:10:11.315850 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:11.312550 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a85dd1f3-b655-4c52-b1ef-c5471365634a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a85dd1f3-b655-4c52-b1ef-c5471365634a-istio-podinfo\") pod \"a85dd1f3-b655-4c52-b1ef-c5471365634a\" (UID: \"a85dd1f3-b655-4c52-b1ef-c5471365634a\") " Oct 30 19:10:11.315850 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:11.312732 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a85dd1f3-b655-4c52-b1ef-c5471365634a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a85dd1f3-b655-4c52-b1ef-c5471365634a-workload-socket\") pod \"a85dd1f3-b655-4c52-b1ef-c5471365634a\" (UID: \"a85dd1f3-b655-4c52-b1ef-c5471365634a\") " Oct 30 19:10:11.315850 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:11.312804 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a85dd1f3-b655-4c52-b1ef-c5471365634a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a85dd1f3-b655-4c52-b1ef-c5471365634a-workload-certs\") pod \"a85dd1f3-b655-4c52-b1ef-c5471365634a\" (UID: \"a85dd1f3-b655-4c52-b1ef-c5471365634a\") " Oct 30 19:10:11.315850 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:11.312898 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a85dd1f3-b655-4c52-b1ef-c5471365634a-istio-token\" (UniqueName: \"kubernetes.io/projected/a85dd1f3-b655-4c52-b1ef-c5471365634a-istio-token\") pod \"a85dd1f3-b655-4c52-b1ef-c5471365634a\" (UID: \"a85dd1f3-b655-4c52-b1ef-c5471365634a\") " Oct 30 19:10:11.315850 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:11.312957 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a85dd1f3-b655-4c52-b1ef-c5471365634a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a85dd1f3-b655-4c52-b1ef-c5471365634a-istiod-ca-cert\") pod \"a85dd1f3-b655-4c52-b1ef-c5471365634a\" (UID: \"a85dd1f3-b655-4c52-b1ef-c5471365634a\") " Oct 30 19:10:11.315850 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:11.313067 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a85dd1f3-b655-4c52-b1ef-c5471365634a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a85dd1f3-b655-4c52-b1ef-c5471365634a-credential-socket\") pod \"a85dd1f3-b655-4c52-b1ef-c5471365634a\" (UID: \"a85dd1f3-b655-4c52-b1ef-c5471365634a\") " Oct 30 19:10:11.315850 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:11.313179 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a85dd1f3-b655-4c52-b1ef-c5471365634a-kube-api-access-rfc8v\" (UniqueName: \"kubernetes.io/projected/a85dd1f3-b655-4c52-b1ef-c5471365634a-kube-api-access-rfc8v\") pod \"a85dd1f3-b655-4c52-b1ef-c5471365634a\" (UID: \"a85dd1f3-b655-4c52-b1ef-c5471365634a\") " Oct 30 19:10:11.315850 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:11.313228 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a85dd1f3-b655-4c52-b1ef-c5471365634a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a85dd1f3-b655-4c52-b1ef-c5471365634a-istio-envoy\") pod \"a85dd1f3-b655-4c52-b1ef-c5471365634a\" (UID: \"a85dd1f3-b655-4c52-b1ef-c5471365634a\") " Oct 30 19:10:11.315850 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:11.313293 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a85dd1f3-b655-4c52-b1ef-c5471365634a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a85dd1f3-b655-4c52-b1ef-c5471365634a-istio-data\") pod \"a85dd1f3-b655-4c52-b1ef-c5471365634a\" (UID: \"a85dd1f3-b655-4c52-b1ef-c5471365634a\") " Oct 30 19:10:11.315850 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:11.314399 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a85dd1f3-b655-4c52-b1ef-c5471365634a-istio-data" pod "a85dd1f3-b655-4c52-b1ef-c5471365634a" (UID: "a85dd1f3-b655-4c52-b1ef-c5471365634a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:11.315850 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:11.315104 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a85dd1f3-b655-4c52-b1ef-c5471365634a-istiod-ca-cert" pod "a85dd1f3-b655-4c52-b1ef-c5471365634a" (UID: "a85dd1f3-b655-4c52-b1ef-c5471365634a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:10:11.315850 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:11.315366 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a85dd1f3-b655-4c52-b1ef-c5471365634a-workload-socket" pod "a85dd1f3-b655-4c52-b1ef-c5471365634a" (UID: "a85dd1f3-b655-4c52-b1ef-c5471365634a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:11.315850 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:11.315579 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a85dd1f3-b655-4c52-b1ef-c5471365634a-workload-certs" pod "a85dd1f3-b655-4c52-b1ef-c5471365634a" (UID: "a85dd1f3-b655-4c52-b1ef-c5471365634a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:11.344104 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:11.342899 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a85dd1f3-b655-4c52-b1ef-c5471365634a-credential-socket" pod "a85dd1f3-b655-4c52-b1ef-c5471365634a" (UID: "a85dd1f3-b655-4c52-b1ef-c5471365634a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:11.385322 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:11.385281 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a85dd1f3-b655-4c52-b1ef-c5471365634a-istio-token" pod "a85dd1f3-b655-4c52-b1ef-c5471365634a" (UID: "a85dd1f3-b655-4c52-b1ef-c5471365634a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:10:11.388937 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:11.388902 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a85dd1f3-b655-4c52-b1ef-c5471365634a-kube-api-access-rfc8v" pod "a85dd1f3-b655-4c52-b1ef-c5471365634a" (UID: "a85dd1f3-b655-4c52-b1ef-c5471365634a"). InnerVolumeSpecName "kube-api-access-rfc8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:10:11.403153 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:11.403099 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a85dd1f3-b655-4c52-b1ef-c5471365634a-istio-envoy" pod "a85dd1f3-b655-4c52-b1ef-c5471365634a" (UID: "a85dd1f3-b655-4c52-b1ef-c5471365634a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:11.403304 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:11.403113 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a85dd1f3-b655-4c52-b1ef-c5471365634a-istio-podinfo" pod "a85dd1f3-b655-4c52-b1ef-c5471365634a" (UID: "a85dd1f3-b655-4c52-b1ef-c5471365634a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:10:11.415173 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:11.414987 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a85dd1f3-b655-4c52-b1ef-c5471365634a-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:10:11.415173 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:11.415015 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a85dd1f3-b655-4c52-b1ef-c5471365634a-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:10:11.415173 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:11.415029 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a85dd1f3-b655-4c52-b1ef-c5471365634a-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:10:11.415173 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:11.415044 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfc8v\" (UniqueName: \"kubernetes.io/projected/a85dd1f3-b655-4c52-b1ef-c5471365634a-kube-api-access-rfc8v\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:10:11.415173 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:11.415056 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a85dd1f3-b655-4c52-b1ef-c5471365634a-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:10:11.415173 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:11.415067 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a85dd1f3-b655-4c52-b1ef-c5471365634a-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:10:11.415173 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:11.415077 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a85dd1f3-b655-4c52-b1ef-c5471365634a-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:10:11.415173 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:11.415088 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a85dd1f3-b655-4c52-b1ef-c5471365634a-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:10:11.415173 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:11.415104 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a85dd1f3-b655-4c52-b1ef-c5471365634a-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:10:12.086385 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:12.086351 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" Oct 30 19:10:12.086841 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:12.086390 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr" event={"ID":"a85dd1f3-b655-4c52-b1ef-c5471365634a","Type":"ContainerDied","Data":"144ca841c1172762fe73ea0035354966b55523940e9a88322c709e5be43d5885"} Oct 30 19:10:12.086841 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:12.086453 4442 scope.go:122] "RemoveContainer" containerID="2c0cdec945b34a8b050175d5d48fe7c3bd1d39781f53d2d0dd284dab4af85eab" Oct 30 19:10:12.150781 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:12.150747 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr"] Oct 30 19:10:12.162573 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:12.162538 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-zb7dr"] Oct 30 19:10:13.067021 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:13.066984 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a85dd1f3-b655-4c52-b1ef-c5471365634a" path="/var/lib/kubelet/pods/a85dd1f3-b655-4c52-b1ef-c5471365634a/volumes" Oct 30 19:10:15.239741 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:15.239702 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-380c08-65f99966c6-8p2v9" podUID="a3b6ea07-7632-44a9-9343-20b7ac9526c5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:10:15.240209 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:15.239707 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-380c08-67478544f9-4pdqp" podUID="9b4cff9f-9eb5-4fc5-a125-0d92216168e6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:10:16.766160 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:16.763498 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-de7e3a-7b788bbfd6-mlkk9"] Oct 30 19:10:16.766823 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:16.766559 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-de7e3a-6768cfb995-99nqx"] Oct 30 19:10:17.562745 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:10:17.562699 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 19:10:21.177211 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:21.177111 4442 generic.go:356] "Generic (PLEG): container finished" podID="9b4cff9f-9eb5-4fc5-a125-0d92216168e6" containerID="7c0527787e5aa1965595b7875173efdb86ae30c58ab6612cbbc58e79582bdd6b" exitCode=0 Oct 30 19:10:21.177950 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:21.177340 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-380c08-67478544f9-4pdqp" event={"ID":"9b4cff9f-9eb5-4fc5-a125-0d92216168e6","Type":"ContainerDied","Data":"7c0527787e5aa1965595b7875173efdb86ae30c58ab6612cbbc58e79582bdd6b"} Oct 30 19:10:21.193500 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:21.193464 4442 generic.go:356] "Generic (PLEG): container finished" podID="a3b6ea07-7632-44a9-9343-20b7ac9526c5" containerID="f98a7cabbe20754eb4b650e704ede40269e9fe27cd2a5df9413d1a2465b9cddd" exitCode=0 Oct 30 19:10:21.193704 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:21.193682 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-380c08-65f99966c6-8p2v9" event={"ID":"a3b6ea07-7632-44a9-9343-20b7ac9526c5","Type":"ContainerDied","Data":"f98a7cabbe20754eb4b650e704ede40269e9fe27cd2a5df9413d1a2465b9cddd"} Oct 30 19:10:21.233537 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:21.233512 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-380c08-65f99966c6-8p2v9" Oct 30 19:10:21.253186 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:21.253155 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-380c08-67478544f9-4pdqp" Oct 30 19:10:21.369700 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:21.369658 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b4cff9f-9eb5-4fc5-a125-0d92216168e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b4cff9f-9eb5-4fc5-a125-0d92216168e6-plugins-config-volume\") pod \"9b4cff9f-9eb5-4fc5-a125-0d92216168e6\" (UID: \"9b4cff9f-9eb5-4fc5-a125-0d92216168e6\") " Oct 30 19:10:21.369700 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:21.369717 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3b6ea07-7632-44a9-9343-20b7ac9526c5-kube-api-access-tflz8\" (UniqueName: \"kubernetes.io/projected/a3b6ea07-7632-44a9-9343-20b7ac9526c5-kube-api-access-tflz8\") pod \"a3b6ea07-7632-44a9-9343-20b7ac9526c5\" (UID: \"a3b6ea07-7632-44a9-9343-20b7ac9526c5\") " Oct 30 19:10:21.369993 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:21.369887 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b4cff9f-9eb5-4fc5-a125-0d92216168e6-kube-api-access-pzsz8\" (UniqueName: \"kubernetes.io/projected/9b4cff9f-9eb5-4fc5-a125-0d92216168e6-kube-api-access-pzsz8\") pod \"9b4cff9f-9eb5-4fc5-a125-0d92216168e6\" (UID: \"9b4cff9f-9eb5-4fc5-a125-0d92216168e6\") " Oct 30 19:10:21.370053 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:21.369996 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3b6ea07-7632-44a9-9343-20b7ac9526c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3b6ea07-7632-44a9-9343-20b7ac9526c5-plugins-config-volume\") pod \"a3b6ea07-7632-44a9-9343-20b7ac9526c5\" (UID: \"a3b6ea07-7632-44a9-9343-20b7ac9526c5\") " Oct 30 19:10:21.370592 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:21.370563 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b4cff9f-9eb5-4fc5-a125-0d92216168e6-plugins-config-volume" pod "9b4cff9f-9eb5-4fc5-a125-0d92216168e6" (UID: "9b4cff9f-9eb5-4fc5-a125-0d92216168e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:10:21.370592 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:21.370567 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3b6ea07-7632-44a9-9343-20b7ac9526c5-plugins-config-volume" pod "a3b6ea07-7632-44a9-9343-20b7ac9526c5" (UID: "a3b6ea07-7632-44a9-9343-20b7ac9526c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:10:21.371214 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:21.371195 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b4cff9f-9eb5-4fc5-a125-0d92216168e6-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:10:21.371290 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:21.371221 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3b6ea07-7632-44a9-9343-20b7ac9526c5-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:10:21.372611 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:21.372588 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3b6ea07-7632-44a9-9343-20b7ac9526c5-kube-api-access-tflz8" pod "a3b6ea07-7632-44a9-9343-20b7ac9526c5" (UID: "a3b6ea07-7632-44a9-9343-20b7ac9526c5"). InnerVolumeSpecName "kube-api-access-tflz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:10:21.372739 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:21.372723 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b4cff9f-9eb5-4fc5-a125-0d92216168e6-kube-api-access-pzsz8" pod "9b4cff9f-9eb5-4fc5-a125-0d92216168e6" (UID: "9b4cff9f-9eb5-4fc5-a125-0d92216168e6"). InnerVolumeSpecName "kube-api-access-pzsz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:10:21.472516 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:21.472435 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzsz8\" (UniqueName: \"kubernetes.io/projected/9b4cff9f-9eb5-4fc5-a125-0d92216168e6-kube-api-access-pzsz8\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:10:21.472516 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:21.472463 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tflz8\" (UniqueName: \"kubernetes.io/projected/a3b6ea07-7632-44a9-9343-20b7ac9526c5-kube-api-access-tflz8\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:10:21.764856 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:21.764749 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-de7e3a-7b788bbfd6-mlkk9" podUID="fc4a9a79-c6c6-4660-ac88-16920ac96298" containerName="payload-processing" containerID="cri-o://d98bf67132b75a1ebd692bdf37c97569b86164b9669518d76d313eee15a2f45f" gracePeriod=25 Oct 30 19:10:21.767323 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:21.767289 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-de7e3a-6768cfb995-99nqx" podUID="c4f54adc-5fd4-45b0-acb5-761613564059" containerName="payload-pre-processing" containerID="cri-o://c457c428be9fa917a05438e3f488d88ce84b33caf6025743d9416a7f8807c1c0" gracePeriod=25 Oct 30 19:10:22.203686 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:22.203649 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-380c08-65f99966c6-8p2v9" Oct 30 19:10:22.204246 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:22.203700 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-380c08-65f99966c6-8p2v9" event={"ID":"a3b6ea07-7632-44a9-9343-20b7ac9526c5","Type":"ContainerDied","Data":"b4f37265acb6a533ea93a161b14e14e2a8072d19c273a3ba907a9e03dcb269d0"} Oct 30 19:10:22.204246 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:22.203760 4442 scope.go:122] "RemoveContainer" containerID="f98a7cabbe20754eb4b650e704ede40269e9fe27cd2a5df9413d1a2465b9cddd" Oct 30 19:10:22.205529 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:22.205438 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-380c08-67478544f9-4pdqp" Oct 30 19:10:22.205529 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:22.205466 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-380c08-67478544f9-4pdqp" event={"ID":"9b4cff9f-9eb5-4fc5-a125-0d92216168e6","Type":"ContainerDied","Data":"628ddd89de0850e3e41ce17a9b349e713ca443ec58df99f28bcc96eb842a319b"} Oct 30 19:10:22.215289 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:22.215146 4442 scope.go:122] "RemoveContainer" containerID="7c0527787e5aa1965595b7875173efdb86ae30c58ab6612cbbc58e79582bdd6b" Oct 30 19:10:22.255852 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:22.255825 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-380c08-65f99966c6-8p2v9"] Oct 30 19:10:22.267225 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:22.267200 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-380c08-65f99966c6-8p2v9"] Oct 30 19:10:22.299577 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:22.299553 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-380c08-67478544f9-4pdqp"] Oct 30 19:10:22.303200 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:22.303177 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-380c08-67478544f9-4pdqp"] Oct 30 19:10:23.065775 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:23.065744 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b4cff9f-9eb5-4fc5-a125-0d92216168e6" path="/var/lib/kubelet/pods/9b4cff9f-9eb5-4fc5-a125-0d92216168e6/volumes" Oct 30 19:10:23.066268 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:23.066254 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3b6ea07-7632-44a9-9343-20b7ac9526c5" path="/var/lib/kubelet/pods/a3b6ea07-7632-44a9-9343-20b7ac9526c5/volumes" Oct 30 19:10:25.951251 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:25.951209 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8" podUID="1e2901e8-4919-4e5c-88c7-39126a7cca56" containerName="main" containerID="cri-o://a57bd7f7932f4112640be3d875fce6ae10248d7fa9cb1301e325a2e52a810dda" gracePeriod=45 Oct 30 19:10:26.246435 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:26.246399 4442 generic.go:356] "Generic (PLEG): container finished" podID="1e2901e8-4919-4e5c-88c7-39126a7cca56" containerID="a57bd7f7932f4112640be3d875fce6ae10248d7fa9cb1301e325a2e52a810dda" exitCode=0 Oct 30 19:10:26.246615 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:26.246478 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8" event={"ID":"1e2901e8-4919-4e5c-88c7-39126a7cca56","Type":"ContainerDied","Data":"a57bd7f7932f4112640be3d875fce6ae10248d7fa9cb1301e325a2e52a810dda"} Oct 30 19:10:26.246615 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:26.246532 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8" event={"ID":"1e2901e8-4919-4e5c-88c7-39126a7cca56","Type":"ContainerDied","Data":"6bd36c17b22b4339f27badd03092ccb20cd63afddb7f40db4c8f11c6ea0eadfa"} Oct 30 19:10:26.246615 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:26.246546 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bd36c17b22b4339f27badd03092ccb20cd63afddb7f40db4c8f11c6ea0eadfa" Oct 30 19:10:26.253262 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:26.253240 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8" Oct 30 19:10:26.344664 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:26.344634 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e2901e8-4919-4e5c-88c7-39126a7cca56-kube-api-access-zx52v\" (UniqueName: \"kubernetes.io/projected/1e2901e8-4919-4e5c-88c7-39126a7cca56-kube-api-access-zx52v\") pod \"1e2901e8-4919-4e5c-88c7-39126a7cca56\" (UID: \"1e2901e8-4919-4e5c-88c7-39126a7cca56\") " Oct 30 19:10:26.344835 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:26.344739 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e2901e8-4919-4e5c-88c7-39126a7cca56-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e2901e8-4919-4e5c-88c7-39126a7cca56-model-cache\") pod \"1e2901e8-4919-4e5c-88c7-39126a7cca56\" (UID: \"1e2901e8-4919-4e5c-88c7-39126a7cca56\") " Oct 30 19:10:26.344881 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:26.344845 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e2901e8-4919-4e5c-88c7-39126a7cca56-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e2901e8-4919-4e5c-88c7-39126a7cca56-tls-certs\") pod \"1e2901e8-4919-4e5c-88c7-39126a7cca56\" (UID: \"1e2901e8-4919-4e5c-88c7-39126a7cca56\") " Oct 30 19:10:26.344881 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:26.344864 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e2901e8-4919-4e5c-88c7-39126a7cca56-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e2901e8-4919-4e5c-88c7-39126a7cca56-dshm\") pod \"1e2901e8-4919-4e5c-88c7-39126a7cca56\" (UID: \"1e2901e8-4919-4e5c-88c7-39126a7cca56\") " Oct 30 19:10:26.344950 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:26.344912 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e2901e8-4919-4e5c-88c7-39126a7cca56-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e2901e8-4919-4e5c-88c7-39126a7cca56-tmp-dir\") pod \"1e2901e8-4919-4e5c-88c7-39126a7cca56\" (UID: \"1e2901e8-4919-4e5c-88c7-39126a7cca56\") " Oct 30 19:10:26.344996 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:26.344986 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e2901e8-4919-4e5c-88c7-39126a7cca56-home\" (UniqueName: \"kubernetes.io/empty-dir/1e2901e8-4919-4e5c-88c7-39126a7cca56-home\") pod \"1e2901e8-4919-4e5c-88c7-39126a7cca56\" (UID: \"1e2901e8-4919-4e5c-88c7-39126a7cca56\") " Oct 30 19:10:26.345573 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:26.345546 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e2901e8-4919-4e5c-88c7-39126a7cca56-home" pod "1e2901e8-4919-4e5c-88c7-39126a7cca56" (UID: "1e2901e8-4919-4e5c-88c7-39126a7cca56"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:26.346065 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:26.346043 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e2901e8-4919-4e5c-88c7-39126a7cca56-model-cache" pod "1e2901e8-4919-4e5c-88c7-39126a7cca56" (UID: "1e2901e8-4919-4e5c-88c7-39126a7cca56"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:26.346487 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:26.346466 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e2901e8-4919-4e5c-88c7-39126a7cca56-tmp-dir" pod "1e2901e8-4919-4e5c-88c7-39126a7cca56" (UID: "1e2901e8-4919-4e5c-88c7-39126a7cca56"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:26.348721 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:26.348699 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e2901e8-4919-4e5c-88c7-39126a7cca56-tls-certs" pod "1e2901e8-4919-4e5c-88c7-39126a7cca56" (UID: "1e2901e8-4919-4e5c-88c7-39126a7cca56"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:10:26.348856 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:26.348829 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e2901e8-4919-4e5c-88c7-39126a7cca56-dshm" pod "1e2901e8-4919-4e5c-88c7-39126a7cca56" (UID: "1e2901e8-4919-4e5c-88c7-39126a7cca56"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:26.350087 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:26.350065 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e2901e8-4919-4e5c-88c7-39126a7cca56-kube-api-access-zx52v" pod "1e2901e8-4919-4e5c-88c7-39126a7cca56" (UID: "1e2901e8-4919-4e5c-88c7-39126a7cca56"). InnerVolumeSpecName "kube-api-access-zx52v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:10:26.446992 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:26.446958 4442 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e2901e8-4919-4e5c-88c7-39126a7cca56-tls-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:10:26.446992 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:26.446991 4442 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e2901e8-4919-4e5c-88c7-39126a7cca56-dshm\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:10:26.447237 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:26.447005 4442 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e2901e8-4919-4e5c-88c7-39126a7cca56-tmp-dir\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:10:26.447237 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:26.447016 4442 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e2901e8-4919-4e5c-88c7-39126a7cca56-home\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:10:26.447237 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:26.447027 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx52v\" (UniqueName: \"kubernetes.io/projected/1e2901e8-4919-4e5c-88c7-39126a7cca56-kube-api-access-zx52v\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:10:26.447237 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:26.447039 4442 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e2901e8-4919-4e5c-88c7-39126a7cca56-model-cache\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:10:27.263585 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:27.263551 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8" Oct 30 19:10:27.326976 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:27.326898 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8"] Oct 30 19:10:27.339064 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:27.339036 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-de7e3a/distinct-2-w3-de7e3a-kserve-6dd8444546-7bwv8"] Oct 30 19:10:29.095253 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:29.095209 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e2901e8-4919-4e5c-88c7-39126a7cca56" path="/var/lib/kubelet/pods/1e2901e8-4919-4e5c-88c7-39126a7cca56/volumes" Oct 30 19:10:29.100403 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:29.100377 4442 scope.go:122] "RemoveContainer" containerID="a57bd7f7932f4112640be3d875fce6ae10248d7fa9cb1301e325a2e52a810dda" Oct 30 19:10:29.169441 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:29.169402 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-de7e3a-7b788bbfd6-mlkk9" podUID="fc4a9a79-c6c6-4660-ac88-16920ac96298" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:10:29.169441 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:29.169420 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-de7e3a-6768cfb995-99nqx" podUID="c4f54adc-5fd4-45b0-acb5-761613564059" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:10:36.141161 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:36.138289 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb35b1f4-nghc5"] Oct 30 19:10:36.151156 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:36.148152 4442 cpu_manager.go:414] "containerMap: removing container" podUID="a3b6ea07-7632-44a9-9343-20b7ac9526c5" containerName="payload-processing" Oct 30 19:10:36.151156 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:36.148183 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3b6ea07-7632-44a9-9343-20b7ac9526c5" containerName="payload-processing" Oct 30 19:10:36.151156 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:36.148221 4442 cpu_manager.go:414] "containerMap: removing container" podUID="a85dd1f3-b655-4c52-b1ef-c5471365634a" containerName="istio-proxy" Oct 30 19:10:36.151156 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:36.148229 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a85dd1f3-b655-4c52-b1ef-c5471365634a" containerName="istio-proxy" Oct 30 19:10:36.151156 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:36.148255 4442 cpu_manager.go:414] "containerMap: removing container" podUID="1e2901e8-4919-4e5c-88c7-39126a7cca56" containerName="main" Oct 30 19:10:36.151156 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:36.148262 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e2901e8-4919-4e5c-88c7-39126a7cca56" containerName="main" Oct 30 19:10:36.151156 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:36.148303 4442 cpu_manager.go:414] "containerMap: removing container" podUID="9b4cff9f-9eb5-4fc5-a125-0d92216168e6" containerName="payload-pre-processing" Oct 30 19:10:36.151156 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:36.148311 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b4cff9f-9eb5-4fc5-a125-0d92216168e6" containerName="payload-pre-processing" Oct 30 19:10:36.151156 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:36.148334 4442 cpu_manager.go:414] "containerMap: removing container" podUID="de08df66-6bcc-44a5-bd23-1944c0a4eed1" containerName="revoke-keys" Oct 30 19:10:36.151156 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:36.148342 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de08df66-6bcc-44a5-bd23-1944c0a4eed1" containerName="revoke-keys" Oct 30 19:10:36.151156 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:36.148520 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e2901e8-4919-4e5c-88c7-39126a7cca56" containerName="main" Oct 30 19:10:36.151156 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:36.148541 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b4cff9f-9eb5-4fc5-a125-0d92216168e6" containerName="payload-pre-processing" Oct 30 19:10:36.151156 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:36.148552 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="a85dd1f3-b655-4c52-b1ef-c5471365634a" containerName="istio-proxy" Oct 30 19:10:36.151156 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:36.148564 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="de08df66-6bcc-44a5-bd23-1944c0a4eed1" containerName="revoke-keys" Oct 30 19:10:36.151156 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:36.148574 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3b6ea07-7632-44a9-9343-20b7ac9526c5" containerName="payload-processing" Oct 30 19:10:36.151156 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:36.148586 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="a85dd1f3-b655-4c52-b1ef-c5471365634a" containerName="istio-proxy" Oct 30 19:10:36.227245 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:36.199440 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb35b1f4-nghc5" Oct 30 19:10:36.259200 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:36.246118 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:10:36.276157 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:36.263312 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:10:36.303394 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:36.290917 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vb5k\" (UniqueName: \"kubernetes.io/projected/34f236f3-7dde-40d7-a8bc-41c56ede366c-kube-api-access-2vb5k\") pod \"maas-api-revoke-sub-bb35b1f4-nghc5\" (UID: \"34f236f3-7dde-40d7-a8bc-41c56ede366c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb35b1f4-nghc5" Oct 30 19:10:36.303394 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:36.291041 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34f236f3-7dde-40d7-a8bc-41c56ede366c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb35b1f4-nghc5\" (UID: \"34f236f3-7dde-40d7-a8bc-41c56ede366c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb35b1f4-nghc5" Oct 30 19:10:36.303394 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:36.291233 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34f236f3-7dde-40d7-a8bc-41c56ede366c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb35b1f4-nghc5\" (UID: \"34f236f3-7dde-40d7-a8bc-41c56ede366c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb35b1f4-nghc5" Oct 30 19:10:36.304646 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:36.304604 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb35b1f4-nghc5"] Oct 30 19:10:36.393198 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:36.393083 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vb5k\" (UniqueName: \"kubernetes.io/projected/34f236f3-7dde-40d7-a8bc-41c56ede366c-kube-api-access-2vb5k\") pod \"maas-api-revoke-sub-bb35b1f4-nghc5\" (UID: \"34f236f3-7dde-40d7-a8bc-41c56ede366c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb35b1f4-nghc5" Oct 30 19:10:36.393524 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:36.393499 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34f236f3-7dde-40d7-a8bc-41c56ede366c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb35b1f4-nghc5\" (UID: \"34f236f3-7dde-40d7-a8bc-41c56ede366c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb35b1f4-nghc5" Oct 30 19:10:36.393696 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:36.393680 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34f236f3-7dde-40d7-a8bc-41c56ede366c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb35b1f4-nghc5\" (UID: \"34f236f3-7dde-40d7-a8bc-41c56ede366c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb35b1f4-nghc5" Oct 30 19:10:36.394699 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:36.394672 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34f236f3-7dde-40d7-a8bc-41c56ede366c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb35b1f4-nghc5\" (UID: \"34f236f3-7dde-40d7-a8bc-41c56ede366c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb35b1f4-nghc5" Oct 30 19:10:36.404796 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:36.404742 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34f236f3-7dde-40d7-a8bc-41c56ede366c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb35b1f4-nghc5\" (UID: \"34f236f3-7dde-40d7-a8bc-41c56ede366c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb35b1f4-nghc5" Oct 30 19:10:36.416910 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:36.416886 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vb5k\" (UniqueName: \"kubernetes.io/projected/34f236f3-7dde-40d7-a8bc-41c56ede366c-kube-api-access-2vb5k\") pod \"maas-api-revoke-sub-bb35b1f4-nghc5\" (UID: \"34f236f3-7dde-40d7-a8bc-41c56ede366c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb35b1f4-nghc5" Oct 30 19:10:36.709703 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:36.709620 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb35b1f4-nghc5" Oct 30 19:10:37.073105 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:37.073074 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb35b1f4-nghc5"] Oct 30 19:10:37.404548 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:37.404514 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb35b1f4-nghc5" event={"ID":"34f236f3-7dde-40d7-a8bc-41c56ede366c","Type":"ContainerStarted","Data":"92ecf0690bd306f3752ae0743502610994f8c9cab774076a8dcade11bd9425d7"} Oct 30 19:10:38.414610 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:38.414571 4442 generic.go:356] "Generic (PLEG): container finished" podID="34f236f3-7dde-40d7-a8bc-41c56ede366c" containerID="e22862c383f74a970cf2f2c76ac36565690f698e7b14d09da055b830b97e1f69" exitCode=0 Oct 30 19:10:38.415026 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:38.414670 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb35b1f4-nghc5" event={"ID":"34f236f3-7dde-40d7-a8bc-41c56ede366c","Type":"ContainerDied","Data":"e22862c383f74a970cf2f2c76ac36565690f698e7b14d09da055b830b97e1f69"} Oct 30 19:10:39.169492 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:39.169457 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-de7e3a-6768cfb995-99nqx" podUID="c4f54adc-5fd4-45b0-acb5-761613564059" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:10:39.169492 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:39.169474 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-de7e3a-7b788bbfd6-mlkk9" podUID="fc4a9a79-c6c6-4660-ac88-16920ac96298" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:10:39.800560 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:39.800537 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb35b1f4-nghc5" Oct 30 19:10:39.868773 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:39.868708 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34f236f3-7dde-40d7-a8bc-41c56ede366c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34f236f3-7dde-40d7-a8bc-41c56ede366c-maas-api-xks-ca\") pod \"34f236f3-7dde-40d7-a8bc-41c56ede366c\" (UID: \"34f236f3-7dde-40d7-a8bc-41c56ede366c\") " Oct 30 19:10:39.869799 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:39.869758 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34f236f3-7dde-40d7-a8bc-41c56ede366c-kube-api-access-2vb5k\" (UniqueName: \"kubernetes.io/projected/34f236f3-7dde-40d7-a8bc-41c56ede366c-kube-api-access-2vb5k\") pod \"34f236f3-7dde-40d7-a8bc-41c56ede366c\" (UID: \"34f236f3-7dde-40d7-a8bc-41c56ede366c\") " Oct 30 19:10:39.870378 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:39.870359 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34f236f3-7dde-40d7-a8bc-41c56ede366c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34f236f3-7dde-40d7-a8bc-41c56ede366c-maas-api-openshift-ca\") pod \"34f236f3-7dde-40d7-a8bc-41c56ede366c\" (UID: \"34f236f3-7dde-40d7-a8bc-41c56ede366c\") " Oct 30 19:10:39.870809 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:39.870758 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34f236f3-7dde-40d7-a8bc-41c56ede366c-maas-api-openshift-ca" pod "34f236f3-7dde-40d7-a8bc-41c56ede366c" (UID: "34f236f3-7dde-40d7-a8bc-41c56ede366c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:10:39.871353 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:39.871329 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34f236f3-7dde-40d7-a8bc-41c56ede366c-maas-api-openshift-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:10:39.872266 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:39.872239 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34f236f3-7dde-40d7-a8bc-41c56ede366c-maas-api-xks-ca" pod "34f236f3-7dde-40d7-a8bc-41c56ede366c" (UID: "34f236f3-7dde-40d7-a8bc-41c56ede366c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:10:39.873983 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:39.873960 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34f236f3-7dde-40d7-a8bc-41c56ede366c-kube-api-access-2vb5k" pod "34f236f3-7dde-40d7-a8bc-41c56ede366c" (UID: "34f236f3-7dde-40d7-a8bc-41c56ede366c"). InnerVolumeSpecName "kube-api-access-2vb5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:10:39.972634 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:39.972600 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34f236f3-7dde-40d7-a8bc-41c56ede366c-maas-api-xks-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:10:39.972634 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:39.972633 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vb5k\" (UniqueName: \"kubernetes.io/projected/34f236f3-7dde-40d7-a8bc-41c56ede366c-kube-api-access-2vb5k\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:10:40.443151 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:40.443104 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb35b1f4-nghc5" Oct 30 19:10:40.443369 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:40.443164 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb35b1f4-nghc5" event={"ID":"34f236f3-7dde-40d7-a8bc-41c56ede366c","Type":"ContainerDied","Data":"92ecf0690bd306f3752ae0743502610994f8c9cab774076a8dcade11bd9425d7"} Oct 30 19:10:40.443369 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:40.443221 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92ecf0690bd306f3752ae0743502610994f8c9cab774076a8dcade11bd9425d7" Oct 30 19:10:42.081085 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:42.081064 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-de7e3a-7b788bbfd6-mlkk9" Oct 30 19:10:42.100657 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:42.100634 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-de7e3a-6768cfb995-99nqx" Oct 30 19:10:42.209703 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:42.209673 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4f54adc-5fd4-45b0-acb5-761613564059-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4f54adc-5fd4-45b0-acb5-761613564059-plugins-config-volume\") pod \"c4f54adc-5fd4-45b0-acb5-761613564059\" (UID: \"c4f54adc-5fd4-45b0-acb5-761613564059\") " Oct 30 19:10:42.209902 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:42.209784 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4f54adc-5fd4-45b0-acb5-761613564059-kube-api-access-lgsk9\" (UniqueName: \"kubernetes.io/projected/c4f54adc-5fd4-45b0-acb5-761613564059-kube-api-access-lgsk9\") pod \"c4f54adc-5fd4-45b0-acb5-761613564059\" (UID: \"c4f54adc-5fd4-45b0-acb5-761613564059\") " Oct 30 19:10:42.209902 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:42.209888 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc4a9a79-c6c6-4660-ac88-16920ac96298-kube-api-access-r5gk5\" (UniqueName: \"kubernetes.io/projected/fc4a9a79-c6c6-4660-ac88-16920ac96298-kube-api-access-r5gk5\") pod \"fc4a9a79-c6c6-4660-ac88-16920ac96298\" (UID: \"fc4a9a79-c6c6-4660-ac88-16920ac96298\") " Oct 30 19:10:42.210030 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:42.209955 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc4a9a79-c6c6-4660-ac88-16920ac96298-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc4a9a79-c6c6-4660-ac88-16920ac96298-plugins-config-volume\") pod \"fc4a9a79-c6c6-4660-ac88-16920ac96298\" (UID: \"fc4a9a79-c6c6-4660-ac88-16920ac96298\") " Oct 30 19:10:42.210030 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:42.210002 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4f54adc-5fd4-45b0-acb5-761613564059-plugins-config-volume" pod "c4f54adc-5fd4-45b0-acb5-761613564059" (UID: "c4f54adc-5fd4-45b0-acb5-761613564059"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:10:42.210500 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:42.210477 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc4a9a79-c6c6-4660-ac88-16920ac96298-plugins-config-volume" pod "fc4a9a79-c6c6-4660-ac88-16920ac96298" (UID: "fc4a9a79-c6c6-4660-ac88-16920ac96298"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:10:42.210638 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:42.210624 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4f54adc-5fd4-45b0-acb5-761613564059-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:10:42.210703 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:42.210644 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc4a9a79-c6c6-4660-ac88-16920ac96298-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:10:42.212205 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:42.212177 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4f54adc-5fd4-45b0-acb5-761613564059-kube-api-access-lgsk9" pod "c4f54adc-5fd4-45b0-acb5-761613564059" (UID: "c4f54adc-5fd4-45b0-acb5-761613564059"). InnerVolumeSpecName "kube-api-access-lgsk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:10:42.212618 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:42.212600 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc4a9a79-c6c6-4660-ac88-16920ac96298-kube-api-access-r5gk5" pod "fc4a9a79-c6c6-4660-ac88-16920ac96298" (UID: "fc4a9a79-c6c6-4660-ac88-16920ac96298"). InnerVolumeSpecName "kube-api-access-r5gk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:10:42.311886 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:42.311799 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgsk9\" (UniqueName: \"kubernetes.io/projected/c4f54adc-5fd4-45b0-acb5-761613564059-kube-api-access-lgsk9\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:10:42.311886 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:42.311831 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5gk5\" (UniqueName: \"kubernetes.io/projected/fc4a9a79-c6c6-4660-ac88-16920ac96298-kube-api-access-r5gk5\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:10:42.460309 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:42.460269 4442 generic.go:356] "Generic (PLEG): container finished" podID="c4f54adc-5fd4-45b0-acb5-761613564059" containerID="c457c428be9fa917a05438e3f488d88ce84b33caf6025743d9416a7f8807c1c0" exitCode=0 Oct 30 19:10:42.460504 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:42.460318 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-de7e3a-6768cfb995-99nqx" Oct 30 19:10:42.460504 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:42.460355 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-de7e3a-6768cfb995-99nqx" event={"ID":"c4f54adc-5fd4-45b0-acb5-761613564059","Type":"ContainerDied","Data":"c457c428be9fa917a05438e3f488d88ce84b33caf6025743d9416a7f8807c1c0"} Oct 30 19:10:42.460504 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:42.460408 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-de7e3a-6768cfb995-99nqx" event={"ID":"c4f54adc-5fd4-45b0-acb5-761613564059","Type":"ContainerDied","Data":"dcbd0d48f620e6a5af90d91f275db52624fdc39e8b04c649867af3ad3d09e756"} Oct 30 19:10:42.460504 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:42.460429 4442 scope.go:122] "RemoveContainer" containerID="c457c428be9fa917a05438e3f488d88ce84b33caf6025743d9416a7f8807c1c0" Oct 30 19:10:42.461945 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:42.461926 4442 generic.go:356] "Generic (PLEG): container finished" podID="fc4a9a79-c6c6-4660-ac88-16920ac96298" containerID="d98bf67132b75a1ebd692bdf37c97569b86164b9669518d76d313eee15a2f45f" exitCode=0 Oct 30 19:10:42.462263 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:42.461956 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-de7e3a-7b788bbfd6-mlkk9" event={"ID":"fc4a9a79-c6c6-4660-ac88-16920ac96298","Type":"ContainerDied","Data":"d98bf67132b75a1ebd692bdf37c97569b86164b9669518d76d313eee15a2f45f"} Oct 30 19:10:42.462263 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:42.462012 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-de7e3a-7b788bbfd6-mlkk9" event={"ID":"fc4a9a79-c6c6-4660-ac88-16920ac96298","Type":"ContainerDied","Data":"b49b514d8684101497137cfe1b75d2b140550de3237a556e3908f04e984c74ce"} Oct 30 19:10:42.462263 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:42.461967 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-de7e3a-7b788bbfd6-mlkk9" Oct 30 19:10:42.472558 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:42.472534 4442 scope.go:122] "RemoveContainer" containerID="c457c428be9fa917a05438e3f488d88ce84b33caf6025743d9416a7f8807c1c0" Oct 30 19:10:42.472914 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:10:42.472893 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c457c428be9fa917a05438e3f488d88ce84b33caf6025743d9416a7f8807c1c0\": container with ID starting with c457c428be9fa917a05438e3f488d88ce84b33caf6025743d9416a7f8807c1c0 not found: ID does not exist" containerID="c457c428be9fa917a05438e3f488d88ce84b33caf6025743d9416a7f8807c1c0" Oct 30 19:10:42.473020 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:42.472925 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c457c428be9fa917a05438e3f488d88ce84b33caf6025743d9416a7f8807c1c0"} err="failed to get container status \"c457c428be9fa917a05438e3f488d88ce84b33caf6025743d9416a7f8807c1c0\": rpc error: code = NotFound desc = could not find container \"c457c428be9fa917a05438e3f488d88ce84b33caf6025743d9416a7f8807c1c0\": container with ID starting with c457c428be9fa917a05438e3f488d88ce84b33caf6025743d9416a7f8807c1c0 not found: ID does not exist" Oct 30 19:10:42.473020 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:42.472948 4442 scope.go:122] "RemoveContainer" containerID="d98bf67132b75a1ebd692bdf37c97569b86164b9669518d76d313eee15a2f45f" Oct 30 19:10:42.482922 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:42.482904 4442 scope.go:122] "RemoveContainer" containerID="d98bf67132b75a1ebd692bdf37c97569b86164b9669518d76d313eee15a2f45f" Oct 30 19:10:42.483286 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:10:42.483263 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d98bf67132b75a1ebd692bdf37c97569b86164b9669518d76d313eee15a2f45f\": container with ID starting with d98bf67132b75a1ebd692bdf37c97569b86164b9669518d76d313eee15a2f45f not found: ID does not exist" containerID="d98bf67132b75a1ebd692bdf37c97569b86164b9669518d76d313eee15a2f45f" Oct 30 19:10:42.483361 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:42.483295 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d98bf67132b75a1ebd692bdf37c97569b86164b9669518d76d313eee15a2f45f"} err="failed to get container status \"d98bf67132b75a1ebd692bdf37c97569b86164b9669518d76d313eee15a2f45f\": rpc error: code = NotFound desc = could not find container \"d98bf67132b75a1ebd692bdf37c97569b86164b9669518d76d313eee15a2f45f\": container with ID starting with d98bf67132b75a1ebd692bdf37c97569b86164b9669518d76d313eee15a2f45f not found: ID does not exist" Oct 30 19:10:42.514247 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:42.514208 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-de7e3a-6768cfb995-99nqx"] Oct 30 19:10:42.518455 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:42.518430 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-de7e3a-6768cfb995-99nqx"] Oct 30 19:10:42.552071 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:42.552037 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-de7e3a-7b788bbfd6-mlkk9"] Oct 30 19:10:42.555687 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:42.555666 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-de7e3a-7b788bbfd6-mlkk9"] Oct 30 19:10:43.067041 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:43.066964 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4f54adc-5fd4-45b0-acb5-761613564059" path="/var/lib/kubelet/pods/c4f54adc-5fd4-45b0-acb5-761613564059/volumes" Oct 30 19:10:43.067466 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:10:43.067450 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc4a9a79-c6c6-4660-ac88-16920ac96298" path="/var/lib/kubelet/pods/fc4a9a79-c6c6-4660-ac88-16920ac96298/volumes" Oct 30 19:10:47.603159 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:10:47.602767 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 19:11:05.305343 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:11:05.305255 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-de7e3a-78ddc7c46b-lv7hj"] Oct 30 19:11:05.305867 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:11:05.305520 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-de7e3a-78ddc7c46b-lv7hj" podUID="3884511b-b54a-4d44-94b5-69e405ffce06" containerName="maas-api" containerID="cri-o://8f2bfbaa1fd86cc0492716057d304eddf82d07f631deeef1ee4a2ccbe3a21375" gracePeriod=30 Oct 30 19:11:05.676873 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:11:05.676843 4442 generic.go:356] "Generic (PLEG): container finished" podID="3884511b-b54a-4d44-94b5-69e405ffce06" containerID="8f2bfbaa1fd86cc0492716057d304eddf82d07f631deeef1ee4a2ccbe3a21375" exitCode=0 Oct 30 19:11:05.677012 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:11:05.676913 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-de7e3a-78ddc7c46b-lv7hj" event={"ID":"3884511b-b54a-4d44-94b5-69e405ffce06","Type":"ContainerDied","Data":"8f2bfbaa1fd86cc0492716057d304eddf82d07f631deeef1ee4a2ccbe3a21375"} Oct 30 19:11:05.742717 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:11:05.742352 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-de7e3a-78ddc7c46b-lv7hj" Oct 30 19:11:05.842988 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:11:05.842957 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3884511b-b54a-4d44-94b5-69e405ffce06-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3884511b-b54a-4d44-94b5-69e405ffce06-metrics-certs\") pod \"3884511b-b54a-4d44-94b5-69e405ffce06\" (UID: \"3884511b-b54a-4d44-94b5-69e405ffce06\") " Oct 30 19:11:05.843264 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:11:05.843024 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3884511b-b54a-4d44-94b5-69e405ffce06-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3884511b-b54a-4d44-94b5-69e405ffce06-openshift-service-ca\") pod \"3884511b-b54a-4d44-94b5-69e405ffce06\" (UID: \"3884511b-b54a-4d44-94b5-69e405ffce06\") " Oct 30 19:11:05.843264 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:11:05.843151 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3884511b-b54a-4d44-94b5-69e405ffce06-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3884511b-b54a-4d44-94b5-69e405ffce06-maas-api-tls\") pod \"3884511b-b54a-4d44-94b5-69e405ffce06\" (UID: \"3884511b-b54a-4d44-94b5-69e405ffce06\") " Oct 30 19:11:05.843404 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:11:05.843335 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3884511b-b54a-4d44-94b5-69e405ffce06-kube-api-access-gj9g7\" (UniqueName: \"kubernetes.io/projected/3884511b-b54a-4d44-94b5-69e405ffce06-kube-api-access-gj9g7\") pod \"3884511b-b54a-4d44-94b5-69e405ffce06\" (UID: \"3884511b-b54a-4d44-94b5-69e405ffce06\") " Oct 30 19:11:05.843847 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:11:05.843823 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3884511b-b54a-4d44-94b5-69e405ffce06-openshift-service-ca" pod "3884511b-b54a-4d44-94b5-69e405ffce06" (UID: "3884511b-b54a-4d44-94b5-69e405ffce06"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:11:05.859434 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:11:05.859400 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3884511b-b54a-4d44-94b5-69e405ffce06-kube-api-access-gj9g7" pod "3884511b-b54a-4d44-94b5-69e405ffce06" (UID: "3884511b-b54a-4d44-94b5-69e405ffce06"). InnerVolumeSpecName "kube-api-access-gj9g7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:11:05.859770 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:11:05.859730 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3884511b-b54a-4d44-94b5-69e405ffce06-metrics-certs" pod "3884511b-b54a-4d44-94b5-69e405ffce06" (UID: "3884511b-b54a-4d44-94b5-69e405ffce06"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:11:05.859980 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:11:05.859953 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3884511b-b54a-4d44-94b5-69e405ffce06-maas-api-tls" pod "3884511b-b54a-4d44-94b5-69e405ffce06" (UID: "3884511b-b54a-4d44-94b5-69e405ffce06"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:11:05.945635 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:11:05.945552 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj9g7\" (UniqueName: \"kubernetes.io/projected/3884511b-b54a-4d44-94b5-69e405ffce06-kube-api-access-gj9g7\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:11:05.945635 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:11:05.945586 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3884511b-b54a-4d44-94b5-69e405ffce06-metrics-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:11:05.945635 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:11:05.945596 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3884511b-b54a-4d44-94b5-69e405ffce06-openshift-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:11:05.945635 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:11:05.945605 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3884511b-b54a-4d44-94b5-69e405ffce06-maas-api-tls\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:11:06.143649 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:11:06.143608 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-de7e3a-w5vxx"] Oct 30 19:11:06.147452 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:11:06.147424 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-de7e3a-w5vxx"] Oct 30 19:11:06.687372 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:11:06.687341 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-de7e3a-78ddc7c46b-lv7hj" Oct 30 19:11:06.687372 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:11:06.687366 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-de7e3a-78ddc7c46b-lv7hj" event={"ID":"3884511b-b54a-4d44-94b5-69e405ffce06","Type":"ContainerDied","Data":"970b65a251a7c2c2d321e7564c3f2b29261fd0e3fa4173a1c8528a9bef21fc9f"} Oct 30 19:11:06.687945 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:11:06.687431 4442 scope.go:122] "RemoveContainer" containerID="8f2bfbaa1fd86cc0492716057d304eddf82d07f631deeef1ee4a2ccbe3a21375" Oct 30 19:11:06.739811 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:11:06.739753 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-de7e3a-78ddc7c46b-lv7hj"] Oct 30 19:11:06.750760 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:11:06.750722 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-de7e3a-78ddc7c46b-lv7hj"] Oct 30 19:11:07.081828 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:11:07.081750 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3884511b-b54a-4d44-94b5-69e405ffce06" path="/var/lib/kubelet/pods/3884511b-b54a-4d44-94b5-69e405ffce06/volumes" Oct 30 19:11:07.082298 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:11:07.082283 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de08df66-6bcc-44a5-bd23-1944c0a4eed1" path="/var/lib/kubelet/pods/de08df66-6bcc-44a5-bd23-1944c0a4eed1/volumes" Oct 30 19:11:17.560663 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:11:17.560619 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 19:11:47.562605 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:11:47.562568 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 19:12:17.560690 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:12:17.560651 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 19:12:24.896030 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:24.895986 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a33da54-qffvt"] Oct 30 19:12:24.897657 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:24.897631 4442 cpu_manager.go:414] "containerMap: removing container" podUID="fc4a9a79-c6c6-4660-ac88-16920ac96298" containerName="payload-processing" Oct 30 19:12:24.897829 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:24.897814 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc4a9a79-c6c6-4660-ac88-16920ac96298" containerName="payload-processing" Oct 30 19:12:24.897957 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:24.897945 4442 cpu_manager.go:414] "containerMap: removing container" podUID="34f236f3-7dde-40d7-a8bc-41c56ede366c" containerName="revoke-keys" Oct 30 19:12:24.898052 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:24.898040 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34f236f3-7dde-40d7-a8bc-41c56ede366c" containerName="revoke-keys" Oct 30 19:12:24.898192 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:24.898177 4442 cpu_manager.go:414] "containerMap: removing container" podUID="a85dd1f3-b655-4c52-b1ef-c5471365634a" containerName="istio-proxy" Oct 30 19:12:24.898308 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:24.898295 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a85dd1f3-b655-4c52-b1ef-c5471365634a" containerName="istio-proxy" Oct 30 19:12:24.898423 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:24.898410 4442 cpu_manager.go:414] "containerMap: removing container" podUID="3884511b-b54a-4d44-94b5-69e405ffce06" containerName="maas-api" Oct 30 19:12:24.898515 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:24.898503 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3884511b-b54a-4d44-94b5-69e405ffce06" containerName="maas-api" Oct 30 19:12:24.898631 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:24.898619 4442 cpu_manager.go:414] "containerMap: removing container" podUID="c4f54adc-5fd4-45b0-acb5-761613564059" containerName="payload-pre-processing" Oct 30 19:12:24.898725 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:24.898711 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4f54adc-5fd4-45b0-acb5-761613564059" containerName="payload-pre-processing" Oct 30 19:12:24.899031 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:24.899010 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc4a9a79-c6c6-4660-ac88-16920ac96298" containerName="payload-processing" Oct 30 19:12:24.899189 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:24.899175 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="34f236f3-7dde-40d7-a8bc-41c56ede366c" containerName="revoke-keys" Oct 30 19:12:24.899299 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:24.899286 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="3884511b-b54a-4d44-94b5-69e405ffce06" containerName="maas-api" Oct 30 19:12:24.899407 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:24.899396 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4f54adc-5fd4-45b0-acb5-761613564059" containerName="payload-pre-processing" Oct 30 19:12:24.903956 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:24.903936 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a33da54-qffvt" Oct 30 19:12:24.918371 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:24.918338 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a33da54-qffvt"] Oct 30 19:12:24.928159 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:24.925414 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:12:24.928159 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:24.925551 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:12:25.059159 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:25.055458 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2f660b9-2642-4b8a-95ff-c090e33ae02e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a33da54-qffvt\" (UID: \"d2f660b9-2642-4b8a-95ff-c090e33ae02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a33da54-qffvt" Oct 30 19:12:25.059159 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:25.055517 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2f660b9-2642-4b8a-95ff-c090e33ae02e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a33da54-qffvt\" (UID: \"d2f660b9-2642-4b8a-95ff-c090e33ae02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a33da54-qffvt" Oct 30 19:12:25.059159 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:25.055828 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-997tm\" (UniqueName: \"kubernetes.io/projected/d2f660b9-2642-4b8a-95ff-c090e33ae02e-kube-api-access-997tm\") pod \"maas-api-revoke-sub-8a33da54-qffvt\" (UID: \"d2f660b9-2642-4b8a-95ff-c090e33ae02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a33da54-qffvt" Oct 30 19:12:25.157808 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:25.157702 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2f660b9-2642-4b8a-95ff-c090e33ae02e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a33da54-qffvt\" (UID: \"d2f660b9-2642-4b8a-95ff-c090e33ae02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a33da54-qffvt" Oct 30 19:12:25.159672 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:25.159641 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2f660b9-2642-4b8a-95ff-c090e33ae02e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a33da54-qffvt\" (UID: \"d2f660b9-2642-4b8a-95ff-c090e33ae02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a33da54-qffvt" Oct 30 19:12:25.160461 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:25.160427 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-997tm\" (UniqueName: \"kubernetes.io/projected/d2f660b9-2642-4b8a-95ff-c090e33ae02e-kube-api-access-997tm\") pod \"maas-api-revoke-sub-8a33da54-qffvt\" (UID: \"d2f660b9-2642-4b8a-95ff-c090e33ae02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a33da54-qffvt" Oct 30 19:12:25.160950 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:25.159562 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2f660b9-2642-4b8a-95ff-c090e33ae02e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a33da54-qffvt\" (UID: \"d2f660b9-2642-4b8a-95ff-c090e33ae02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a33da54-qffvt" Oct 30 19:12:25.165176 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:25.163882 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2f660b9-2642-4b8a-95ff-c090e33ae02e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a33da54-qffvt\" (UID: \"d2f660b9-2642-4b8a-95ff-c090e33ae02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a33da54-qffvt" Oct 30 19:12:25.188771 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:25.188732 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-997tm\" (UniqueName: \"kubernetes.io/projected/d2f660b9-2642-4b8a-95ff-c090e33ae02e-kube-api-access-997tm\") pod \"maas-api-revoke-sub-8a33da54-qffvt\" (UID: \"d2f660b9-2642-4b8a-95ff-c090e33ae02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a33da54-qffvt" Oct 30 19:12:25.229995 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:25.229966 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a33da54-qffvt" Oct 30 19:12:25.595120 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:25.595093 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a33da54-qffvt"] Oct 30 19:12:25.595475 ip-10-0-138-176 kubenswrapper[4442]: W1030 19:12:25.595453 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2f660b9_2642_4b8a_95ff_c090e33ae02e.slice/crio-02e4fd3973d09238b48fc3b3909fcd981e418c01de64a9e058a6efdb096b5a33 WatchSource:0}: Error finding container 02e4fd3973d09238b48fc3b3909fcd981e418c01de64a9e058a6efdb096b5a33: Status 404 returned error can't find the container with id 02e4fd3973d09238b48fc3b3909fcd981e418c01de64a9e058a6efdb096b5a33 Oct 30 19:12:26.488377 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:26.488335 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a33da54-qffvt" event={"ID":"d2f660b9-2642-4b8a-95ff-c090e33ae02e","Type":"ContainerStarted","Data":"a80f1cec5fbd38e74ddaf8033dd75f787c3e2723075301c1e8ed4d816adf005e"} Oct 30 19:12:26.488818 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:26.488403 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a33da54-qffvt" event={"ID":"d2f660b9-2642-4b8a-95ff-c090e33ae02e","Type":"ContainerStarted","Data":"02e4fd3973d09238b48fc3b3909fcd981e418c01de64a9e058a6efdb096b5a33"} Oct 30 19:12:26.531298 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:26.531244 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a33da54-qffvt" podStartSLOduration=2.531226851 podStartE2EDuration="2.531226851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:26.525965081 +0000 UTC m=+4443.616027422" watchObservedRunningTime="2026-10-30 19:12:26.531226851 +0000 UTC m=+4443.621289248" Oct 30 19:12:27.497084 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:27.497042 4442 generic.go:356] "Generic (PLEG): container finished" podID="d2f660b9-2642-4b8a-95ff-c090e33ae02e" containerID="a80f1cec5fbd38e74ddaf8033dd75f787c3e2723075301c1e8ed4d816adf005e" exitCode=0 Oct 30 19:12:27.497557 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:27.497128 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a33da54-qffvt" event={"ID":"d2f660b9-2642-4b8a-95ff-c090e33ae02e","Type":"ContainerDied","Data":"a80f1cec5fbd38e74ddaf8033dd75f787c3e2723075301c1e8ed4d816adf005e"} Oct 30 19:12:28.663847 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:28.663820 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a33da54-qffvt" Oct 30 19:12:28.733390 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:28.733356 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2f660b9-2642-4b8a-95ff-c090e33ae02e-kube-api-access-997tm\" (UniqueName: \"kubernetes.io/projected/d2f660b9-2642-4b8a-95ff-c090e33ae02e-kube-api-access-997tm\") pod \"d2f660b9-2642-4b8a-95ff-c090e33ae02e\" (UID: \"d2f660b9-2642-4b8a-95ff-c090e33ae02e\") " Oct 30 19:12:28.733566 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:28.733548 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2f660b9-2642-4b8a-95ff-c090e33ae02e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2f660b9-2642-4b8a-95ff-c090e33ae02e-maas-api-xks-ca\") pod \"d2f660b9-2642-4b8a-95ff-c090e33ae02e\" (UID: \"d2f660b9-2642-4b8a-95ff-c090e33ae02e\") " Oct 30 19:12:28.733695 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:28.733677 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2f660b9-2642-4b8a-95ff-c090e33ae02e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2f660b9-2642-4b8a-95ff-c090e33ae02e-maas-api-openshift-ca\") pod \"d2f660b9-2642-4b8a-95ff-c090e33ae02e\" (UID: \"d2f660b9-2642-4b8a-95ff-c090e33ae02e\") " Oct 30 19:12:28.734495 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:28.734212 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2f660b9-2642-4b8a-95ff-c090e33ae02e-maas-api-openshift-ca" pod "d2f660b9-2642-4b8a-95ff-c090e33ae02e" (UID: "d2f660b9-2642-4b8a-95ff-c090e33ae02e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:12:28.734495 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:28.734353 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2f660b9-2642-4b8a-95ff-c090e33ae02e-maas-api-openshift-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:12:28.736330 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:28.736303 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2f660b9-2642-4b8a-95ff-c090e33ae02e-maas-api-xks-ca" pod "d2f660b9-2642-4b8a-95ff-c090e33ae02e" (UID: "d2f660b9-2642-4b8a-95ff-c090e33ae02e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:12:28.737389 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:28.737363 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2f660b9-2642-4b8a-95ff-c090e33ae02e-kube-api-access-997tm" pod "d2f660b9-2642-4b8a-95ff-c090e33ae02e" (UID: "d2f660b9-2642-4b8a-95ff-c090e33ae02e"). InnerVolumeSpecName "kube-api-access-997tm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:12:28.835802 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:28.835720 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-997tm\" (UniqueName: \"kubernetes.io/projected/d2f660b9-2642-4b8a-95ff-c090e33ae02e-kube-api-access-997tm\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:12:28.835802 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:28.835757 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2f660b9-2642-4b8a-95ff-c090e33ae02e-maas-api-xks-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:12:29.513973 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:29.513941 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a33da54-qffvt" Oct 30 19:12:29.514210 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:29.513979 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a33da54-qffvt" event={"ID":"d2f660b9-2642-4b8a-95ff-c090e33ae02e","Type":"ContainerDied","Data":"02e4fd3973d09238b48fc3b3909fcd981e418c01de64a9e058a6efdb096b5a33"} Oct 30 19:12:29.514210 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:29.514036 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02e4fd3973d09238b48fc3b3909fcd981e418c01de64a9e058a6efdb096b5a33" Oct 30 19:12:34.352557 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:34.352515 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1c4bcd-55d686745d-4hhzp"] Oct 30 19:12:34.390438 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:34.390408 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1c4bcd-76ffbffdcf-q4ttx"] Oct 30 19:12:39.352907 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:39.352864 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-1c4bcd-55d686745d-4hhzp" podUID="034e375c-3ae0-40db-94ed-9d328fee5639" containerName="payload-processing" containerID="cri-o://3cc92cf6008580c1750453be5efb7bb823500363f66fa6ef47c52b78a1c51661" gracePeriod=25 Oct 30 19:12:39.391268 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:39.391205 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1c4bcd-76ffbffdcf-q4ttx" podUID="f1e71600-36ee-4faa-872d-da1203e65bdc" containerName="payload-pre-processing" containerID="cri-o://5ed1af2f9609f59c1d498bd70e609d75a638fccd5cf704814b38adda385a986c" gracePeriod=25 Oct 30 19:12:41.194798 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:41.194758 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1c4bcd-76ffbffdcf-q4ttx" podUID="f1e71600-36ee-4faa-872d-da1203e65bdc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:12:41.195331 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:41.194776 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-1c4bcd-55d686745d-4hhzp" podUID="034e375c-3ae0-40db-94ed-9d328fee5639" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:12:45.798750 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:45.798703 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s"] Oct 30 19:12:45.799178 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:45.798908 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" podUID="921aaf3a-c623-4688-8c39-601e42b9d567" containerName="istio-proxy" containerID="cri-o://2943958ae2bc7e045eec02109beb2100116eb274e165f40726bdc0dac7c787c8" gracePeriod=30 Oct 30 19:12:45.810856 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:45.810828 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb"] Oct 30 19:12:45.811032 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:45.811017 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" podUID="49d854e8-50eb-4747-9c43-29f5fcdbc955" containerName="istio-proxy" containerID="cri-o://05f1580b0d3e567e290b24af7fb44afb12c172e7868e2d5d7ee392cea5de669a" gracePeriod=30 Oct 30 19:12:45.814989 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:45.814965 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9"] Oct 30 19:12:45.815181 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:45.815161 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" podUID="e1e466ee-ec7a-4640-bbc5-2f16795107cf" containerName="istio-proxy" containerID="cri-o://57d653004220b68a80462e3bc6e347e90dd37acae6e8a362c5591a364cfe74fa" gracePeriod=30 Oct 30 19:12:45.823224 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:45.823194 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2"] Oct 30 19:12:45.823434 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:45.823408 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" podUID="e03523d4-1b6f-4a80-9006-26b0345d809d" containerName="istio-proxy" containerID="cri-o://535cf080e35833fb2fd74c327e7e095053c76ae3ecfb740a5517847d4f0e60f9" gracePeriod=30 Oct 30 19:12:47.563691 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:12:47.563636 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 19:12:51.147428 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.147341 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" Oct 30 19:12:51.194999 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.194963 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1c4bcd-76ffbffdcf-q4ttx" podUID="f1e71600-36ee-4faa-872d-da1203e65bdc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:12:51.195252 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.195221 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-1c4bcd-55d686745d-4hhzp" podUID="034e375c-3ae0-40db-94ed-9d328fee5639" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:12:51.279364 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.279329 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49d854e8-50eb-4747-9c43-29f5fcdbc955-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49d854e8-50eb-4747-9c43-29f5fcdbc955-istio-envoy\") pod \"49d854e8-50eb-4747-9c43-29f5fcdbc955\" (UID: \"49d854e8-50eb-4747-9c43-29f5fcdbc955\") " Oct 30 19:12:51.279458 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.279408 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49d854e8-50eb-4747-9c43-29f5fcdbc955-istio-token\" (UniqueName: \"kubernetes.io/projected/49d854e8-50eb-4747-9c43-29f5fcdbc955-istio-token\") pod \"49d854e8-50eb-4747-9c43-29f5fcdbc955\" (UID: \"49d854e8-50eb-4747-9c43-29f5fcdbc955\") " Oct 30 19:12:51.279458 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.279441 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49d854e8-50eb-4747-9c43-29f5fcdbc955-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d854e8-50eb-4747-9c43-29f5fcdbc955-workload-socket\") pod \"49d854e8-50eb-4747-9c43-29f5fcdbc955\" (UID: \"49d854e8-50eb-4747-9c43-29f5fcdbc955\") " Oct 30 19:12:51.279571 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.279479 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49d854e8-50eb-4747-9c43-29f5fcdbc955-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49d854e8-50eb-4747-9c43-29f5fcdbc955-istiod-ca-cert\") pod \"49d854e8-50eb-4747-9c43-29f5fcdbc955\" (UID: \"49d854e8-50eb-4747-9c43-29f5fcdbc955\") " Oct 30 19:12:51.279571 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.279532 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49d854e8-50eb-4747-9c43-29f5fcdbc955-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d854e8-50eb-4747-9c43-29f5fcdbc955-credential-socket\") pod \"49d854e8-50eb-4747-9c43-29f5fcdbc955\" (UID: \"49d854e8-50eb-4747-9c43-29f5fcdbc955\") " Oct 30 19:12:51.279744 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.279725 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49d854e8-50eb-4747-9c43-29f5fcdbc955-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49d854e8-50eb-4747-9c43-29f5fcdbc955-workload-certs\") pod \"49d854e8-50eb-4747-9c43-29f5fcdbc955\" (UID: \"49d854e8-50eb-4747-9c43-29f5fcdbc955\") " Oct 30 19:12:51.279955 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.279931 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/49d854e8-50eb-4747-9c43-29f5fcdbc955-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49d854e8-50eb-4747-9c43-29f5fcdbc955-istio-podinfo\") pod \"49d854e8-50eb-4747-9c43-29f5fcdbc955\" (UID: \"49d854e8-50eb-4747-9c43-29f5fcdbc955\") " Oct 30 19:12:51.280080 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.279705 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49d854e8-50eb-4747-9c43-29f5fcdbc955-workload-socket" pod "49d854e8-50eb-4747-9c43-29f5fcdbc955" (UID: "49d854e8-50eb-4747-9c43-29f5fcdbc955"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:12:51.280080 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.279899 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49d854e8-50eb-4747-9c43-29f5fcdbc955-workload-certs" pod "49d854e8-50eb-4747-9c43-29f5fcdbc955" (UID: "49d854e8-50eb-4747-9c43-29f5fcdbc955"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:12:51.280080 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.280003 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49d854e8-50eb-4747-9c43-29f5fcdbc955-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49d854e8-50eb-4747-9c43-29f5fcdbc955-istio-data\") pod \"49d854e8-50eb-4747-9c43-29f5fcdbc955\" (UID: \"49d854e8-50eb-4747-9c43-29f5fcdbc955\") " Oct 30 19:12:51.280080 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.279919 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49d854e8-50eb-4747-9c43-29f5fcdbc955-credential-socket" pod "49d854e8-50eb-4747-9c43-29f5fcdbc955" (UID: "49d854e8-50eb-4747-9c43-29f5fcdbc955"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:12:51.280080 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.279948 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49d854e8-50eb-4747-9c43-29f5fcdbc955-istiod-ca-cert" pod "49d854e8-50eb-4747-9c43-29f5fcdbc955" (UID: "49d854e8-50eb-4747-9c43-29f5fcdbc955"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:12:51.280639 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.280118 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49d854e8-50eb-4747-9c43-29f5fcdbc955-kube-api-access-h92ms\" (UniqueName: \"kubernetes.io/projected/49d854e8-50eb-4747-9c43-29f5fcdbc955-kube-api-access-h92ms\") pod \"49d854e8-50eb-4747-9c43-29f5fcdbc955\" (UID: \"49d854e8-50eb-4747-9c43-29f5fcdbc955\") " Oct 30 19:12:51.280639 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.280338 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49d854e8-50eb-4747-9c43-29f5fcdbc955-istio-data" pod "49d854e8-50eb-4747-9c43-29f5fcdbc955" (UID: "49d854e8-50eb-4747-9c43-29f5fcdbc955"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:12:51.280765 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.280701 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49d854e8-50eb-4747-9c43-29f5fcdbc955-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:12:51.280765 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.280717 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49d854e8-50eb-4747-9c43-29f5fcdbc955-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:12:51.280765 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.280725 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d854e8-50eb-4747-9c43-29f5fcdbc955-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:12:51.280765 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.280733 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49d854e8-50eb-4747-9c43-29f5fcdbc955-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:12:51.280765 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.280741 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d854e8-50eb-4747-9c43-29f5fcdbc955-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:12:51.282400 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.282366 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49d854e8-50eb-4747-9c43-29f5fcdbc955-istio-envoy" pod "49d854e8-50eb-4747-9c43-29f5fcdbc955" (UID: "49d854e8-50eb-4747-9c43-29f5fcdbc955"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:12:51.282512 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.282401 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49d854e8-50eb-4747-9c43-29f5fcdbc955-istio-token" pod "49d854e8-50eb-4747-9c43-29f5fcdbc955" (UID: "49d854e8-50eb-4747-9c43-29f5fcdbc955"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:12:51.282591 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.282569 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49d854e8-50eb-4747-9c43-29f5fcdbc955-kube-api-access-h92ms" pod "49d854e8-50eb-4747-9c43-29f5fcdbc955" (UID: "49d854e8-50eb-4747-9c43-29f5fcdbc955"). InnerVolumeSpecName "kube-api-access-h92ms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:12:51.282788 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.282766 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49d854e8-50eb-4747-9c43-29f5fcdbc955-istio-podinfo" pod "49d854e8-50eb-4747-9c43-29f5fcdbc955" (UID: "49d854e8-50eb-4747-9c43-29f5fcdbc955"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:12:51.325860 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.325827 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" Oct 30 19:12:51.328546 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.328268 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" Oct 30 19:12:51.332877 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.332848 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" Oct 30 19:12:51.381146 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.381112 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1e466ee-ec7a-4640-bbc5-2f16795107cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1e466ee-ec7a-4640-bbc5-2f16795107cf-credential-socket\") pod \"e1e466ee-ec7a-4640-bbc5-2f16795107cf\" (UID: \"e1e466ee-ec7a-4640-bbc5-2f16795107cf\") " Oct 30 19:12:51.381344 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.381178 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1e466ee-ec7a-4640-bbc5-2f16795107cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1e466ee-ec7a-4640-bbc5-2f16795107cf-workload-certs\") pod \"e1e466ee-ec7a-4640-bbc5-2f16795107cf\" (UID: \"e1e466ee-ec7a-4640-bbc5-2f16795107cf\") " Oct 30 19:12:51.381344 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.381236 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1e466ee-ec7a-4640-bbc5-2f16795107cf-istio-token\" (UniqueName: \"kubernetes.io/projected/e1e466ee-ec7a-4640-bbc5-2f16795107cf-istio-token\") pod \"e1e466ee-ec7a-4640-bbc5-2f16795107cf\" (UID: \"e1e466ee-ec7a-4640-bbc5-2f16795107cf\") " Oct 30 19:12:51.381344 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.381284 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e03523d4-1b6f-4a80-9006-26b0345d809d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e03523d4-1b6f-4a80-9006-26b0345d809d-workload-certs\") pod \"e03523d4-1b6f-4a80-9006-26b0345d809d\" (UID: \"e03523d4-1b6f-4a80-9006-26b0345d809d\") " Oct 30 19:12:51.381344 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.381312 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/921aaf3a-c623-4688-8c39-601e42b9d567-kube-api-access-zxdcc\" (UniqueName: \"kubernetes.io/projected/921aaf3a-c623-4688-8c39-601e42b9d567-kube-api-access-zxdcc\") pod \"921aaf3a-c623-4688-8c39-601e42b9d567\" (UID: \"921aaf3a-c623-4688-8c39-601e42b9d567\") " Oct 30 19:12:51.381559 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.381350 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/921aaf3a-c623-4688-8c39-601e42b9d567-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/921aaf3a-c623-4688-8c39-601e42b9d567-workload-socket\") pod \"921aaf3a-c623-4688-8c39-601e42b9d567\" (UID: \"921aaf3a-c623-4688-8c39-601e42b9d567\") " Oct 30 19:12:51.381559 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.381463 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1e466ee-ec7a-4640-bbc5-2f16795107cf-credential-socket" pod "e1e466ee-ec7a-4640-bbc5-2f16795107cf" (UID: "e1e466ee-ec7a-4640-bbc5-2f16795107cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:12:51.381559 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.381471 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/921aaf3a-c623-4688-8c39-601e42b9d567-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/921aaf3a-c623-4688-8c39-601e42b9d567-istio-podinfo\") pod \"921aaf3a-c623-4688-8c39-601e42b9d567\" (UID: \"921aaf3a-c623-4688-8c39-601e42b9d567\") " Oct 30 19:12:51.381559 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.381512 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/921aaf3a-c623-4688-8c39-601e42b9d567-istio-token\" (UniqueName: \"kubernetes.io/projected/921aaf3a-c623-4688-8c39-601e42b9d567-istio-token\") pod \"921aaf3a-c623-4688-8c39-601e42b9d567\" (UID: \"921aaf3a-c623-4688-8c39-601e42b9d567\") " Oct 30 19:12:51.381559 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.381504 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1e466ee-ec7a-4640-bbc5-2f16795107cf-workload-certs" pod "e1e466ee-ec7a-4640-bbc5-2f16795107cf" (UID: "e1e466ee-ec7a-4640-bbc5-2f16795107cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:12:51.381559 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.381550 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e03523d4-1b6f-4a80-9006-26b0345d809d-workload-certs" pod "e03523d4-1b6f-4a80-9006-26b0345d809d" (UID: "e03523d4-1b6f-4a80-9006-26b0345d809d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:12:51.381899 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.381688 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e03523d4-1b6f-4a80-9006-26b0345d809d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e03523d4-1b6f-4a80-9006-26b0345d809d-istio-data\") pod \"e03523d4-1b6f-4a80-9006-26b0345d809d\" (UID: \"e03523d4-1b6f-4a80-9006-26b0345d809d\") " Oct 30 19:12:51.381899 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.381735 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1e466ee-ec7a-4640-bbc5-2f16795107cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1e466ee-ec7a-4640-bbc5-2f16795107cf-istio-data\") pod \"e1e466ee-ec7a-4640-bbc5-2f16795107cf\" (UID: \"e1e466ee-ec7a-4640-bbc5-2f16795107cf\") " Oct 30 19:12:51.381899 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.381771 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/921aaf3a-c623-4688-8c39-601e42b9d567-workload-socket" pod "921aaf3a-c623-4688-8c39-601e42b9d567" (UID: "921aaf3a-c623-4688-8c39-601e42b9d567"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:12:51.381899 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.381775 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1e466ee-ec7a-4640-bbc5-2f16795107cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1e466ee-ec7a-4640-bbc5-2f16795107cf-istiod-ca-cert\") pod \"e1e466ee-ec7a-4640-bbc5-2f16795107cf\" (UID: \"e1e466ee-ec7a-4640-bbc5-2f16795107cf\") " Oct 30 19:12:51.381899 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.381863 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1e466ee-ec7a-4640-bbc5-2f16795107cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1e466ee-ec7a-4640-bbc5-2f16795107cf-istio-envoy\") pod \"e1e466ee-ec7a-4640-bbc5-2f16795107cf\" (UID: \"e1e466ee-ec7a-4640-bbc5-2f16795107cf\") " Oct 30 19:12:51.382200 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.381911 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/921aaf3a-c623-4688-8c39-601e42b9d567-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/921aaf3a-c623-4688-8c39-601e42b9d567-workload-certs\") pod \"921aaf3a-c623-4688-8c39-601e42b9d567\" (UID: \"921aaf3a-c623-4688-8c39-601e42b9d567\") " Oct 30 19:12:51.382200 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.381964 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e03523d4-1b6f-4a80-9006-26b0345d809d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e03523d4-1b6f-4a80-9006-26b0345d809d-credential-socket\") pod \"e03523d4-1b6f-4a80-9006-26b0345d809d\" (UID: \"e03523d4-1b6f-4a80-9006-26b0345d809d\") " Oct 30 19:12:51.382200 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.381993 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/921aaf3a-c623-4688-8c39-601e42b9d567-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/921aaf3a-c623-4688-8c39-601e42b9d567-credential-socket\") pod \"921aaf3a-c623-4688-8c39-601e42b9d567\" (UID: \"921aaf3a-c623-4688-8c39-601e42b9d567\") " Oct 30 19:12:51.382200 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.382054 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/921aaf3a-c623-4688-8c39-601e42b9d567-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/921aaf3a-c623-4688-8c39-601e42b9d567-istio-data\") pod \"921aaf3a-c623-4688-8c39-601e42b9d567\" (UID: \"921aaf3a-c623-4688-8c39-601e42b9d567\") " Oct 30 19:12:51.382200 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.382091 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1e466ee-ec7a-4640-bbc5-2f16795107cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1e466ee-ec7a-4640-bbc5-2f16795107cf-istio-podinfo\") pod \"e1e466ee-ec7a-4640-bbc5-2f16795107cf\" (UID: \"e1e466ee-ec7a-4640-bbc5-2f16795107cf\") " Oct 30 19:12:51.382549 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.382524 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/921aaf3a-c623-4688-8c39-601e42b9d567-credential-socket" pod "921aaf3a-c623-4688-8c39-601e42b9d567" (UID: "921aaf3a-c623-4688-8c39-601e42b9d567"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:12:51.382635 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.382618 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e03523d4-1b6f-4a80-9006-26b0345d809d-istio-data" pod "e03523d4-1b6f-4a80-9006-26b0345d809d" (UID: "e03523d4-1b6f-4a80-9006-26b0345d809d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:12:51.382699 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.382642 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/921aaf3a-c623-4688-8c39-601e42b9d567-istio-data" pod "921aaf3a-c623-4688-8c39-601e42b9d567" (UID: "921aaf3a-c623-4688-8c39-601e42b9d567"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:12:51.382909 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.382882 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1e466ee-ec7a-4640-bbc5-2f16795107cf-istio-data" pod "e1e466ee-ec7a-4640-bbc5-2f16795107cf" (UID: "e1e466ee-ec7a-4640-bbc5-2f16795107cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:12:51.383020 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.382999 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1e466ee-ec7a-4640-bbc5-2f16795107cf-istiod-ca-cert" pod "e1e466ee-ec7a-4640-bbc5-2f16795107cf" (UID: "e1e466ee-ec7a-4640-bbc5-2f16795107cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:12:51.383020 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.383011 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/921aaf3a-c623-4688-8c39-601e42b9d567-workload-certs" pod "921aaf3a-c623-4688-8c39-601e42b9d567" (UID: "921aaf3a-c623-4688-8c39-601e42b9d567"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:12:51.383206 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.383066 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e03523d4-1b6f-4a80-9006-26b0345d809d-credential-socket" pod "e03523d4-1b6f-4a80-9006-26b0345d809d" (UID: "e03523d4-1b6f-4a80-9006-26b0345d809d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:12:51.383206 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.382128 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e03523d4-1b6f-4a80-9006-26b0345d809d-kube-api-access-ckdf6\" (UniqueName: \"kubernetes.io/projected/e03523d4-1b6f-4a80-9006-26b0345d809d-kube-api-access-ckdf6\") pod \"e03523d4-1b6f-4a80-9006-26b0345d809d\" (UID: \"e03523d4-1b6f-4a80-9006-26b0345d809d\") " Oct 30 19:12:51.383206 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.383183 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1e466ee-ec7a-4640-bbc5-2f16795107cf-kube-api-access-gs4sd\" (UniqueName: \"kubernetes.io/projected/e1e466ee-ec7a-4640-bbc5-2f16795107cf-kube-api-access-gs4sd\") pod \"e1e466ee-ec7a-4640-bbc5-2f16795107cf\" (UID: \"e1e466ee-ec7a-4640-bbc5-2f16795107cf\") " Oct 30 19:12:51.383386 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.383220 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e03523d4-1b6f-4a80-9006-26b0345d809d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e03523d4-1b6f-4a80-9006-26b0345d809d-istiod-ca-cert\") pod \"e03523d4-1b6f-4a80-9006-26b0345d809d\" (UID: \"e03523d4-1b6f-4a80-9006-26b0345d809d\") " Oct 30 19:12:51.383386 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.383258 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/921aaf3a-c623-4688-8c39-601e42b9d567-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/921aaf3a-c623-4688-8c39-601e42b9d567-istio-envoy\") pod \"921aaf3a-c623-4688-8c39-601e42b9d567\" (UID: \"921aaf3a-c623-4688-8c39-601e42b9d567\") " Oct 30 19:12:51.383386 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.383322 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e03523d4-1b6f-4a80-9006-26b0345d809d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e03523d4-1b6f-4a80-9006-26b0345d809d-istio-podinfo\") pod \"e03523d4-1b6f-4a80-9006-26b0345d809d\" (UID: \"e03523d4-1b6f-4a80-9006-26b0345d809d\") " Oct 30 19:12:51.383386 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.383353 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e03523d4-1b6f-4a80-9006-26b0345d809d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e03523d4-1b6f-4a80-9006-26b0345d809d-istio-envoy\") pod \"e03523d4-1b6f-4a80-9006-26b0345d809d\" (UID: \"e03523d4-1b6f-4a80-9006-26b0345d809d\") " Oct 30 19:12:51.383612 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.383462 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e03523d4-1b6f-4a80-9006-26b0345d809d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e03523d4-1b6f-4a80-9006-26b0345d809d-workload-socket\") pod \"e03523d4-1b6f-4a80-9006-26b0345d809d\" (UID: \"e03523d4-1b6f-4a80-9006-26b0345d809d\") " Oct 30 19:12:51.383612 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.383488 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1e466ee-ec7a-4640-bbc5-2f16795107cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1e466ee-ec7a-4640-bbc5-2f16795107cf-workload-socket\") pod \"e1e466ee-ec7a-4640-bbc5-2f16795107cf\" (UID: \"e1e466ee-ec7a-4640-bbc5-2f16795107cf\") " Oct 30 19:12:51.383612 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.383518 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/921aaf3a-c623-4688-8c39-601e42b9d567-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/921aaf3a-c623-4688-8c39-601e42b9d567-istiod-ca-cert\") pod \"921aaf3a-c623-4688-8c39-601e42b9d567\" (UID: \"921aaf3a-c623-4688-8c39-601e42b9d567\") " Oct 30 19:12:51.383612 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.383554 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e03523d4-1b6f-4a80-9006-26b0345d809d-istio-token\" (UniqueName: \"kubernetes.io/projected/e03523d4-1b6f-4a80-9006-26b0345d809d-istio-token\") pod \"e03523d4-1b6f-4a80-9006-26b0345d809d\" (UID: \"e03523d4-1b6f-4a80-9006-26b0345d809d\") " Oct 30 19:12:51.386081 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.384746 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e03523d4-1b6f-4a80-9006-26b0345d809d-workload-socket" pod "e03523d4-1b6f-4a80-9006-26b0345d809d" (UID: "e03523d4-1b6f-4a80-9006-26b0345d809d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:12:51.386081 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.384810 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e03523d4-1b6f-4a80-9006-26b0345d809d-istiod-ca-cert" pod "e03523d4-1b6f-4a80-9006-26b0345d809d" (UID: "e03523d4-1b6f-4a80-9006-26b0345d809d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:12:51.386081 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.385293 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/921aaf3a-c623-4688-8c39-601e42b9d567-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:12:51.386081 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.385318 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e03523d4-1b6f-4a80-9006-26b0345d809d-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:12:51.386081 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.385339 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/921aaf3a-c623-4688-8c39-601e42b9d567-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:12:51.386081 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.385355 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h92ms\" (UniqueName: \"kubernetes.io/projected/49d854e8-50eb-4747-9c43-29f5fcdbc955-kube-api-access-h92ms\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:12:51.386081 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.385371 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/921aaf3a-c623-4688-8c39-601e42b9d567-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:12:51.386081 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.385386 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49d854e8-50eb-4747-9c43-29f5fcdbc955-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:12:51.386081 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.385400 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49d854e8-50eb-4747-9c43-29f5fcdbc955-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:12:51.386081 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.385414 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1e466ee-ec7a-4640-bbc5-2f16795107cf-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:12:51.386081 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.385427 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1e466ee-ec7a-4640-bbc5-2f16795107cf-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:12:51.386081 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.385442 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e03523d4-1b6f-4a80-9006-26b0345d809d-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:12:51.386081 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.385457 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/921aaf3a-c623-4688-8c39-601e42b9d567-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:12:51.386081 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.385472 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49d854e8-50eb-4747-9c43-29f5fcdbc955-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:12:51.386081 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.385487 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e03523d4-1b6f-4a80-9006-26b0345d809d-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:12:51.386081 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.385501 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1e466ee-ec7a-4640-bbc5-2f16795107cf-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:12:51.386081 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.385516 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1e466ee-ec7a-4640-bbc5-2f16795107cf-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:12:51.386081 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.385561 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1e466ee-ec7a-4640-bbc5-2f16795107cf-workload-socket" pod "e1e466ee-ec7a-4640-bbc5-2f16795107cf" (UID: "e1e466ee-ec7a-4640-bbc5-2f16795107cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:12:51.386081 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.385849 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/921aaf3a-c623-4688-8c39-601e42b9d567-istiod-ca-cert" pod "921aaf3a-c623-4688-8c39-601e42b9d567" (UID: "921aaf3a-c623-4688-8c39-601e42b9d567"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:12:51.387780 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.386262 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/921aaf3a-c623-4688-8c39-601e42b9d567-kube-api-access-zxdcc" pod "921aaf3a-c623-4688-8c39-601e42b9d567" (UID: "921aaf3a-c623-4688-8c39-601e42b9d567"). InnerVolumeSpecName "kube-api-access-zxdcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:12:51.387780 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.386338 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/921aaf3a-c623-4688-8c39-601e42b9d567-istio-token" pod "921aaf3a-c623-4688-8c39-601e42b9d567" (UID: "921aaf3a-c623-4688-8c39-601e42b9d567"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:12:51.388289 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.388251 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/921aaf3a-c623-4688-8c39-601e42b9d567-istio-podinfo" pod "921aaf3a-c623-4688-8c39-601e42b9d567" (UID: "921aaf3a-c623-4688-8c39-601e42b9d567"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:12:51.389249 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.388766 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1e466ee-ec7a-4640-bbc5-2f16795107cf-istio-podinfo" pod "e1e466ee-ec7a-4640-bbc5-2f16795107cf" (UID: "e1e466ee-ec7a-4640-bbc5-2f16795107cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:12:51.390701 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.390591 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e03523d4-1b6f-4a80-9006-26b0345d809d-kube-api-access-ckdf6" pod "e03523d4-1b6f-4a80-9006-26b0345d809d" (UID: "e03523d4-1b6f-4a80-9006-26b0345d809d"). InnerVolumeSpecName "kube-api-access-ckdf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:12:51.390701 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.390667 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/921aaf3a-c623-4688-8c39-601e42b9d567-istio-envoy" pod "921aaf3a-c623-4688-8c39-601e42b9d567" (UID: "921aaf3a-c623-4688-8c39-601e42b9d567"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:12:51.391019 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.390979 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1e466ee-ec7a-4640-bbc5-2f16795107cf-istio-token" pod "e1e466ee-ec7a-4640-bbc5-2f16795107cf" (UID: "e1e466ee-ec7a-4640-bbc5-2f16795107cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:12:51.391405 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.391381 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e03523d4-1b6f-4a80-9006-26b0345d809d-istio-envoy" pod "e03523d4-1b6f-4a80-9006-26b0345d809d" (UID: "e03523d4-1b6f-4a80-9006-26b0345d809d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:12:51.391405 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.391391 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e03523d4-1b6f-4a80-9006-26b0345d809d-istio-podinfo" pod "e03523d4-1b6f-4a80-9006-26b0345d809d" (UID: "e03523d4-1b6f-4a80-9006-26b0345d809d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:12:51.391540 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.391458 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1e466ee-ec7a-4640-bbc5-2f16795107cf-istio-envoy" pod "e1e466ee-ec7a-4640-bbc5-2f16795107cf" (UID: "e1e466ee-ec7a-4640-bbc5-2f16795107cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:12:51.391579 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.391540 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e03523d4-1b6f-4a80-9006-26b0345d809d-istio-token" pod "e03523d4-1b6f-4a80-9006-26b0345d809d" (UID: "e03523d4-1b6f-4a80-9006-26b0345d809d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:12:51.393102 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.393084 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1e466ee-ec7a-4640-bbc5-2f16795107cf-kube-api-access-gs4sd" pod "e1e466ee-ec7a-4640-bbc5-2f16795107cf" (UID: "e1e466ee-ec7a-4640-bbc5-2f16795107cf"). InnerVolumeSpecName "kube-api-access-gs4sd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:12:51.486421 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.486385 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1e466ee-ec7a-4640-bbc5-2f16795107cf-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:12:51.486421 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.486420 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckdf6\" (UniqueName: \"kubernetes.io/projected/e03523d4-1b6f-4a80-9006-26b0345d809d-kube-api-access-ckdf6\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:12:51.486421 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.486432 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs4sd\" (UniqueName: \"kubernetes.io/projected/e1e466ee-ec7a-4640-bbc5-2f16795107cf-kube-api-access-gs4sd\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:12:51.486767 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.486441 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e03523d4-1b6f-4a80-9006-26b0345d809d-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:12:51.486767 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.486451 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/921aaf3a-c623-4688-8c39-601e42b9d567-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:12:51.486767 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.486459 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e03523d4-1b6f-4a80-9006-26b0345d809d-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:12:51.486767 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.486468 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e03523d4-1b6f-4a80-9006-26b0345d809d-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:12:51.486767 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.486483 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e03523d4-1b6f-4a80-9006-26b0345d809d-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:12:51.486767 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.486496 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1e466ee-ec7a-4640-bbc5-2f16795107cf-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:12:51.486767 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.486504 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/921aaf3a-c623-4688-8c39-601e42b9d567-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:12:51.486767 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.486513 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e03523d4-1b6f-4a80-9006-26b0345d809d-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:12:51.486767 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.486522 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1e466ee-ec7a-4640-bbc5-2f16795107cf-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:12:51.486767 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.486534 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxdcc\" (UniqueName: \"kubernetes.io/projected/921aaf3a-c623-4688-8c39-601e42b9d567-kube-api-access-zxdcc\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:12:51.486767 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.486544 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/921aaf3a-c623-4688-8c39-601e42b9d567-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:12:51.486767 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.486557 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/921aaf3a-c623-4688-8c39-601e42b9d567-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:12:51.486767 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.486569 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1e466ee-ec7a-4640-bbc5-2f16795107cf-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:12:51.693822 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.693729 4442 generic.go:356] "Generic (PLEG): container finished" podID="e1e466ee-ec7a-4640-bbc5-2f16795107cf" containerID="57d653004220b68a80462e3bc6e347e90dd37acae6e8a362c5591a364cfe74fa" exitCode=0 Oct 30 19:12:51.693822 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.693782 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" Oct 30 19:12:51.693822 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.693812 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" event={"ID":"e1e466ee-ec7a-4640-bbc5-2f16795107cf","Type":"ContainerDied","Data":"57d653004220b68a80462e3bc6e347e90dd37acae6e8a362c5591a364cfe74fa"} Oct 30 19:12:51.694111 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.693861 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" event={"ID":"e1e466ee-ec7a-4640-bbc5-2f16795107cf","Type":"ContainerDied","Data":"9b452ed1ad564eeeb4d719e925cc2471b196a27c48bfb46a27b0cebd4ccf72d9"} Oct 30 19:12:51.694111 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.693890 4442 scope.go:122] "RemoveContainer" containerID="57d653004220b68a80462e3bc6e347e90dd37acae6e8a362c5591a364cfe74fa" Oct 30 19:12:51.695680 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.695650 4442 generic.go:356] "Generic (PLEG): container finished" podID="921aaf3a-c623-4688-8c39-601e42b9d567" containerID="2943958ae2bc7e045eec02109beb2100116eb274e165f40726bdc0dac7c787c8" exitCode=0 Oct 30 19:12:51.695968 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.695711 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" Oct 30 19:12:51.695968 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.695729 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" event={"ID":"921aaf3a-c623-4688-8c39-601e42b9d567","Type":"ContainerDied","Data":"2943958ae2bc7e045eec02109beb2100116eb274e165f40726bdc0dac7c787c8"} Oct 30 19:12:51.695968 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.695772 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s" event={"ID":"921aaf3a-c623-4688-8c39-601e42b9d567","Type":"ContainerDied","Data":"7f92ab7fc67a7c5455d2eedbb808564df45050c6acd6d093c64eece3d4488036"} Oct 30 19:12:51.697996 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.697972 4442 generic.go:356] "Generic (PLEG): container finished" podID="e03523d4-1b6f-4a80-9006-26b0345d809d" containerID="535cf080e35833fb2fd74c327e7e095053c76ae3ecfb740a5517847d4f0e60f9" exitCode=0 Oct 30 19:12:51.698193 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.698030 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" Oct 30 19:12:51.698193 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.698050 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" event={"ID":"e03523d4-1b6f-4a80-9006-26b0345d809d","Type":"ContainerDied","Data":"535cf080e35833fb2fd74c327e7e095053c76ae3ecfb740a5517847d4f0e60f9"} Oct 30 19:12:51.698193 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.698086 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2" event={"ID":"e03523d4-1b6f-4a80-9006-26b0345d809d","Type":"ContainerDied","Data":"ba3c9a9a0977439c5e6c9338c7011c57f2571ae0ab7ac0670bbd13eeb06fad67"} Oct 30 19:12:51.699772 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.699749 4442 generic.go:356] "Generic (PLEG): container finished" podID="49d854e8-50eb-4747-9c43-29f5fcdbc955" containerID="05f1580b0d3e567e290b24af7fb44afb12c172e7868e2d5d7ee392cea5de669a" exitCode=0 Oct 30 19:12:51.699903 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.699790 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" Oct 30 19:12:51.699903 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.699838 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" event={"ID":"49d854e8-50eb-4747-9c43-29f5fcdbc955","Type":"ContainerDied","Data":"05f1580b0d3e567e290b24af7fb44afb12c172e7868e2d5d7ee392cea5de669a"} Oct 30 19:12:51.699903 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.699892 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb" event={"ID":"49d854e8-50eb-4747-9c43-29f5fcdbc955","Type":"ContainerDied","Data":"0cc5bd37e8d0b9954b0dfe1f6de559128c3b763cf99d9ea2a7dc09f2e0b5edc1"} Oct 30 19:12:51.707834 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.707678 4442 scope.go:122] "RemoveContainer" containerID="57d653004220b68a80462e3bc6e347e90dd37acae6e8a362c5591a364cfe74fa" Oct 30 19:12:51.708120 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:12:51.708098 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57d653004220b68a80462e3bc6e347e90dd37acae6e8a362c5591a364cfe74fa\": container with ID starting with 57d653004220b68a80462e3bc6e347e90dd37acae6e8a362c5591a364cfe74fa not found: ID does not exist" containerID="57d653004220b68a80462e3bc6e347e90dd37acae6e8a362c5591a364cfe74fa" Oct 30 19:12:51.708252 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.708127 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57d653004220b68a80462e3bc6e347e90dd37acae6e8a362c5591a364cfe74fa"} err="failed to get container status \"57d653004220b68a80462e3bc6e347e90dd37acae6e8a362c5591a364cfe74fa\": rpc error: code = NotFound desc = could not find container \"57d653004220b68a80462e3bc6e347e90dd37acae6e8a362c5591a364cfe74fa\": container with ID starting with 57d653004220b68a80462e3bc6e347e90dd37acae6e8a362c5591a364cfe74fa not found: ID does not exist" Oct 30 19:12:51.708252 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.708186 4442 scope.go:122] "RemoveContainer" containerID="2943958ae2bc7e045eec02109beb2100116eb274e165f40726bdc0dac7c787c8" Oct 30 19:12:51.719636 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.719619 4442 scope.go:122] "RemoveContainer" containerID="2943958ae2bc7e045eec02109beb2100116eb274e165f40726bdc0dac7c787c8" Oct 30 19:12:51.719942 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:12:51.719917 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2943958ae2bc7e045eec02109beb2100116eb274e165f40726bdc0dac7c787c8\": container with ID starting with 2943958ae2bc7e045eec02109beb2100116eb274e165f40726bdc0dac7c787c8 not found: ID does not exist" containerID="2943958ae2bc7e045eec02109beb2100116eb274e165f40726bdc0dac7c787c8" Oct 30 19:12:51.720022 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.719947 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2943958ae2bc7e045eec02109beb2100116eb274e165f40726bdc0dac7c787c8"} err="failed to get container status \"2943958ae2bc7e045eec02109beb2100116eb274e165f40726bdc0dac7c787c8\": rpc error: code = NotFound desc = could not find container \"2943958ae2bc7e045eec02109beb2100116eb274e165f40726bdc0dac7c787c8\": container with ID starting with 2943958ae2bc7e045eec02109beb2100116eb274e165f40726bdc0dac7c787c8 not found: ID does not exist" Oct 30 19:12:51.720022 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.719964 4442 scope.go:122] "RemoveContainer" containerID="535cf080e35833fb2fd74c327e7e095053c76ae3ecfb740a5517847d4f0e60f9" Oct 30 19:12:51.731934 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.731915 4442 scope.go:122] "RemoveContainer" containerID="f32a9e2f167f134d719f6dc9b1f94211c22dbd5921034d7500cccb171b8b3321" Oct 30 19:12:51.741908 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.741887 4442 scope.go:122] "RemoveContainer" containerID="535cf080e35833fb2fd74c327e7e095053c76ae3ecfb740a5517847d4f0e60f9" Oct 30 19:12:51.742258 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:12:51.742225 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"535cf080e35833fb2fd74c327e7e095053c76ae3ecfb740a5517847d4f0e60f9\": container with ID starting with 535cf080e35833fb2fd74c327e7e095053c76ae3ecfb740a5517847d4f0e60f9 not found: ID does not exist" containerID="535cf080e35833fb2fd74c327e7e095053c76ae3ecfb740a5517847d4f0e60f9" Oct 30 19:12:51.742350 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.742269 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"535cf080e35833fb2fd74c327e7e095053c76ae3ecfb740a5517847d4f0e60f9"} err="failed to get container status \"535cf080e35833fb2fd74c327e7e095053c76ae3ecfb740a5517847d4f0e60f9\": rpc error: code = NotFound desc = could not find container \"535cf080e35833fb2fd74c327e7e095053c76ae3ecfb740a5517847d4f0e60f9\": container with ID starting with 535cf080e35833fb2fd74c327e7e095053c76ae3ecfb740a5517847d4f0e60f9 not found: ID does not exist" Oct 30 19:12:51.742350 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.742288 4442 scope.go:122] "RemoveContainer" containerID="f32a9e2f167f134d719f6dc9b1f94211c22dbd5921034d7500cccb171b8b3321" Oct 30 19:12:51.742598 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:12:51.742578 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f32a9e2f167f134d719f6dc9b1f94211c22dbd5921034d7500cccb171b8b3321\": container with ID starting with f32a9e2f167f134d719f6dc9b1f94211c22dbd5921034d7500cccb171b8b3321 not found: ID does not exist" containerID="f32a9e2f167f134d719f6dc9b1f94211c22dbd5921034d7500cccb171b8b3321" Oct 30 19:12:51.742708 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.742603 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f32a9e2f167f134d719f6dc9b1f94211c22dbd5921034d7500cccb171b8b3321"} err="failed to get container status \"f32a9e2f167f134d719f6dc9b1f94211c22dbd5921034d7500cccb171b8b3321\": rpc error: code = NotFound desc = could not find container \"f32a9e2f167f134d719f6dc9b1f94211c22dbd5921034d7500cccb171b8b3321\": container with ID starting with f32a9e2f167f134d719f6dc9b1f94211c22dbd5921034d7500cccb171b8b3321 not found: ID does not exist" Oct 30 19:12:51.742708 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.742622 4442 scope.go:122] "RemoveContainer" containerID="05f1580b0d3e567e290b24af7fb44afb12c172e7868e2d5d7ee392cea5de669a" Oct 30 19:12:51.753086 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.753064 4442 scope.go:122] "RemoveContainer" containerID="05f1580b0d3e567e290b24af7fb44afb12c172e7868e2d5d7ee392cea5de669a" Oct 30 19:12:51.753398 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:12:51.753379 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05f1580b0d3e567e290b24af7fb44afb12c172e7868e2d5d7ee392cea5de669a\": container with ID starting with 05f1580b0d3e567e290b24af7fb44afb12c172e7868e2d5d7ee392cea5de669a not found: ID does not exist" containerID="05f1580b0d3e567e290b24af7fb44afb12c172e7868e2d5d7ee392cea5de669a" Oct 30 19:12:51.753482 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.753415 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05f1580b0d3e567e290b24af7fb44afb12c172e7868e2d5d7ee392cea5de669a"} err="failed to get container status \"05f1580b0d3e567e290b24af7fb44afb12c172e7868e2d5d7ee392cea5de669a\": rpc error: code = NotFound desc = could not find container \"05f1580b0d3e567e290b24af7fb44afb12c172e7868e2d5d7ee392cea5de669a\": container with ID starting with 05f1580b0d3e567e290b24af7fb44afb12c172e7868e2d5d7ee392cea5de669a not found: ID does not exist" Oct 30 19:12:51.789455 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.789426 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb"] Oct 30 19:12:51.801078 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.801050 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkggb"] Oct 30 19:12:51.827777 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.827735 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9"] Oct 30 19:12:51.833890 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.833862 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9"] Oct 30 19:12:51.863538 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.863511 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2"] Oct 30 19:12:51.867521 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.867495 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-x2rr2"] Oct 30 19:12:51.901556 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.901526 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s"] Oct 30 19:12:51.905467 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:51.905444 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-qwf7s"] Oct 30 19:12:52.183028 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:52.182997 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-z2dn9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:12:52.183445 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:52.183060 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-z2dn9" podUID="e1e466ee-ec7a-4640-bbc5-2f16795107cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:12:53.067817 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:53.067783 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49d854e8-50eb-4747-9c43-29f5fcdbc955" path="/var/lib/kubelet/pods/49d854e8-50eb-4747-9c43-29f5fcdbc955/volumes" Oct 30 19:12:53.068570 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:53.068547 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="921aaf3a-c623-4688-8c39-601e42b9d567" path="/var/lib/kubelet/pods/921aaf3a-c623-4688-8c39-601e42b9d567/volumes" Oct 30 19:12:53.069164 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:53.069117 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e03523d4-1b6f-4a80-9006-26b0345d809d" path="/var/lib/kubelet/pods/e03523d4-1b6f-4a80-9006-26b0345d809d/volumes" Oct 30 19:12:53.070201 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:53.070179 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1e466ee-ec7a-4640-bbc5-2f16795107cf" path="/var/lib/kubelet/pods/e1e466ee-ec7a-4640-bbc5-2f16795107cf/volumes" Oct 30 19:12:59.714868 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:59.714837 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-1c4bcd-55d686745d-4hhzp" Oct 30 19:12:59.763286 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:59.763259 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1c4bcd-76ffbffdcf-q4ttx" Oct 30 19:12:59.807782 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:59.807743 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1e71600-36ee-4faa-872d-da1203e65bdc-kube-api-access-98ldz\" (UniqueName: \"kubernetes.io/projected/f1e71600-36ee-4faa-872d-da1203e65bdc-kube-api-access-98ldz\") pod \"f1e71600-36ee-4faa-872d-da1203e65bdc\" (UID: \"f1e71600-36ee-4faa-872d-da1203e65bdc\") " Oct 30 19:12:59.807995 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:59.807856 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1e71600-36ee-4faa-872d-da1203e65bdc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1e71600-36ee-4faa-872d-da1203e65bdc-plugins-config-volume\") pod \"f1e71600-36ee-4faa-872d-da1203e65bdc\" (UID: \"f1e71600-36ee-4faa-872d-da1203e65bdc\") " Oct 30 19:12:59.807995 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:59.807905 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/034e375c-3ae0-40db-94ed-9d328fee5639-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/034e375c-3ae0-40db-94ed-9d328fee5639-plugins-config-volume\") pod \"034e375c-3ae0-40db-94ed-9d328fee5639\" (UID: \"034e375c-3ae0-40db-94ed-9d328fee5639\") " Oct 30 19:12:59.807995 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:59.807963 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/034e375c-3ae0-40db-94ed-9d328fee5639-kube-api-access-74qnd\" (UniqueName: \"kubernetes.io/projected/034e375c-3ae0-40db-94ed-9d328fee5639-kube-api-access-74qnd\") pod \"034e375c-3ae0-40db-94ed-9d328fee5639\" (UID: \"034e375c-3ae0-40db-94ed-9d328fee5639\") " Oct 30 19:12:59.809346 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:59.809155 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1e71600-36ee-4faa-872d-da1203e65bdc-plugins-config-volume" pod "f1e71600-36ee-4faa-872d-da1203e65bdc" (UID: "f1e71600-36ee-4faa-872d-da1203e65bdc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:12:59.811957 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:59.810478 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/034e375c-3ae0-40db-94ed-9d328fee5639-plugins-config-volume" pod "034e375c-3ae0-40db-94ed-9d328fee5639" (UID: "034e375c-3ae0-40db-94ed-9d328fee5639"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:12:59.813029 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:59.812907 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/034e375c-3ae0-40db-94ed-9d328fee5639-kube-api-access-74qnd" pod "034e375c-3ae0-40db-94ed-9d328fee5639" (UID: "034e375c-3ae0-40db-94ed-9d328fee5639"). InnerVolumeSpecName "kube-api-access-74qnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:12:59.813029 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:59.812964 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1e71600-36ee-4faa-872d-da1203e65bdc-kube-api-access-98ldz" pod "f1e71600-36ee-4faa-872d-da1203e65bdc" (UID: "f1e71600-36ee-4faa-872d-da1203e65bdc"). InnerVolumeSpecName "kube-api-access-98ldz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:12:59.816606 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:59.816572 4442 generic.go:356] "Generic (PLEG): container finished" podID="034e375c-3ae0-40db-94ed-9d328fee5639" containerID="3cc92cf6008580c1750453be5efb7bb823500363f66fa6ef47c52b78a1c51661" exitCode=0 Oct 30 19:12:59.816767 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:59.816752 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-1c4bcd-55d686745d-4hhzp" Oct 30 19:12:59.816833 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:59.816771 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-1c4bcd-55d686745d-4hhzp" event={"ID":"034e375c-3ae0-40db-94ed-9d328fee5639","Type":"ContainerDied","Data":"3cc92cf6008580c1750453be5efb7bb823500363f66fa6ef47c52b78a1c51661"} Oct 30 19:12:59.816833 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:59.816826 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-1c4bcd-55d686745d-4hhzp" event={"ID":"034e375c-3ae0-40db-94ed-9d328fee5639","Type":"ContainerDied","Data":"da40dbccb7cce3a415514f2bc22392d9173e45b35737679528ab43dc95018598"} Oct 30 19:12:59.816937 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:59.816849 4442 scope.go:122] "RemoveContainer" containerID="3cc92cf6008580c1750453be5efb7bb823500363f66fa6ef47c52b78a1c51661" Oct 30 19:12:59.819047 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:59.819020 4442 generic.go:356] "Generic (PLEG): container finished" podID="f1e71600-36ee-4faa-872d-da1203e65bdc" containerID="5ed1af2f9609f59c1d498bd70e609d75a638fccd5cf704814b38adda385a986c" exitCode=0 Oct 30 19:12:59.819322 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:59.819294 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1c4bcd-76ffbffdcf-q4ttx" event={"ID":"f1e71600-36ee-4faa-872d-da1203e65bdc","Type":"ContainerDied","Data":"5ed1af2f9609f59c1d498bd70e609d75a638fccd5cf704814b38adda385a986c"} Oct 30 19:12:59.819416 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:59.819348 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1c4bcd-76ffbffdcf-q4ttx" event={"ID":"f1e71600-36ee-4faa-872d-da1203e65bdc","Type":"ContainerDied","Data":"9ece9b5ffc8693361d83b83192b7e0cd346e0677d828271cecc339d912aa55b6"} Oct 30 19:12:59.819491 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:59.819456 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1c4bcd-76ffbffdcf-q4ttx" Oct 30 19:12:59.830540 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:59.830513 4442 scope.go:122] "RemoveContainer" containerID="3cc92cf6008580c1750453be5efb7bb823500363f66fa6ef47c52b78a1c51661" Oct 30 19:12:59.831419 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:12:59.831371 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cc92cf6008580c1750453be5efb7bb823500363f66fa6ef47c52b78a1c51661\": container with ID starting with 3cc92cf6008580c1750453be5efb7bb823500363f66fa6ef47c52b78a1c51661 not found: ID does not exist" containerID="3cc92cf6008580c1750453be5efb7bb823500363f66fa6ef47c52b78a1c51661" Oct 30 19:12:59.831544 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:59.831418 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cc92cf6008580c1750453be5efb7bb823500363f66fa6ef47c52b78a1c51661"} err="failed to get container status \"3cc92cf6008580c1750453be5efb7bb823500363f66fa6ef47c52b78a1c51661\": rpc error: code = NotFound desc = could not find container \"3cc92cf6008580c1750453be5efb7bb823500363f66fa6ef47c52b78a1c51661\": container with ID starting with 3cc92cf6008580c1750453be5efb7bb823500363f66fa6ef47c52b78a1c51661 not found: ID does not exist" Oct 30 19:12:59.831544 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:59.831442 4442 scope.go:122] "RemoveContainer" containerID="5ed1af2f9609f59c1d498bd70e609d75a638fccd5cf704814b38adda385a986c" Oct 30 19:12:59.847242 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:59.847223 4442 scope.go:122] "RemoveContainer" containerID="5ed1af2f9609f59c1d498bd70e609d75a638fccd5cf704814b38adda385a986c" Oct 30 19:12:59.847591 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:12:59.847568 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ed1af2f9609f59c1d498bd70e609d75a638fccd5cf704814b38adda385a986c\": container with ID starting with 5ed1af2f9609f59c1d498bd70e609d75a638fccd5cf704814b38adda385a986c not found: ID does not exist" containerID="5ed1af2f9609f59c1d498bd70e609d75a638fccd5cf704814b38adda385a986c" Oct 30 19:12:59.847675 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:59.847598 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ed1af2f9609f59c1d498bd70e609d75a638fccd5cf704814b38adda385a986c"} err="failed to get container status \"5ed1af2f9609f59c1d498bd70e609d75a638fccd5cf704814b38adda385a986c\": rpc error: code = NotFound desc = could not find container \"5ed1af2f9609f59c1d498bd70e609d75a638fccd5cf704814b38adda385a986c\": container with ID starting with 5ed1af2f9609f59c1d498bd70e609d75a638fccd5cf704814b38adda385a986c not found: ID does not exist" Oct 30 19:12:59.882904 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:59.882872 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1c4bcd-76ffbffdcf-q4ttx"] Oct 30 19:12:59.886641 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:59.886613 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1c4bcd-76ffbffdcf-q4ttx"] Oct 30 19:12:59.909395 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:59.909363 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98ldz\" (UniqueName: \"kubernetes.io/projected/f1e71600-36ee-4faa-872d-da1203e65bdc-kube-api-access-98ldz\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:12:59.909395 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:59.909392 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1e71600-36ee-4faa-872d-da1203e65bdc-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:12:59.909597 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:59.909403 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/034e375c-3ae0-40db-94ed-9d328fee5639-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:12:59.909597 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:59.909412 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74qnd\" (UniqueName: \"kubernetes.io/projected/034e375c-3ae0-40db-94ed-9d328fee5639-kube-api-access-74qnd\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:12:59.918270 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:59.918200 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1c4bcd-55d686745d-4hhzp"] Oct 30 19:12:59.922069 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:12:59.922045 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1c4bcd-55d686745d-4hhzp"] Oct 30 19:13:01.066612 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:01.066578 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="034e375c-3ae0-40db-94ed-9d328fee5639" path="/var/lib/kubelet/pods/034e375c-3ae0-40db-94ed-9d328fee5639/volumes" Oct 30 19:13:01.067248 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:01.067229 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1e71600-36ee-4faa-872d-da1203e65bdc" path="/var/lib/kubelet/pods/f1e71600-36ee-4faa-872d-da1203e65bdc/volumes" Oct 30 19:13:03.145320 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:03.145285 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-09ef1d/rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8"] Oct 30 19:13:03.462302 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:03.462201 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d1a330-8zpht"] Oct 30 19:13:03.462872 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:03.462854 4442 cpu_manager.go:414] "containerMap: removing container" podUID="921aaf3a-c623-4688-8c39-601e42b9d567" containerName="istio-proxy" Oct 30 19:13:03.462946 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:03.462873 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="921aaf3a-c623-4688-8c39-601e42b9d567" containerName="istio-proxy" Oct 30 19:13:03.462946 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:03.462890 4442 cpu_manager.go:414] "containerMap: removing container" podUID="f1e71600-36ee-4faa-872d-da1203e65bdc" containerName="payload-pre-processing" Oct 30 19:13:03.462946 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:03.462896 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1e71600-36ee-4faa-872d-da1203e65bdc" containerName="payload-pre-processing" Oct 30 19:13:03.462946 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:03.462905 4442 cpu_manager.go:414] "containerMap: removing container" podUID="e03523d4-1b6f-4a80-9006-26b0345d809d" containerName="istio-proxy" Oct 30 19:13:03.462946 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:03.462910 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e03523d4-1b6f-4a80-9006-26b0345d809d" containerName="istio-proxy" Oct 30 19:13:03.462946 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:03.462918 4442 cpu_manager.go:414] "containerMap: removing container" podUID="49d854e8-50eb-4747-9c43-29f5fcdbc955" containerName="istio-proxy" Oct 30 19:13:03.462946 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:03.462922 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49d854e8-50eb-4747-9c43-29f5fcdbc955" containerName="istio-proxy" Oct 30 19:13:03.462946 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:03.462948 4442 cpu_manager.go:414] "containerMap: removing container" podUID="e1e466ee-ec7a-4640-bbc5-2f16795107cf" containerName="istio-proxy" Oct 30 19:13:03.463432 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:03.462956 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1e466ee-ec7a-4640-bbc5-2f16795107cf" containerName="istio-proxy" Oct 30 19:13:03.463432 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:03.462966 4442 cpu_manager.go:414] "containerMap: removing container" podUID="e03523d4-1b6f-4a80-9006-26b0345d809d" containerName="istio-proxy" Oct 30 19:13:03.463432 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:03.462971 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e03523d4-1b6f-4a80-9006-26b0345d809d" containerName="istio-proxy" Oct 30 19:13:03.463432 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:03.462984 4442 cpu_manager.go:414] "containerMap: removing container" podUID="d2f660b9-2642-4b8a-95ff-c090e33ae02e" containerName="revoke-keys" Oct 30 19:13:03.463432 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:03.462988 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2f660b9-2642-4b8a-95ff-c090e33ae02e" containerName="revoke-keys" Oct 30 19:13:03.463432 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:03.462995 4442 cpu_manager.go:414] "containerMap: removing container" podUID="034e375c-3ae0-40db-94ed-9d328fee5639" containerName="payload-processing" Oct 30 19:13:03.463432 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:03.463000 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="034e375c-3ae0-40db-94ed-9d328fee5639" containerName="payload-processing" Oct 30 19:13:03.463432 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:03.463092 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="e03523d4-1b6f-4a80-9006-26b0345d809d" containerName="istio-proxy" Oct 30 19:13:03.463432 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:03.463102 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2f660b9-2642-4b8a-95ff-c090e33ae02e" containerName="revoke-keys" Oct 30 19:13:03.463432 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:03.463110 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="034e375c-3ae0-40db-94ed-9d328fee5639" containerName="payload-processing" Oct 30 19:13:03.463432 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:03.463118 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="921aaf3a-c623-4688-8c39-601e42b9d567" containerName="istio-proxy" Oct 30 19:13:03.463432 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:03.463124 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1e71600-36ee-4faa-872d-da1203e65bdc" containerName="payload-pre-processing" Oct 30 19:13:03.463432 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:03.463145 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="e03523d4-1b6f-4a80-9006-26b0345d809d" containerName="istio-proxy" Oct 30 19:13:03.463432 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:03.463156 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="49d854e8-50eb-4747-9c43-29f5fcdbc955" containerName="istio-proxy" Oct 30 19:13:03.463432 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:03.463169 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1e466ee-ec7a-4640-bbc5-2f16795107cf" containerName="istio-proxy" Oct 30 19:13:03.467291 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:03.467264 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d1a330-8zpht" Oct 30 19:13:03.490890 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:03.490859 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d1a330-8zpht"] Oct 30 19:13:03.499541 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:03.497639 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:13:03.499541 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:03.497712 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:13:03.583365 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:03.583326 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5224824-05eb-4e81-bbfc-761d8faadd07-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d1a330-8zpht\" (UID: \"f5224824-05eb-4e81-bbfc-761d8faadd07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d1a330-8zpht" Oct 30 19:13:03.583567 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:03.583543 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5224824-05eb-4e81-bbfc-761d8faadd07-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d1a330-8zpht\" (UID: \"f5224824-05eb-4e81-bbfc-761d8faadd07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d1a330-8zpht" Oct 30 19:13:03.583636 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:03.583579 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg9k4\" (UniqueName: \"kubernetes.io/projected/f5224824-05eb-4e81-bbfc-761d8faadd07-kube-api-access-gg9k4\") pod \"maas-api-revoke-keys-e2e-extmod-a-d1a330-8zpht\" (UID: \"f5224824-05eb-4e81-bbfc-761d8faadd07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d1a330-8zpht" Oct 30 19:13:03.685024 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:03.684983 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5224824-05eb-4e81-bbfc-761d8faadd07-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d1a330-8zpht\" (UID: \"f5224824-05eb-4e81-bbfc-761d8faadd07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d1a330-8zpht" Oct 30 19:13:03.685266 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:03.685173 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5224824-05eb-4e81-bbfc-761d8faadd07-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d1a330-8zpht\" (UID: \"f5224824-05eb-4e81-bbfc-761d8faadd07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d1a330-8zpht" Oct 30 19:13:03.685266 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:03.685204 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg9k4\" (UniqueName: \"kubernetes.io/projected/f5224824-05eb-4e81-bbfc-761d8faadd07-kube-api-access-gg9k4\") pod \"maas-api-revoke-keys-e2e-extmod-a-d1a330-8zpht\" (UID: \"f5224824-05eb-4e81-bbfc-761d8faadd07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d1a330-8zpht" Oct 30 19:13:03.685674 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:03.685644 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5224824-05eb-4e81-bbfc-761d8faadd07-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d1a330-8zpht\" (UID: \"f5224824-05eb-4e81-bbfc-761d8faadd07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d1a330-8zpht" Oct 30 19:13:03.687691 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:03.687665 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5224824-05eb-4e81-bbfc-761d8faadd07-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d1a330-8zpht\" (UID: \"f5224824-05eb-4e81-bbfc-761d8faadd07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d1a330-8zpht" Oct 30 19:13:03.711845 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:03.711820 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg9k4\" (UniqueName: \"kubernetes.io/projected/f5224824-05eb-4e81-bbfc-761d8faadd07-kube-api-access-gg9k4\") pod \"maas-api-revoke-keys-e2e-extmod-a-d1a330-8zpht\" (UID: \"f5224824-05eb-4e81-bbfc-761d8faadd07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d1a330-8zpht" Oct 30 19:13:03.783510 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:03.783408 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d1a330-8zpht" Oct 30 19:13:04.113524 ip-10-0-138-176 kubenswrapper[4442]: W1030 19:13:04.113495 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5224824_05eb_4e81_bbfc_761d8faadd07.slice/crio-859e724ad88423183b52e2db4cc278438a6fa673f222e8fc1607e411ac7283cb WatchSource:0}: Error finding container 859e724ad88423183b52e2db4cc278438a6fa673f222e8fc1607e411ac7283cb: Status 404 returned error can't find the container with id 859e724ad88423183b52e2db4cc278438a6fa673f222e8fc1607e411ac7283cb Oct 30 19:13:04.114086 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:04.114064 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d1a330-8zpht"] Oct 30 19:13:04.898536 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:04.898504 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d1a330-8zpht" event={"ID":"f5224824-05eb-4e81-bbfc-761d8faadd07","Type":"ContainerStarted","Data":"0354bf4ada8b0136cbef92a7ae2c94cc38a803573976f929afd2048cbce0020b"} Oct 30 19:13:04.898976 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:04.898555 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d1a330-8zpht" event={"ID":"f5224824-05eb-4e81-bbfc-761d8faadd07","Type":"ContainerStarted","Data":"859e724ad88423183b52e2db4cc278438a6fa673f222e8fc1607e411ac7283cb"} Oct 30 19:13:04.955264 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:04.955198 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d1a330-8zpht" podStartSLOduration=1.955177925 podStartE2EDuration="1.955177925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:13:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:13:04.950477686 +0000 UTC m=+4482.040540037" watchObservedRunningTime="2026-10-30 19:13:04.955177925 +0000 UTC m=+4482.045240265" Oct 30 19:13:05.914722 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:05.914460 4442 generic.go:356] "Generic (PLEG): container finished" podID="f5224824-05eb-4e81-bbfc-761d8faadd07" containerID="0354bf4ada8b0136cbef92a7ae2c94cc38a803573976f929afd2048cbce0020b" exitCode=0 Oct 30 19:13:05.914722 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:05.914609 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d1a330-8zpht" event={"ID":"f5224824-05eb-4e81-bbfc-761d8faadd07","Type":"ContainerDied","Data":"0354bf4ada8b0136cbef92a7ae2c94cc38a803573976f929afd2048cbce0020b"} Oct 30 19:13:07.098437 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:07.098408 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d1a330-8zpht" Oct 30 19:13:07.152524 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:07.152493 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5224824-05eb-4e81-bbfc-761d8faadd07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5224824-05eb-4e81-bbfc-761d8faadd07-maas-api-xks-ca\") pod \"f5224824-05eb-4e81-bbfc-761d8faadd07\" (UID: \"f5224824-05eb-4e81-bbfc-761d8faadd07\") " Oct 30 19:13:07.152717 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:07.152599 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5224824-05eb-4e81-bbfc-761d8faadd07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5224824-05eb-4e81-bbfc-761d8faadd07-maas-api-openshift-ca\") pod \"f5224824-05eb-4e81-bbfc-761d8faadd07\" (UID: \"f5224824-05eb-4e81-bbfc-761d8faadd07\") " Oct 30 19:13:07.152975 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:07.152943 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5224824-05eb-4e81-bbfc-761d8faadd07-maas-api-openshift-ca" pod "f5224824-05eb-4e81-bbfc-761d8faadd07" (UID: "f5224824-05eb-4e81-bbfc-761d8faadd07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:13:07.153124 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:07.152632 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5224824-05eb-4e81-bbfc-761d8faadd07-kube-api-access-gg9k4\" (UniqueName: \"kubernetes.io/projected/f5224824-05eb-4e81-bbfc-761d8faadd07-kube-api-access-gg9k4\") pod \"f5224824-05eb-4e81-bbfc-761d8faadd07\" (UID: \"f5224824-05eb-4e81-bbfc-761d8faadd07\") " Oct 30 19:13:07.153701 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:07.153681 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5224824-05eb-4e81-bbfc-761d8faadd07-maas-api-openshift-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:07.155545 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:07.155498 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5224824-05eb-4e81-bbfc-761d8faadd07-maas-api-xks-ca" pod "f5224824-05eb-4e81-bbfc-761d8faadd07" (UID: "f5224824-05eb-4e81-bbfc-761d8faadd07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:13:07.155796 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:07.155767 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5224824-05eb-4e81-bbfc-761d8faadd07-kube-api-access-gg9k4" pod "f5224824-05eb-4e81-bbfc-761d8faadd07" (UID: "f5224824-05eb-4e81-bbfc-761d8faadd07"). InnerVolumeSpecName "kube-api-access-gg9k4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:07.255328 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:07.255246 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg9k4\" (UniqueName: \"kubernetes.io/projected/f5224824-05eb-4e81-bbfc-761d8faadd07-kube-api-access-gg9k4\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:07.255328 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:07.255272 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5224824-05eb-4e81-bbfc-761d8faadd07-maas-api-xks-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:07.933943 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:07.933916 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d1a330-8zpht" Oct 30 19:13:07.934159 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:07.933938 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d1a330-8zpht" event={"ID":"f5224824-05eb-4e81-bbfc-761d8faadd07","Type":"ContainerDied","Data":"859e724ad88423183b52e2db4cc278438a6fa673f222e8fc1607e411ac7283cb"} Oct 30 19:13:07.934159 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:07.934033 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="859e724ad88423183b52e2db4cc278438a6fa673f222e8fc1607e411ac7283cb" Oct 30 19:13:09.293108 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:09.293074 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq"] Oct 30 19:13:09.293646 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:09.293353 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" podUID="b0040e62-2a9b-4af5-82bd-730b0f5aba4f" containerName="istio-proxy" containerID="cri-o://e9bb720ca2ef9b43720554fb70468edcd32a224006209bb7433772ae4e552716" gracePeriod=30 Oct 30 19:13:09.906078 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:09.906046 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8b8e54-f8mqf"] Oct 30 19:13:09.906678 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:09.906661 4442 cpu_manager.go:414] "containerMap: removing container" podUID="f5224824-05eb-4e81-bbfc-761d8faadd07" containerName="revoke-keys" Oct 30 19:13:09.906678 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:09.906678 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5224824-05eb-4e81-bbfc-761d8faadd07" containerName="revoke-keys" Oct 30 19:13:09.906834 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:09.906784 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5224824-05eb-4e81-bbfc-761d8faadd07" containerName="revoke-keys" Oct 30 19:13:09.910282 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:09.910261 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8b8e54-f8mqf" Oct 30 19:13:09.931841 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:09.931808 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8b8e54-f8mqf"] Oct 30 19:13:09.942434 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:09.942406 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:13:09.942673 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:09.942450 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:13:09.996626 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:09.996596 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-8b8e54-f8mqf\" (UID: \"d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8b8e54-f8mqf" Oct 30 19:13:09.996774 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:09.996732 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-8b8e54-f8mqf\" (UID: \"d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8b8e54-f8mqf" Oct 30 19:13:09.996848 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:09.996834 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf2td\" (UniqueName: \"kubernetes.io/projected/d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb-kube-api-access-kf2td\") pod \"maas-api-revoke-keys-e2e-extmod-b-8b8e54-f8mqf\" (UID: \"d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8b8e54-f8mqf" Oct 30 19:13:10.098430 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:10.098393 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf2td\" (UniqueName: \"kubernetes.io/projected/d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb-kube-api-access-kf2td\") pod \"maas-api-revoke-keys-e2e-extmod-b-8b8e54-f8mqf\" (UID: \"d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8b8e54-f8mqf" Oct 30 19:13:10.098629 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:10.098505 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-8b8e54-f8mqf\" (UID: \"d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8b8e54-f8mqf" Oct 30 19:13:10.098698 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:10.098648 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-8b8e54-f8mqf\" (UID: \"d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8b8e54-f8mqf" Oct 30 19:13:10.099376 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:10.099343 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-8b8e54-f8mqf\" (UID: \"d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8b8e54-f8mqf" Oct 30 19:13:10.101326 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:10.101300 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-8b8e54-f8mqf\" (UID: \"d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8b8e54-f8mqf" Oct 30 19:13:10.117822 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:10.117795 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf2td\" (UniqueName: \"kubernetes.io/projected/d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb-kube-api-access-kf2td\") pod \"maas-api-revoke-keys-e2e-extmod-b-8b8e54-f8mqf\" (UID: \"d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8b8e54-f8mqf" Oct 30 19:13:10.224996 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:10.224912 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8b8e54-f8mqf" Oct 30 19:13:10.526256 ip-10-0-138-176 kubenswrapper[4442]: W1030 19:13:10.526223 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4d6bfbe_8c6b_46d7_be96_a76ac188d1fb.slice/crio-1911e6c194ac61549bdc5bd59a6ed0b59ab160eb87aa3659317db82930c9e90f WatchSource:0}: Error finding container 1911e6c194ac61549bdc5bd59a6ed0b59ab160eb87aa3659317db82930c9e90f: Status 404 returned error can't find the container with id 1911e6c194ac61549bdc5bd59a6ed0b59ab160eb87aa3659317db82930c9e90f Oct 30 19:13:10.526638 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:10.526623 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8b8e54-f8mqf"] Oct 30 19:13:10.962039 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:10.961995 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8b8e54-f8mqf" event={"ID":"d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb","Type":"ContainerStarted","Data":"537ce0e83e0f5dca9cbf7deb42a4fa3c8bec1c2ddf8a22d44211066957ccfcc4"} Oct 30 19:13:10.962284 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:10.962057 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8b8e54-f8mqf" event={"ID":"d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb","Type":"ContainerStarted","Data":"1911e6c194ac61549bdc5bd59a6ed0b59ab160eb87aa3659317db82930c9e90f"} Oct 30 19:13:11.000858 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:11.000809 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8b8e54-f8mqf" podStartSLOduration=2.000794878 podStartE2EDuration="2.000794878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:13:10.999119203 +0000 UTC m=+4488.089181543" watchObservedRunningTime="2026-10-30 19:13:11.000794878 +0000 UTC m=+4488.090857217" Oct 30 19:13:11.972999 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:11.972966 4442 generic.go:356] "Generic (PLEG): container finished" podID="d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb" containerID="537ce0e83e0f5dca9cbf7deb42a4fa3c8bec1c2ddf8a22d44211066957ccfcc4" exitCode=0 Oct 30 19:13:11.973439 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:11.973023 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8b8e54-f8mqf" event={"ID":"d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb","Type":"ContainerDied","Data":"537ce0e83e0f5dca9cbf7deb42a4fa3c8bec1c2ddf8a22d44211066957ccfcc4"} Oct 30 19:13:13.158785 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:13.158756 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8b8e54-f8mqf" Oct 30 19:13:13.255525 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:13.255448 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb-maas-api-openshift-ca\") pod \"d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb\" (UID: \"d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb\") " Oct 30 19:13:13.255685 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:13.255645 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb-kube-api-access-kf2td\" (UniqueName: \"kubernetes.io/projected/d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb-kube-api-access-kf2td\") pod \"d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb\" (UID: \"d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb\") " Oct 30 19:13:13.255736 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:13.255710 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb-maas-api-xks-ca\") pod \"d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb\" (UID: \"d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb\") " Oct 30 19:13:13.255963 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:13.255930 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb-maas-api-openshift-ca" pod "d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb" (UID: "d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:13:13.258164 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:13.258097 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb-maas-api-xks-ca" pod "d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb" (UID: "d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:13:13.258164 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:13.258148 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb-kube-api-access-kf2td" pod "d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb" (UID: "d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb"). InnerVolumeSpecName "kube-api-access-kf2td". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:13.357722 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:13.357689 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf2td\" (UniqueName: \"kubernetes.io/projected/d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb-kube-api-access-kf2td\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:13.357722 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:13.357721 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb-maas-api-xks-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:13.357941 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:13.357732 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb-maas-api-openshift-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:13.989874 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:13.989844 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8b8e54-f8mqf" Oct 30 19:13:13.990093 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:13.989905 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8b8e54-f8mqf" event={"ID":"d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb","Type":"ContainerDied","Data":"1911e6c194ac61549bdc5bd59a6ed0b59ab160eb87aa3659317db82930c9e90f"} Oct 30 19:13:13.990093 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:13.989957 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1911e6c194ac61549bdc5bd59a6ed0b59ab160eb87aa3659317db82930c9e90f" Oct 30 19:13:14.698513 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:14.692717 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" Oct 30 19:13:14.812536 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:14.812500 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-istiod-ca-cert\") pod \"b0040e62-2a9b-4af5-82bd-730b0f5aba4f\" (UID: \"b0040e62-2a9b-4af5-82bd-730b0f5aba4f\") " Oct 30 19:13:14.813235 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:14.812561 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-credential-socket\") pod \"b0040e62-2a9b-4af5-82bd-730b0f5aba4f\" (UID: \"b0040e62-2a9b-4af5-82bd-730b0f5aba4f\") " Oct 30 19:13:14.813235 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:14.812604 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-workload-certs\") pod \"b0040e62-2a9b-4af5-82bd-730b0f5aba4f\" (UID: \"b0040e62-2a9b-4af5-82bd-730b0f5aba4f\") " Oct 30 19:13:14.813235 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:14.812673 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-istio-token\" (UniqueName: \"kubernetes.io/projected/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-istio-token\") pod \"b0040e62-2a9b-4af5-82bd-730b0f5aba4f\" (UID: \"b0040e62-2a9b-4af5-82bd-730b0f5aba4f\") " Oct 30 19:13:14.813235 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:14.812767 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-istio-envoy\") pod \"b0040e62-2a9b-4af5-82bd-730b0f5aba4f\" (UID: \"b0040e62-2a9b-4af5-82bd-730b0f5aba4f\") " Oct 30 19:13:14.813235 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:14.812795 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-workload-socket\") pod \"b0040e62-2a9b-4af5-82bd-730b0f5aba4f\" (UID: \"b0040e62-2a9b-4af5-82bd-730b0f5aba4f\") " Oct 30 19:13:14.813235 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:14.812908 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-istio-podinfo\") pod \"b0040e62-2a9b-4af5-82bd-730b0f5aba4f\" (UID: \"b0040e62-2a9b-4af5-82bd-730b0f5aba4f\") " Oct 30 19:13:14.813235 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:14.812959 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-istio-data\") pod \"b0040e62-2a9b-4af5-82bd-730b0f5aba4f\" (UID: \"b0040e62-2a9b-4af5-82bd-730b0f5aba4f\") " Oct 30 19:13:14.813235 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:14.813060 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-kube-api-access-ndtd7\" (UniqueName: \"kubernetes.io/projected/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-kube-api-access-ndtd7\") pod \"b0040e62-2a9b-4af5-82bd-730b0f5aba4f\" (UID: \"b0040e62-2a9b-4af5-82bd-730b0f5aba4f\") " Oct 30 19:13:14.814029 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:14.813996 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-workload-socket" pod "b0040e62-2a9b-4af5-82bd-730b0f5aba4f" (UID: "b0040e62-2a9b-4af5-82bd-730b0f5aba4f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:14.814321 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:14.814293 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-istio-data" pod "b0040e62-2a9b-4af5-82bd-730b0f5aba4f" (UID: "b0040e62-2a9b-4af5-82bd-730b0f5aba4f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:14.814540 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:14.814497 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-credential-socket" pod "b0040e62-2a9b-4af5-82bd-730b0f5aba4f" (UID: "b0040e62-2a9b-4af5-82bd-730b0f5aba4f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:14.814811 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:14.814789 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-istiod-ca-cert" pod "b0040e62-2a9b-4af5-82bd-730b0f5aba4f" (UID: "b0040e62-2a9b-4af5-82bd-730b0f5aba4f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:13:14.815070 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:14.815041 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-workload-certs" pod "b0040e62-2a9b-4af5-82bd-730b0f5aba4f" (UID: "b0040e62-2a9b-4af5-82bd-730b0f5aba4f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:14.823599 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:14.823540 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-istio-podinfo" pod "b0040e62-2a9b-4af5-82bd-730b0f5aba4f" (UID: "b0040e62-2a9b-4af5-82bd-730b0f5aba4f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:13:14.823902 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:14.823877 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-istio-token" pod "b0040e62-2a9b-4af5-82bd-730b0f5aba4f" (UID: "b0040e62-2a9b-4af5-82bd-730b0f5aba4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:14.824208 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:14.824175 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-kube-api-access-ndtd7" pod "b0040e62-2a9b-4af5-82bd-730b0f5aba4f" (UID: "b0040e62-2a9b-4af5-82bd-730b0f5aba4f"). InnerVolumeSpecName "kube-api-access-ndtd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:14.829497 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:14.829437 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-istio-envoy" pod "b0040e62-2a9b-4af5-82bd-730b0f5aba4f" (UID: "b0040e62-2a9b-4af5-82bd-730b0f5aba4f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:14.914460 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:14.914378 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:14.914460 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:14.914408 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:14.914460 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:14.914418 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndtd7\" (UniqueName: \"kubernetes.io/projected/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-kube-api-access-ndtd7\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:14.914460 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:14.914427 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:14.914460 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:14.914435 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:14.914460 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:14.914443 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:14.914460 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:14.914452 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:14.914460 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:14.914459 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:14.914460 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:14.914467 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0040e62-2a9b-4af5-82bd-730b0f5aba4f-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:15.000873 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:15.000843 4442 generic.go:356] "Generic (PLEG): container finished" podID="b0040e62-2a9b-4af5-82bd-730b0f5aba4f" containerID="e9bb720ca2ef9b43720554fb70468edcd32a224006209bb7433772ae4e552716" exitCode=0 Oct 30 19:13:15.001087 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:15.000908 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" Oct 30 19:13:15.001087 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:15.000917 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" event={"ID":"b0040e62-2a9b-4af5-82bd-730b0f5aba4f","Type":"ContainerDied","Data":"e9bb720ca2ef9b43720554fb70468edcd32a224006209bb7433772ae4e552716"} Oct 30 19:13:15.001087 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:15.000959 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq" event={"ID":"b0040e62-2a9b-4af5-82bd-730b0f5aba4f","Type":"ContainerDied","Data":"c67c9577d1e4c574f84d99aa258353094fd0ce87905b854b62cc7c93f8388a95"} Oct 30 19:13:15.001087 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:15.000980 4442 scope.go:122] "RemoveContainer" containerID="e9bb720ca2ef9b43720554fb70468edcd32a224006209bb7433772ae4e552716" Oct 30 19:13:15.011990 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:15.011966 4442 scope.go:122] "RemoveContainer" containerID="da4ca7073b149a7b14639f368ab7d085b7db275859c059436f4cffa352298e1f" Oct 30 19:13:15.023325 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:15.023306 4442 scope.go:122] "RemoveContainer" containerID="e9bb720ca2ef9b43720554fb70468edcd32a224006209bb7433772ae4e552716" Oct 30 19:13:15.023648 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:13:15.023628 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9bb720ca2ef9b43720554fb70468edcd32a224006209bb7433772ae4e552716\": container with ID starting with e9bb720ca2ef9b43720554fb70468edcd32a224006209bb7433772ae4e552716 not found: ID does not exist" containerID="e9bb720ca2ef9b43720554fb70468edcd32a224006209bb7433772ae4e552716" Oct 30 19:13:15.023748 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:15.023655 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9bb720ca2ef9b43720554fb70468edcd32a224006209bb7433772ae4e552716"} err="failed to get container status \"e9bb720ca2ef9b43720554fb70468edcd32a224006209bb7433772ae4e552716\": rpc error: code = NotFound desc = could not find container \"e9bb720ca2ef9b43720554fb70468edcd32a224006209bb7433772ae4e552716\": container with ID starting with e9bb720ca2ef9b43720554fb70468edcd32a224006209bb7433772ae4e552716 not found: ID does not exist" Oct 30 19:13:15.023748 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:15.023672 4442 scope.go:122] "RemoveContainer" containerID="da4ca7073b149a7b14639f368ab7d085b7db275859c059436f4cffa352298e1f" Oct 30 19:13:15.023956 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:13:15.023939 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da4ca7073b149a7b14639f368ab7d085b7db275859c059436f4cffa352298e1f\": container with ID starting with da4ca7073b149a7b14639f368ab7d085b7db275859c059436f4cffa352298e1f not found: ID does not exist" containerID="da4ca7073b149a7b14639f368ab7d085b7db275859c059436f4cffa352298e1f" Oct 30 19:13:15.024033 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:15.023962 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da4ca7073b149a7b14639f368ab7d085b7db275859c059436f4cffa352298e1f"} err="failed to get container status \"da4ca7073b149a7b14639f368ab7d085b7db275859c059436f4cffa352298e1f\": rpc error: code = NotFound desc = could not find container \"da4ca7073b149a7b14639f368ab7d085b7db275859c059436f4cffa352298e1f\": container with ID starting with da4ca7073b149a7b14639f368ab7d085b7db275859c059436f4cffa352298e1f not found: ID does not exist" Oct 30 19:13:15.056947 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:15.056888 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq"] Oct 30 19:13:15.065791 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:15.065758 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-1c4bcd-openshift-default-847756f8fc-q75rq"] Oct 30 19:13:17.066720 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:17.066687 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0040e62-2a9b-4af5-82bd-730b0f5aba4f" path="/var/lib/kubelet/pods/b0040e62-2a9b-4af5-82bd-730b0f5aba4f/volumes" Oct 30 19:13:17.561783 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:13:17.561739 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 19:13:18.171921 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:18.171870 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-09ef1d/rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8" podUID="ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106" containerName="main" containerID="cri-o://b3a71c95aa9b6631e47e40fd84d8b7c66e5859f5b938056b4628c84041ee0ed7" gracePeriod=45 Oct 30 19:13:18.243816 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:13:18.243782 4442 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac7e3bc8_8a90_41ac_9ed9_1cad1ae82106.slice/crio-conmon-b3a71c95aa9b6631e47e40fd84d8b7c66e5859f5b938056b4628c84041ee0ed7.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac7e3bc8_8a90_41ac_9ed9_1cad1ae82106.slice/crio-b3a71c95aa9b6631e47e40fd84d8b7c66e5859f5b938056b4628c84041ee0ed7.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:13:18.445723 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:18.445695 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-09ef1d/rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8" Oct 30 19:13:18.577950 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:18.577917 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106-kube-api-access-cmb7n\" (UniqueName: \"kubernetes.io/projected/ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106-kube-api-access-cmb7n\") pod \"ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106\" (UID: \"ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106\") " Oct 30 19:13:18.578123 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:18.578023 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106-model-cache\") pod \"ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106\" (UID: \"ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106\") " Oct 30 19:13:18.578123 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:18.578060 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106-dshm\") pod \"ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106\" (UID: \"ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106\") " Oct 30 19:13:18.578123 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:18.578115 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106-tls-certs\" (UniqueName: \"kubernetes.io/secret/ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106-tls-certs\") pod \"ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106\" (UID: \"ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106\") " Oct 30 19:13:18.578304 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:18.578187 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106-home\" (UniqueName: \"kubernetes.io/empty-dir/ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106-home\") pod \"ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106\" (UID: \"ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106\") " Oct 30 19:13:18.578382 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:18.578347 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106-model-cache" pod "ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106" (UID: "ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:18.578382 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:18.578369 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106-tmp-dir\") pod \"ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106\" (UID: \"ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106\") " Oct 30 19:13:18.578631 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:18.578564 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106-home" pod "ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106" (UID: "ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:18.578631 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:18.578579 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106-tmp-dir" pod "ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106" (UID: "ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:18.578935 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:18.578914 4442 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106-model-cache\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:18.578935 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:18.578938 4442 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106-home\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:18.579089 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:18.578951 4442 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106-tmp-dir\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:18.580665 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:18.580644 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106-tls-certs" pod "ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106" (UID: "ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:13:18.580875 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:18.580858 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106-dshm" pod "ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106" (UID: "ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:18.580938 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:18.580888 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106-kube-api-access-cmb7n" pod "ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106" (UID: "ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106"). InnerVolumeSpecName "kube-api-access-cmb7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:18.680341 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:18.680237 4442 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106-dshm\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:18.680341 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:18.680279 4442 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106-tls-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:18.680341 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:18.680298 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmb7n\" (UniqueName: \"kubernetes.io/projected/ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106-kube-api-access-cmb7n\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:19.113249 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:19.109345 4442 generic.go:356] "Generic (PLEG): container finished" podID="ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106" containerID="b3a71c95aa9b6631e47e40fd84d8b7c66e5859f5b938056b4628c84041ee0ed7" exitCode=0 Oct 30 19:13:19.113249 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:19.109514 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-09ef1d/rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8" Oct 30 19:13:19.121507 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:19.121430 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-09ef1d/rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8" event={"ID":"ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106","Type":"ContainerDied","Data":"b3a71c95aa9b6631e47e40fd84d8b7c66e5859f5b938056b4628c84041ee0ed7"} Oct 30 19:13:19.124736 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:19.124235 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-09ef1d/rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8" event={"ID":"ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106","Type":"ContainerDied","Data":"ca69e47baabbee41216bca33324e1a9459c8d2f98c2a69092aa98c9fb8324833"} Oct 30 19:13:19.124736 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:19.124278 4442 scope.go:122] "RemoveContainer" containerID="b3a71c95aa9b6631e47e40fd84d8b7c66e5859f5b938056b4628c84041ee0ed7" Oct 30 19:13:19.176292 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:19.175728 4442 scope.go:122] "RemoveContainer" containerID="b3a71c95aa9b6631e47e40fd84d8b7c66e5859f5b938056b4628c84041ee0ed7" Oct 30 19:13:19.176292 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:13:19.176194 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3a71c95aa9b6631e47e40fd84d8b7c66e5859f5b938056b4628c84041ee0ed7\": container with ID starting with b3a71c95aa9b6631e47e40fd84d8b7c66e5859f5b938056b4628c84041ee0ed7 not found: ID does not exist" containerID="b3a71c95aa9b6631e47e40fd84d8b7c66e5859f5b938056b4628c84041ee0ed7" Oct 30 19:13:19.176292 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:19.176230 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3a71c95aa9b6631e47e40fd84d8b7c66e5859f5b938056b4628c84041ee0ed7"} err="failed to get container status \"b3a71c95aa9b6631e47e40fd84d8b7c66e5859f5b938056b4628c84041ee0ed7\": rpc error: code = NotFound desc = could not find container \"b3a71c95aa9b6631e47e40fd84d8b7c66e5859f5b938056b4628c84041ee0ed7\": container with ID starting with b3a71c95aa9b6631e47e40fd84d8b7c66e5859f5b938056b4628c84041ee0ed7 not found: ID does not exist" Oct 30 19:13:19.202079 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:19.202039 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-09ef1d/rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8"] Oct 30 19:13:19.216382 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:19.216349 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-09ef1d/rate-test-model-09ef1d-kserve-5bdb854b9-zdqh8"] Oct 30 19:13:19.866984 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:19.866920 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d1a330-8zpht"] Oct 30 19:13:19.879518 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:19.879480 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d1a330-8zpht"] Oct 30 19:13:20.420103 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:20.420068 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-8b8e54-77dc6fdfc4-j8v2w"] Oct 30 19:13:21.066361 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:21.066312 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106" path="/var/lib/kubelet/pods/ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106/volumes" Oct 30 19:13:21.075253 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:21.073450 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5224824-05eb-4e81-bbfc-761d8faadd07" path="/var/lib/kubelet/pods/f5224824-05eb-4e81-bbfc-761d8faadd07/volumes" Oct 30 19:13:24.319065 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:24.319039 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52rn6_516c6b6c-fc40-49cf-93b5-1c4471b02cf0/ovn-acl-logging/0.log" Oct 30 19:13:24.322274 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:24.322253 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52rn6_516c6b6c-fc40-49cf-93b5-1c4471b02cf0/ovn-acl-logging/0.log" Oct 30 19:13:25.420928 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:25.420880 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-8b8e54-77dc6fdfc4-j8v2w" podUID="58035120-0d0b-48c0-9096-c531d8b10f82" containerName="payload-processing" containerID="cri-o://a4ad388a3031f8871663c467537bb09ad22349a65b035c4109a04828a9598ea4" gracePeriod=25 Oct 30 19:13:27.146864 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:27.146816 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78"] Oct 30 19:13:27.147397 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:27.147087 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" podUID="dd0f5b27-6383-4d55-a840-3938d11d9f6f" containerName="istio-proxy" containerID="cri-o://f954150b0c9ed044391ce6a5e6a8a8c56b3fc36d4dc080f184f31b80d8777e35" gracePeriod=30 Oct 30 19:13:27.760721 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:27.760681 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6babca-8krfd"] Oct 30 19:13:27.761341 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:27.761317 4442 cpu_manager.go:414] "containerMap: removing container" podUID="b0040e62-2a9b-4af5-82bd-730b0f5aba4f" containerName="istio-proxy" Oct 30 19:13:27.761341 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:27.761337 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0040e62-2a9b-4af5-82bd-730b0f5aba4f" containerName="istio-proxy" Oct 30 19:13:27.761525 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:27.761366 4442 cpu_manager.go:414] "containerMap: removing container" podUID="d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb" containerName="revoke-keys" Oct 30 19:13:27.761525 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:27.761375 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb" containerName="revoke-keys" Oct 30 19:13:27.761525 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:27.761399 4442 cpu_manager.go:414] "containerMap: removing container" podUID="b0040e62-2a9b-4af5-82bd-730b0f5aba4f" containerName="istio-proxy" Oct 30 19:13:27.761525 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:27.761404 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0040e62-2a9b-4af5-82bd-730b0f5aba4f" containerName="istio-proxy" Oct 30 19:13:27.761525 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:27.761418 4442 cpu_manager.go:414] "containerMap: removing container" podUID="ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106" containerName="main" Oct 30 19:13:27.761525 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:27.761423 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106" containerName="main" Oct 30 19:13:27.761525 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:27.761507 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0040e62-2a9b-4af5-82bd-730b0f5aba4f" containerName="istio-proxy" Oct 30 19:13:27.761525 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:27.761517 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac7e3bc8-8a90-41ac-9ed9-1cad1ae82106" containerName="main" Oct 30 19:13:27.761525 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:27.761526 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0040e62-2a9b-4af5-82bd-730b0f5aba4f" containerName="istio-proxy" Oct 30 19:13:27.761947 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:27.761533 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb" containerName="revoke-keys" Oct 30 19:13:27.766674 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:27.766657 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6babca-8krfd" Oct 30 19:13:27.784424 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:27.784397 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6babca-8krfd"] Oct 30 19:13:27.786589 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:27.786565 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:13:27.786843 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:27.786820 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:13:27.864474 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:27.864443 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95801f03-019f-43ce-888c-88f00884c618-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-6babca-8krfd\" (UID: \"95801f03-019f-43ce-888c-88f00884c618\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6babca-8krfd" Oct 30 19:13:27.864660 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:27.864616 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95801f03-019f-43ce-888c-88f00884c618-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-6babca-8krfd\" (UID: \"95801f03-019f-43ce-888c-88f00884c618\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6babca-8krfd" Oct 30 19:13:27.864660 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:27.864655 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwpnj\" (UniqueName: \"kubernetes.io/projected/95801f03-019f-43ce-888c-88f00884c618-kube-api-access-hwpnj\") pod \"maas-api-revoke-keys-e2e-extmod-b-6babca-8krfd\" (UID: \"95801f03-019f-43ce-888c-88f00884c618\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6babca-8krfd" Oct 30 19:13:27.965327 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:27.965299 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95801f03-019f-43ce-888c-88f00884c618-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-6babca-8krfd\" (UID: \"95801f03-019f-43ce-888c-88f00884c618\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6babca-8krfd" Oct 30 19:13:27.965512 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:27.965430 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95801f03-019f-43ce-888c-88f00884c618-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-6babca-8krfd\" (UID: \"95801f03-019f-43ce-888c-88f00884c618\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6babca-8krfd" Oct 30 19:13:27.965512 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:27.965459 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwpnj\" (UniqueName: \"kubernetes.io/projected/95801f03-019f-43ce-888c-88f00884c618-kube-api-access-hwpnj\") pod \"maas-api-revoke-keys-e2e-extmod-b-6babca-8krfd\" (UID: \"95801f03-019f-43ce-888c-88f00884c618\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6babca-8krfd" Oct 30 19:13:27.966127 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:27.966101 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95801f03-019f-43ce-888c-88f00884c618-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-6babca-8krfd\" (UID: \"95801f03-019f-43ce-888c-88f00884c618\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6babca-8krfd" Oct 30 19:13:27.967830 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:27.967806 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95801f03-019f-43ce-888c-88f00884c618-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-6babca-8krfd\" (UID: \"95801f03-019f-43ce-888c-88f00884c618\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6babca-8krfd" Oct 30 19:13:27.984820 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:27.984795 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwpnj\" (UniqueName: \"kubernetes.io/projected/95801f03-019f-43ce-888c-88f00884c618-kube-api-access-hwpnj\") pod \"maas-api-revoke-keys-e2e-extmod-b-6babca-8krfd\" (UID: \"95801f03-019f-43ce-888c-88f00884c618\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6babca-8krfd" Oct 30 19:13:28.080804 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:28.080720 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6babca-8krfd" Oct 30 19:13:28.387385 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:28.387355 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6babca-8krfd"] Oct 30 19:13:28.387717 ip-10-0-138-176 kubenswrapper[4442]: W1030 19:13:28.387424 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95801f03_019f_43ce_888c_88f00884c618.slice/crio-8ad8d349af92e5aa5e6c7f7d9249a9f0e4f7f8050fff93ac387a34dd7c256f2d WatchSource:0}: Error finding container 8ad8d349af92e5aa5e6c7f7d9249a9f0e4f7f8050fff93ac387a34dd7c256f2d: Status 404 returned error can't find the container with id 8ad8d349af92e5aa5e6c7f7d9249a9f0e4f7f8050fff93ac387a34dd7c256f2d Oct 30 19:13:29.237389 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:29.237362 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6babca-8krfd" event={"ID":"95801f03-019f-43ce-888c-88f00884c618","Type":"ContainerStarted","Data":"5be51acd21723fd3dca020893305307d539baee69f16e1e45498131b8f293b5b"} Oct 30 19:13:29.237505 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:29.237425 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6babca-8krfd" event={"ID":"95801f03-019f-43ce-888c-88f00884c618","Type":"ContainerStarted","Data":"8ad8d349af92e5aa5e6c7f7d9249a9f0e4f7f8050fff93ac387a34dd7c256f2d"} Oct 30 19:13:29.279297 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:29.279246 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6babca-8krfd" podStartSLOduration=2.279229371 podStartE2EDuration="2.279229371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:13:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:13:29.277812661 +0000 UTC m=+4506.367875013" watchObservedRunningTime="2026-10-30 19:13:29.279229371 +0000 UTC m=+4506.369291729" Oct 30 19:13:30.247050 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:30.247015 4442 generic.go:356] "Generic (PLEG): container finished" podID="95801f03-019f-43ce-888c-88f00884c618" containerID="5be51acd21723fd3dca020893305307d539baee69f16e1e45498131b8f293b5b" exitCode=0 Oct 30 19:13:30.247608 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:30.247099 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6babca-8krfd" event={"ID":"95801f03-019f-43ce-888c-88f00884c618","Type":"ContainerDied","Data":"5be51acd21723fd3dca020893305307d539baee69f16e1e45498131b8f293b5b"} Oct 30 19:13:31.452760 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:31.452731 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6babca-8krfd" Oct 30 19:13:31.511082 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:31.510994 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95801f03-019f-43ce-888c-88f00884c618-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95801f03-019f-43ce-888c-88f00884c618-maas-api-openshift-ca\") pod \"95801f03-019f-43ce-888c-88f00884c618\" (UID: \"95801f03-019f-43ce-888c-88f00884c618\") " Oct 30 19:13:31.511347 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:31.511172 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95801f03-019f-43ce-888c-88f00884c618-kube-api-access-hwpnj\" (UniqueName: \"kubernetes.io/projected/95801f03-019f-43ce-888c-88f00884c618-kube-api-access-hwpnj\") pod \"95801f03-019f-43ce-888c-88f00884c618\" (UID: \"95801f03-019f-43ce-888c-88f00884c618\") " Oct 30 19:13:31.511347 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:31.511273 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95801f03-019f-43ce-888c-88f00884c618-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95801f03-019f-43ce-888c-88f00884c618-maas-api-xks-ca\") pod \"95801f03-019f-43ce-888c-88f00884c618\" (UID: \"95801f03-019f-43ce-888c-88f00884c618\") " Oct 30 19:13:31.511491 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:31.511386 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95801f03-019f-43ce-888c-88f00884c618-maas-api-openshift-ca" pod "95801f03-019f-43ce-888c-88f00884c618" (UID: "95801f03-019f-43ce-888c-88f00884c618"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:13:31.511762 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:31.511732 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95801f03-019f-43ce-888c-88f00884c618-maas-api-openshift-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:31.513543 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:31.513515 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95801f03-019f-43ce-888c-88f00884c618-kube-api-access-hwpnj" pod "95801f03-019f-43ce-888c-88f00884c618" (UID: "95801f03-019f-43ce-888c-88f00884c618"). InnerVolumeSpecName "kube-api-access-hwpnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:31.513669 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:31.513561 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95801f03-019f-43ce-888c-88f00884c618-maas-api-xks-ca" pod "95801f03-019f-43ce-888c-88f00884c618" (UID: "95801f03-019f-43ce-888c-88f00884c618"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:13:31.612932 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:31.612893 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwpnj\" (UniqueName: \"kubernetes.io/projected/95801f03-019f-43ce-888c-88f00884c618-kube-api-access-hwpnj\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:31.612932 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:31.612926 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95801f03-019f-43ce-888c-88f00884c618-maas-api-xks-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:32.267973 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:32.267599 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6babca-8krfd" Oct 30 19:13:32.267973 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:32.267597 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6babca-8krfd" event={"ID":"95801f03-019f-43ce-888c-88f00884c618","Type":"ContainerDied","Data":"8ad8d349af92e5aa5e6c7f7d9249a9f0e4f7f8050fff93ac387a34dd7c256f2d"} Oct 30 19:13:32.267973 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:32.267749 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ad8d349af92e5aa5e6c7f7d9249a9f0e4f7f8050fff93ac387a34dd7c256f2d" Oct 30 19:13:32.400880 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:32.400854 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" Oct 30 19:13:32.524868 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:32.524829 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd0f5b27-6383-4d55-a840-3938d11d9f6f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd0f5b27-6383-4d55-a840-3938d11d9f6f-istio-podinfo\") pod \"dd0f5b27-6383-4d55-a840-3938d11d9f6f\" (UID: \"dd0f5b27-6383-4d55-a840-3938d11d9f6f\") " Oct 30 19:13:32.525349 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:32.524888 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd0f5b27-6383-4d55-a840-3938d11d9f6f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd0f5b27-6383-4d55-a840-3938d11d9f6f-istiod-ca-cert\") pod \"dd0f5b27-6383-4d55-a840-3938d11d9f6f\" (UID: \"dd0f5b27-6383-4d55-a840-3938d11d9f6f\") " Oct 30 19:13:32.525349 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:32.524919 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd0f5b27-6383-4d55-a840-3938d11d9f6f-kube-api-access-4ggcx\" (UniqueName: \"kubernetes.io/projected/dd0f5b27-6383-4d55-a840-3938d11d9f6f-kube-api-access-4ggcx\") pod \"dd0f5b27-6383-4d55-a840-3938d11d9f6f\" (UID: \"dd0f5b27-6383-4d55-a840-3938d11d9f6f\") " Oct 30 19:13:32.525349 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:32.524975 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd0f5b27-6383-4d55-a840-3938d11d9f6f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd0f5b27-6383-4d55-a840-3938d11d9f6f-istio-envoy\") pod \"dd0f5b27-6383-4d55-a840-3938d11d9f6f\" (UID: \"dd0f5b27-6383-4d55-a840-3938d11d9f6f\") " Oct 30 19:13:32.525349 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:32.525067 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd0f5b27-6383-4d55-a840-3938d11d9f6f-istio-token\" (UniqueName: \"kubernetes.io/projected/dd0f5b27-6383-4d55-a840-3938d11d9f6f-istio-token\") pod \"dd0f5b27-6383-4d55-a840-3938d11d9f6f\" (UID: \"dd0f5b27-6383-4d55-a840-3938d11d9f6f\") " Oct 30 19:13:32.525349 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:32.525096 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd0f5b27-6383-4d55-a840-3938d11d9f6f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd0f5b27-6383-4d55-a840-3938d11d9f6f-workload-certs\") pod \"dd0f5b27-6383-4d55-a840-3938d11d9f6f\" (UID: \"dd0f5b27-6383-4d55-a840-3938d11d9f6f\") " Oct 30 19:13:32.525349 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:32.525126 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd0f5b27-6383-4d55-a840-3938d11d9f6f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0f5b27-6383-4d55-a840-3938d11d9f6f-workload-socket\") pod \"dd0f5b27-6383-4d55-a840-3938d11d9f6f\" (UID: \"dd0f5b27-6383-4d55-a840-3938d11d9f6f\") " Oct 30 19:13:32.525349 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:32.525342 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd0f5b27-6383-4d55-a840-3938d11d9f6f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd0f5b27-6383-4d55-a840-3938d11d9f6f-istio-data\") pod \"dd0f5b27-6383-4d55-a840-3938d11d9f6f\" (UID: \"dd0f5b27-6383-4d55-a840-3938d11d9f6f\") " Oct 30 19:13:32.525767 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:32.525411 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd0f5b27-6383-4d55-a840-3938d11d9f6f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0f5b27-6383-4d55-a840-3938d11d9f6f-credential-socket\") pod \"dd0f5b27-6383-4d55-a840-3938d11d9f6f\" (UID: \"dd0f5b27-6383-4d55-a840-3938d11d9f6f\") " Oct 30 19:13:32.525767 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:32.525484 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd0f5b27-6383-4d55-a840-3938d11d9f6f-istiod-ca-cert" pod "dd0f5b27-6383-4d55-a840-3938d11d9f6f" (UID: "dd0f5b27-6383-4d55-a840-3938d11d9f6f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:13:32.525767 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:32.525528 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd0f5b27-6383-4d55-a840-3938d11d9f6f-workload-socket" pod "dd0f5b27-6383-4d55-a840-3938d11d9f6f" (UID: "dd0f5b27-6383-4d55-a840-3938d11d9f6f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:32.525767 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:32.525742 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd0f5b27-6383-4d55-a840-3938d11d9f6f-credential-socket" pod "dd0f5b27-6383-4d55-a840-3938d11d9f6f" (UID: "dd0f5b27-6383-4d55-a840-3938d11d9f6f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:32.526005 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:32.525789 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd0f5b27-6383-4d55-a840-3938d11d9f6f-istio-data" pod "dd0f5b27-6383-4d55-a840-3938d11d9f6f" (UID: "dd0f5b27-6383-4d55-a840-3938d11d9f6f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:32.526005 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:32.525968 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd0f5b27-6383-4d55-a840-3938d11d9f6f-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:32.526005 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:32.525992 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0f5b27-6383-4d55-a840-3938d11d9f6f-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:32.526170 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:32.526006 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd0f5b27-6383-4d55-a840-3938d11d9f6f-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:32.526170 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:32.526020 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0f5b27-6383-4d55-a840-3938d11d9f6f-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:32.526170 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:32.526110 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd0f5b27-6383-4d55-a840-3938d11d9f6f-workload-certs" pod "dd0f5b27-6383-4d55-a840-3938d11d9f6f" (UID: "dd0f5b27-6383-4d55-a840-3938d11d9f6f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:32.527996 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:32.527968 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd0f5b27-6383-4d55-a840-3938d11d9f6f-istio-envoy" pod "dd0f5b27-6383-4d55-a840-3938d11d9f6f" (UID: "dd0f5b27-6383-4d55-a840-3938d11d9f6f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:32.528261 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:32.528238 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd0f5b27-6383-4d55-a840-3938d11d9f6f-kube-api-access-4ggcx" pod "dd0f5b27-6383-4d55-a840-3938d11d9f6f" (UID: "dd0f5b27-6383-4d55-a840-3938d11d9f6f"). InnerVolumeSpecName "kube-api-access-4ggcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:32.528375 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:32.528357 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd0f5b27-6383-4d55-a840-3938d11d9f6f-istio-podinfo" pod "dd0f5b27-6383-4d55-a840-3938d11d9f6f" (UID: "dd0f5b27-6383-4d55-a840-3938d11d9f6f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:13:32.528480 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:32.528465 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd0f5b27-6383-4d55-a840-3938d11d9f6f-istio-token" pod "dd0f5b27-6383-4d55-a840-3938d11d9f6f" (UID: "dd0f5b27-6383-4d55-a840-3938d11d9f6f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:32.627015 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:32.626976 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd0f5b27-6383-4d55-a840-3938d11d9f6f-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:32.627015 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:32.627013 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ggcx\" (UniqueName: \"kubernetes.io/projected/dd0f5b27-6383-4d55-a840-3938d11d9f6f-kube-api-access-4ggcx\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:32.627015 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:32.627023 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd0f5b27-6383-4d55-a840-3938d11d9f6f-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:32.627297 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:32.627032 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd0f5b27-6383-4d55-a840-3938d11d9f6f-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:32.627297 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:32.627039 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd0f5b27-6383-4d55-a840-3938d11d9f6f-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:33.277589 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:33.277554 4442 generic.go:356] "Generic (PLEG): container finished" podID="dd0f5b27-6383-4d55-a840-3938d11d9f6f" containerID="f954150b0c9ed044391ce6a5e6a8a8c56b3fc36d4dc080f184f31b80d8777e35" exitCode=0 Oct 30 19:13:33.277746 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:33.277610 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" Oct 30 19:13:33.277746 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:33.277640 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" event={"ID":"dd0f5b27-6383-4d55-a840-3938d11d9f6f","Type":"ContainerDied","Data":"f954150b0c9ed044391ce6a5e6a8a8c56b3fc36d4dc080f184f31b80d8777e35"} Oct 30 19:13:33.277746 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:33.277694 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78" event={"ID":"dd0f5b27-6383-4d55-a840-3938d11d9f6f","Type":"ContainerDied","Data":"ab41cf59b31cad5bc16cf88362293c2a951c2cb5144ad5ad1c30c3f6a1c3e08e"} Oct 30 19:13:33.277746 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:33.277723 4442 scope.go:122] "RemoveContainer" containerID="f954150b0c9ed044391ce6a5e6a8a8c56b3fc36d4dc080f184f31b80d8777e35" Oct 30 19:13:33.289044 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:33.289025 4442 scope.go:122] "RemoveContainer" containerID="12e03a772cb15bc6cd60d5687c642999f0338add7027485fe6b82477e3cfc317" Oct 30 19:13:33.299656 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:33.299631 4442 scope.go:122] "RemoveContainer" containerID="f954150b0c9ed044391ce6a5e6a8a8c56b3fc36d4dc080f184f31b80d8777e35" Oct 30 19:13:33.299945 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:13:33.299927 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f954150b0c9ed044391ce6a5e6a8a8c56b3fc36d4dc080f184f31b80d8777e35\": container with ID starting with f954150b0c9ed044391ce6a5e6a8a8c56b3fc36d4dc080f184f31b80d8777e35 not found: ID does not exist" containerID="f954150b0c9ed044391ce6a5e6a8a8c56b3fc36d4dc080f184f31b80d8777e35" Oct 30 19:13:33.300003 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:33.299954 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f954150b0c9ed044391ce6a5e6a8a8c56b3fc36d4dc080f184f31b80d8777e35"} err="failed to get container status \"f954150b0c9ed044391ce6a5e6a8a8c56b3fc36d4dc080f184f31b80d8777e35\": rpc error: code = NotFound desc = could not find container \"f954150b0c9ed044391ce6a5e6a8a8c56b3fc36d4dc080f184f31b80d8777e35\": container with ID starting with f954150b0c9ed044391ce6a5e6a8a8c56b3fc36d4dc080f184f31b80d8777e35 not found: ID does not exist" Oct 30 19:13:33.300003 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:33.299974 4442 scope.go:122] "RemoveContainer" containerID="12e03a772cb15bc6cd60d5687c642999f0338add7027485fe6b82477e3cfc317" Oct 30 19:13:33.300273 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:13:33.300254 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12e03a772cb15bc6cd60d5687c642999f0338add7027485fe6b82477e3cfc317\": container with ID starting with 12e03a772cb15bc6cd60d5687c642999f0338add7027485fe6b82477e3cfc317 not found: ID does not exist" containerID="12e03a772cb15bc6cd60d5687c642999f0338add7027485fe6b82477e3cfc317" Oct 30 19:13:33.300321 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:33.300279 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12e03a772cb15bc6cd60d5687c642999f0338add7027485fe6b82477e3cfc317"} err="failed to get container status \"12e03a772cb15bc6cd60d5687c642999f0338add7027485fe6b82477e3cfc317\": rpc error: code = NotFound desc = could not find container \"12e03a772cb15bc6cd60d5687c642999f0338add7027485fe6b82477e3cfc317\": container with ID starting with 12e03a772cb15bc6cd60d5687c642999f0338add7027485fe6b82477e3cfc317 not found: ID does not exist" Oct 30 19:13:33.331685 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:33.331652 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78"] Oct 30 19:13:33.343394 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:33.343360 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-d1a330-openshift-default-5f66d94d4f-chz78"] Oct 30 19:13:33.508071 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:33.508038 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-8b8e54-77dc6fdfc4-j8v2w" podUID="58035120-0d0b-48c0-9096-c531d8b10f82" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:13:35.043274 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:35.043247 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn"] Oct 30 19:13:35.043765 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:35.043443 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" podUID="721069e6-60e5-4a5f-b79f-6d8b47ec87aa" containerName="istio-proxy" containerID="cri-o://cda316d7beade1b70bd6f0ae79f0b95ad8153f3eb277dfd0f0f456af5b1c1164" gracePeriod=30 Oct 30 19:13:35.064856 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:35.064824 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd0f5b27-6383-4d55-a840-3938d11d9f6f" path="/var/lib/kubelet/pods/dd0f5b27-6383-4d55-a840-3938d11d9f6f/volumes" Oct 30 19:13:35.637347 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:35.637315 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-39c99d-t5pfg"] Oct 30 19:13:35.637934 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:35.637919 4442 cpu_manager.go:414] "containerMap: removing container" podUID="dd0f5b27-6383-4d55-a840-3938d11d9f6f" containerName="istio-proxy" Oct 30 19:13:35.637988 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:35.637937 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd0f5b27-6383-4d55-a840-3938d11d9f6f" containerName="istio-proxy" Oct 30 19:13:35.637988 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:35.637966 4442 cpu_manager.go:414] "containerMap: removing container" podUID="95801f03-019f-43ce-888c-88f00884c618" containerName="revoke-keys" Oct 30 19:13:35.637988 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:35.637971 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95801f03-019f-43ce-888c-88f00884c618" containerName="revoke-keys" Oct 30 19:13:35.638082 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:35.638062 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd0f5b27-6383-4d55-a840-3938d11d9f6f" containerName="istio-proxy" Oct 30 19:13:35.638082 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:35.638075 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd0f5b27-6383-4d55-a840-3938d11d9f6f" containerName="istio-proxy" Oct 30 19:13:35.638180 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:35.638084 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="95801f03-019f-43ce-888c-88f00884c618" containerName="revoke-keys" Oct 30 19:13:35.641475 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:35.641455 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-39c99d-t5pfg" Oct 30 19:13:35.664599 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:35.664575 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-39c99d-t5pfg"] Oct 30 19:13:35.749694 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:35.749662 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:13:35.749870 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:35.749730 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:13:35.779834 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:35.779795 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-598k7\" (UniqueName: \"kubernetes.io/projected/e6bab201-d278-4e2b-a192-045b58d842ac-kube-api-access-598k7\") pod \"maas-api-revoke-keys-e2e-extmod-b-39c99d-t5pfg\" (UID: \"e6bab201-d278-4e2b-a192-045b58d842ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-39c99d-t5pfg" Oct 30 19:13:35.779834 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:35.779835 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6bab201-d278-4e2b-a192-045b58d842ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-39c99d-t5pfg\" (UID: \"e6bab201-d278-4e2b-a192-045b58d842ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-39c99d-t5pfg" Oct 30 19:13:35.780083 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:35.779978 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6bab201-d278-4e2b-a192-045b58d842ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-39c99d-t5pfg\" (UID: \"e6bab201-d278-4e2b-a192-045b58d842ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-39c99d-t5pfg" Oct 30 19:13:35.881810 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:35.881779 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-598k7\" (UniqueName: \"kubernetes.io/projected/e6bab201-d278-4e2b-a192-045b58d842ac-kube-api-access-598k7\") pod \"maas-api-revoke-keys-e2e-extmod-b-39c99d-t5pfg\" (UID: \"e6bab201-d278-4e2b-a192-045b58d842ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-39c99d-t5pfg" Oct 30 19:13:35.882017 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:35.881815 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6bab201-d278-4e2b-a192-045b58d842ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-39c99d-t5pfg\" (UID: \"e6bab201-d278-4e2b-a192-045b58d842ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-39c99d-t5pfg" Oct 30 19:13:35.882017 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:35.881874 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6bab201-d278-4e2b-a192-045b58d842ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-39c99d-t5pfg\" (UID: \"e6bab201-d278-4e2b-a192-045b58d842ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-39c99d-t5pfg" Oct 30 19:13:35.882537 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:35.882515 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6bab201-d278-4e2b-a192-045b58d842ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-39c99d-t5pfg\" (UID: \"e6bab201-d278-4e2b-a192-045b58d842ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-39c99d-t5pfg" Oct 30 19:13:35.884513 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:35.884490 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6bab201-d278-4e2b-a192-045b58d842ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-39c99d-t5pfg\" (UID: \"e6bab201-d278-4e2b-a192-045b58d842ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-39c99d-t5pfg" Oct 30 19:13:35.903723 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:35.903659 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-598k7\" (UniqueName: \"kubernetes.io/projected/e6bab201-d278-4e2b-a192-045b58d842ac-kube-api-access-598k7\") pod \"maas-api-revoke-keys-e2e-extmod-b-39c99d-t5pfg\" (UID: \"e6bab201-d278-4e2b-a192-045b58d842ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-39c99d-t5pfg" Oct 30 19:13:35.954673 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:35.954644 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-39c99d-t5pfg" Oct 30 19:13:36.255420 ip-10-0-138-176 kubenswrapper[4442]: W1030 19:13:36.255371 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6bab201_d278_4e2b_a192_045b58d842ac.slice/crio-80138ce8a0acb608175f75441930416b6a648c6e3b948e86be94f35cd874f297 WatchSource:0}: Error finding container 80138ce8a0acb608175f75441930416b6a648c6e3b948e86be94f35cd874f297: Status 404 returned error can't find the container with id 80138ce8a0acb608175f75441930416b6a648c6e3b948e86be94f35cd874f297 Oct 30 19:13:36.259298 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:36.259271 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-39c99d-t5pfg"] Oct 30 19:13:36.302673 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:36.302422 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-39c99d-t5pfg" event={"ID":"e6bab201-d278-4e2b-a192-045b58d842ac","Type":"ContainerStarted","Data":"80138ce8a0acb608175f75441930416b6a648c6e3b948e86be94f35cd874f297"} Oct 30 19:13:37.311344 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:37.311308 4442 generic.go:356] "Generic (PLEG): container finished" podID="e6bab201-d278-4e2b-a192-045b58d842ac" containerID="34294a2df6c1a4fe7dab482bc0ee8421ca6920ff02148e033cc203b2c1a4fbfb" exitCode=0 Oct 30 19:13:37.311818 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:37.311395 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-39c99d-t5pfg" event={"ID":"e6bab201-d278-4e2b-a192-045b58d842ac","Type":"ContainerDied","Data":"34294a2df6c1a4fe7dab482bc0ee8421ca6920ff02148e033cc203b2c1a4fbfb"} Oct 30 19:13:38.259378 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:38.259343 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6babca-6dc6645cb4-jw8l5"] Oct 30 19:13:38.536228 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:38.536191 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-39c99d-t5pfg" Oct 30 19:13:38.615579 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:38.615545 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6bab201-d278-4e2b-a192-045b58d842ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6bab201-d278-4e2b-a192-045b58d842ac-maas-api-openshift-ca\") pod \"e6bab201-d278-4e2b-a192-045b58d842ac\" (UID: \"e6bab201-d278-4e2b-a192-045b58d842ac\") " Oct 30 19:13:38.615790 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:38.615744 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6bab201-d278-4e2b-a192-045b58d842ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6bab201-d278-4e2b-a192-045b58d842ac-maas-api-xks-ca\") pod \"e6bab201-d278-4e2b-a192-045b58d842ac\" (UID: \"e6bab201-d278-4e2b-a192-045b58d842ac\") " Oct 30 19:13:38.615790 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:38.615787 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6bab201-d278-4e2b-a192-045b58d842ac-kube-api-access-598k7\" (UniqueName: \"kubernetes.io/projected/e6bab201-d278-4e2b-a192-045b58d842ac-kube-api-access-598k7\") pod \"e6bab201-d278-4e2b-a192-045b58d842ac\" (UID: \"e6bab201-d278-4e2b-a192-045b58d842ac\") " Oct 30 19:13:38.616037 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:38.616008 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6bab201-d278-4e2b-a192-045b58d842ac-maas-api-openshift-ca" pod "e6bab201-d278-4e2b-a192-045b58d842ac" (UID: "e6bab201-d278-4e2b-a192-045b58d842ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:13:38.616298 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:38.616276 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6bab201-d278-4e2b-a192-045b58d842ac-maas-api-openshift-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:38.618146 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:38.618097 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6bab201-d278-4e2b-a192-045b58d842ac-maas-api-xks-ca" pod "e6bab201-d278-4e2b-a192-045b58d842ac" (UID: "e6bab201-d278-4e2b-a192-045b58d842ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:13:38.618247 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:38.618146 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6bab201-d278-4e2b-a192-045b58d842ac-kube-api-access-598k7" pod "e6bab201-d278-4e2b-a192-045b58d842ac" (UID: "e6bab201-d278-4e2b-a192-045b58d842ac"). InnerVolumeSpecName "kube-api-access-598k7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:38.729890 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:38.729022 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6bab201-d278-4e2b-a192-045b58d842ac-maas-api-xks-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:38.729890 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:38.729064 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-598k7\" (UniqueName: \"kubernetes.io/projected/e6bab201-d278-4e2b-a192-045b58d842ac-kube-api-access-598k7\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:39.214643 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:39.214610 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8b8e54-f8mqf"] Oct 30 19:13:39.227627 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:39.227596 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8b8e54-f8mqf"] Oct 30 19:13:39.330204 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:39.330070 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-39c99d-t5pfg" event={"ID":"e6bab201-d278-4e2b-a192-045b58d842ac","Type":"ContainerDied","Data":"80138ce8a0acb608175f75441930416b6a648c6e3b948e86be94f35cd874f297"} Oct 30 19:13:39.330363 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:39.330209 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80138ce8a0acb608175f75441930416b6a648c6e3b948e86be94f35cd874f297" Oct 30 19:13:39.330363 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:39.330091 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-39c99d-t5pfg" Oct 30 19:13:40.311736 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:40.311711 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" Oct 30 19:13:40.341146 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:40.341036 4442 generic.go:356] "Generic (PLEG): container finished" podID="721069e6-60e5-4a5f-b79f-6d8b47ec87aa" containerID="cda316d7beade1b70bd6f0ae79f0b95ad8153f3eb277dfd0f0f456af5b1c1164" exitCode=0 Oct 30 19:13:40.341146 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:40.341078 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" event={"ID":"721069e6-60e5-4a5f-b79f-6d8b47ec87aa","Type":"ContainerDied","Data":"cda316d7beade1b70bd6f0ae79f0b95ad8153f3eb277dfd0f0f456af5b1c1164"} Oct 30 19:13:40.341370 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:40.341143 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" Oct 30 19:13:40.341370 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:40.341178 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn" event={"ID":"721069e6-60e5-4a5f-b79f-6d8b47ec87aa","Type":"ContainerDied","Data":"60a0773e535849461bf30cdab8040be5bf73bc449734257a5988d23bb94680fb"} Oct 30 19:13:40.341370 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:40.341213 4442 scope.go:122] "RemoveContainer" containerID="cda316d7beade1b70bd6f0ae79f0b95ad8153f3eb277dfd0f0f456af5b1c1164" Oct 30 19:13:40.354571 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:40.354550 4442 scope.go:122] "RemoveContainer" containerID="3cf7b8b6065b5f84a4d079839b2ba4f271c173797f04751f9a02f4a64b42a53f" Oct 30 19:13:40.366991 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:40.366972 4442 scope.go:122] "RemoveContainer" containerID="cda316d7beade1b70bd6f0ae79f0b95ad8153f3eb277dfd0f0f456af5b1c1164" Oct 30 19:13:40.367371 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:13:40.367344 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cda316d7beade1b70bd6f0ae79f0b95ad8153f3eb277dfd0f0f456af5b1c1164\": container with ID starting with cda316d7beade1b70bd6f0ae79f0b95ad8153f3eb277dfd0f0f456af5b1c1164 not found: ID does not exist" containerID="cda316d7beade1b70bd6f0ae79f0b95ad8153f3eb277dfd0f0f456af5b1c1164" Oct 30 19:13:40.367434 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:40.367384 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cda316d7beade1b70bd6f0ae79f0b95ad8153f3eb277dfd0f0f456af5b1c1164"} err="failed to get container status \"cda316d7beade1b70bd6f0ae79f0b95ad8153f3eb277dfd0f0f456af5b1c1164\": rpc error: code = NotFound desc = could not find container \"cda316d7beade1b70bd6f0ae79f0b95ad8153f3eb277dfd0f0f456af5b1c1164\": container with ID starting with cda316d7beade1b70bd6f0ae79f0b95ad8153f3eb277dfd0f0f456af5b1c1164 not found: ID does not exist" Oct 30 19:13:40.367434 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:40.367410 4442 scope.go:122] "RemoveContainer" containerID="3cf7b8b6065b5f84a4d079839b2ba4f271c173797f04751f9a02f4a64b42a53f" Oct 30 19:13:40.367699 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:13:40.367677 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cf7b8b6065b5f84a4d079839b2ba4f271c173797f04751f9a02f4a64b42a53f\": container with ID starting with 3cf7b8b6065b5f84a4d079839b2ba4f271c173797f04751f9a02f4a64b42a53f not found: ID does not exist" containerID="3cf7b8b6065b5f84a4d079839b2ba4f271c173797f04751f9a02f4a64b42a53f" Oct 30 19:13:40.367841 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:40.367701 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cf7b8b6065b5f84a4d079839b2ba4f271c173797f04751f9a02f4a64b42a53f"} err="failed to get container status \"3cf7b8b6065b5f84a4d079839b2ba4f271c173797f04751f9a02f4a64b42a53f\": rpc error: code = NotFound desc = could not find container \"3cf7b8b6065b5f84a4d079839b2ba4f271c173797f04751f9a02f4a64b42a53f\": container with ID starting with 3cf7b8b6065b5f84a4d079839b2ba4f271c173797f04751f9a02f4a64b42a53f not found: ID does not exist" Oct 30 19:13:40.456899 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:40.456853 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-istio-token\" (UniqueName: \"kubernetes.io/projected/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-istio-token\") pod \"721069e6-60e5-4a5f-b79f-6d8b47ec87aa\" (UID: \"721069e6-60e5-4a5f-b79f-6d8b47ec87aa\") " Oct 30 19:13:40.457103 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:40.456913 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-workload-socket\") pod \"721069e6-60e5-4a5f-b79f-6d8b47ec87aa\" (UID: \"721069e6-60e5-4a5f-b79f-6d8b47ec87aa\") " Oct 30 19:13:40.457103 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:40.457024 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-credential-socket\") pod \"721069e6-60e5-4a5f-b79f-6d8b47ec87aa\" (UID: \"721069e6-60e5-4a5f-b79f-6d8b47ec87aa\") " Oct 30 19:13:40.457272 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:40.457120 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-kube-api-access-tt79h\" (UniqueName: \"kubernetes.io/projected/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-kube-api-access-tt79h\") pod \"721069e6-60e5-4a5f-b79f-6d8b47ec87aa\" (UID: \"721069e6-60e5-4a5f-b79f-6d8b47ec87aa\") " Oct 30 19:13:40.457272 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:40.457212 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-istio-data\") pod \"721069e6-60e5-4a5f-b79f-6d8b47ec87aa\" (UID: \"721069e6-60e5-4a5f-b79f-6d8b47ec87aa\") " Oct 30 19:13:40.457272 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:40.457266 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-workload-certs\") pod \"721069e6-60e5-4a5f-b79f-6d8b47ec87aa\" (UID: \"721069e6-60e5-4a5f-b79f-6d8b47ec87aa\") " Oct 30 19:13:40.457452 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:40.457273 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-workload-socket" pod "721069e6-60e5-4a5f-b79f-6d8b47ec87aa" (UID: "721069e6-60e5-4a5f-b79f-6d8b47ec87aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:40.457452 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:40.457305 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-istio-podinfo\") pod \"721069e6-60e5-4a5f-b79f-6d8b47ec87aa\" (UID: \"721069e6-60e5-4a5f-b79f-6d8b47ec87aa\") " Oct 30 19:13:40.457452 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:40.457344 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-istio-envoy\") pod \"721069e6-60e5-4a5f-b79f-6d8b47ec87aa\" (UID: \"721069e6-60e5-4a5f-b79f-6d8b47ec87aa\") " Oct 30 19:13:40.457452 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:40.457376 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-credential-socket" pod "721069e6-60e5-4a5f-b79f-6d8b47ec87aa" (UID: "721069e6-60e5-4a5f-b79f-6d8b47ec87aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:40.457452 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:40.457420 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-istiod-ca-cert\") pod \"721069e6-60e5-4a5f-b79f-6d8b47ec87aa\" (UID: \"721069e6-60e5-4a5f-b79f-6d8b47ec87aa\") " Oct 30 19:13:40.457723 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:40.457514 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-istio-data" pod "721069e6-60e5-4a5f-b79f-6d8b47ec87aa" (UID: "721069e6-60e5-4a5f-b79f-6d8b47ec87aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:40.457723 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:40.457586 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-workload-certs" pod "721069e6-60e5-4a5f-b79f-6d8b47ec87aa" (UID: "721069e6-60e5-4a5f-b79f-6d8b47ec87aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:40.458022 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:40.457985 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-istiod-ca-cert" pod "721069e6-60e5-4a5f-b79f-6d8b47ec87aa" (UID: "721069e6-60e5-4a5f-b79f-6d8b47ec87aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:13:40.473164 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:40.466832 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-kube-api-access-tt79h" pod "721069e6-60e5-4a5f-b79f-6d8b47ec87aa" (UID: "721069e6-60e5-4a5f-b79f-6d8b47ec87aa"). InnerVolumeSpecName "kube-api-access-tt79h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:40.473164 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:40.466862 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-istio-token" pod "721069e6-60e5-4a5f-b79f-6d8b47ec87aa" (UID: "721069e6-60e5-4a5f-b79f-6d8b47ec87aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:40.473164 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:40.458478 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:40.473164 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:40.468343 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:40.473164 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:40.468362 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:40.473164 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:40.468381 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:40.473164 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:40.468404 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:40.473164 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:40.472050 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-istio-envoy" pod "721069e6-60e5-4a5f-b79f-6d8b47ec87aa" (UID: "721069e6-60e5-4a5f-b79f-6d8b47ec87aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:40.474838 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:40.474404 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-istio-podinfo" pod "721069e6-60e5-4a5f-b79f-6d8b47ec87aa" (UID: "721069e6-60e5-4a5f-b79f-6d8b47ec87aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:13:40.569969 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:40.569934 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt79h\" (UniqueName: \"kubernetes.io/projected/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-kube-api-access-tt79h\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:40.569969 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:40.569971 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:40.570198 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:40.569985 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:40.570198 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:40.569998 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/721069e6-60e5-4a5f-b79f-6d8b47ec87aa-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:40.687527 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:40.687489 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn"] Oct 30 19:13:40.699292 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:40.699254 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-713dc8-openshift-default-84c5dbdfb9-lqbhn"] Oct 30 19:13:41.066295 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:41.066215 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="721069e6-60e5-4a5f-b79f-6d8b47ec87aa" path="/var/lib/kubelet/pods/721069e6-60e5-4a5f-b79f-6d8b47ec87aa/volumes" Oct 30 19:13:41.066921 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:41.066899 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb" path="/var/lib/kubelet/pods/d4d6bfbe-8c6b-46d7-be96-a76ac188d1fb/volumes" Oct 30 19:13:43.260396 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:43.260355 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-6babca-6dc6645cb4-jw8l5" podUID="0c64d72e-b778-40b9-aeb7-6b7cfad39907" containerName="payload-processing" containerID="cri-o://8c9c3cdd8114253edf129d148611d17da392b74056ec048786ac0ea25e4b49b0" gracePeriod=25 Oct 30 19:13:43.507623 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:43.507586 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-8b8e54-77dc6fdfc4-j8v2w" podUID="58035120-0d0b-48c0-9096-c531d8b10f82" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:13:45.687459 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:45.687396 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-8b8e54-77dc6fdfc4-j8v2w" Oct 30 19:13:45.747754 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:45.747724 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58035120-0d0b-48c0-9096-c531d8b10f82-kube-api-access-p99lb\" (UniqueName: \"kubernetes.io/projected/58035120-0d0b-48c0-9096-c531d8b10f82-kube-api-access-p99lb\") pod \"58035120-0d0b-48c0-9096-c531d8b10f82\" (UID: \"58035120-0d0b-48c0-9096-c531d8b10f82\") " Oct 30 19:13:45.747949 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:45.747779 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58035120-0d0b-48c0-9096-c531d8b10f82-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58035120-0d0b-48c0-9096-c531d8b10f82-plugins-config-volume\") pod \"58035120-0d0b-48c0-9096-c531d8b10f82\" (UID: \"58035120-0d0b-48c0-9096-c531d8b10f82\") " Oct 30 19:13:45.748574 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:45.748545 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58035120-0d0b-48c0-9096-c531d8b10f82-plugins-config-volume" pod "58035120-0d0b-48c0-9096-c531d8b10f82" (UID: "58035120-0d0b-48c0-9096-c531d8b10f82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:13:45.751033 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:45.751004 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58035120-0d0b-48c0-9096-c531d8b10f82-kube-api-access-p99lb" pod "58035120-0d0b-48c0-9096-c531d8b10f82" (UID: "58035120-0d0b-48c0-9096-c531d8b10f82"). InnerVolumeSpecName "kube-api-access-p99lb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:45.850071 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:45.850042 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p99lb\" (UniqueName: \"kubernetes.io/projected/58035120-0d0b-48c0-9096-c531d8b10f82-kube-api-access-p99lb\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:45.850071 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:45.850068 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58035120-0d0b-48c0-9096-c531d8b10f82-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:13:46.066785 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:46.066682 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-6babca-6dc6645cb4-jw8l5" podUID="0c64d72e-b778-40b9-aeb7-6b7cfad39907" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:13:46.393259 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:46.393220 4442 generic.go:356] "Generic (PLEG): container finished" podID="58035120-0d0b-48c0-9096-c531d8b10f82" containerID="a4ad388a3031f8871663c467537bb09ad22349a65b035c4109a04828a9598ea4" exitCode=0 Oct 30 19:13:46.393484 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:46.393264 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-8b8e54-77dc6fdfc4-j8v2w" Oct 30 19:13:46.393484 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:46.393288 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-8b8e54-77dc6fdfc4-j8v2w" event={"ID":"58035120-0d0b-48c0-9096-c531d8b10f82","Type":"ContainerDied","Data":"a4ad388a3031f8871663c467537bb09ad22349a65b035c4109a04828a9598ea4"} Oct 30 19:13:46.393484 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:46.393341 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-8b8e54-77dc6fdfc4-j8v2w" event={"ID":"58035120-0d0b-48c0-9096-c531d8b10f82","Type":"ContainerDied","Data":"b4bf0c355707486f1cd688921f426a9afcc81528ee8e64fc0a507454ec469ffb"} Oct 30 19:13:46.393484 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:46.393371 4442 scope.go:122] "RemoveContainer" containerID="a4ad388a3031f8871663c467537bb09ad22349a65b035c4109a04828a9598ea4" Oct 30 19:13:46.407090 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:46.407073 4442 scope.go:122] "RemoveContainer" containerID="a4ad388a3031f8871663c467537bb09ad22349a65b035c4109a04828a9598ea4" Oct 30 19:13:46.407444 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:13:46.407419 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4ad388a3031f8871663c467537bb09ad22349a65b035c4109a04828a9598ea4\": container with ID starting with a4ad388a3031f8871663c467537bb09ad22349a65b035c4109a04828a9598ea4 not found: ID does not exist" containerID="a4ad388a3031f8871663c467537bb09ad22349a65b035c4109a04828a9598ea4" Oct 30 19:13:46.407543 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:46.407451 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4ad388a3031f8871663c467537bb09ad22349a65b035c4109a04828a9598ea4"} err="failed to get container status \"a4ad388a3031f8871663c467537bb09ad22349a65b035c4109a04828a9598ea4\": rpc error: code = NotFound desc = could not find container \"a4ad388a3031f8871663c467537bb09ad22349a65b035c4109a04828a9598ea4\": container with ID starting with a4ad388a3031f8871663c467537bb09ad22349a65b035c4109a04828a9598ea4 not found: ID does not exist" Oct 30 19:13:46.450249 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:46.450220 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-8b8e54-77dc6fdfc4-j8v2w"] Oct 30 19:13:46.461516 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:46.461491 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-8b8e54-77dc6fdfc4-j8v2w"] Oct 30 19:13:47.064671 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:47.064629 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58035120-0d0b-48c0-9096-c531d8b10f82" path="/var/lib/kubelet/pods/58035120-0d0b-48c0-9096-c531d8b10f82/volumes" Oct 30 19:13:47.560572 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:13:47.560532 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 19:13:52.362460 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:52.362425 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6babca-8krfd"] Oct 30 19:13:52.372173 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:52.372102 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6babca-8krfd"] Oct 30 19:13:52.955771 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:52.948033 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-39c99d-t5pfg"] Oct 30 19:13:52.955771 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:52.955407 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-39c99d-t5pfg"] Oct 30 19:13:53.083695 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:53.083665 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95801f03-019f-43ce-888c-88f00884c618" path="/var/lib/kubelet/pods/95801f03-019f-43ce-888c-88f00884c618/volumes" Oct 30 19:13:53.084100 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:53.084086 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6bab201-d278-4e2b-a192-045b58d842ac" path="/var/lib/kubelet/pods/e6bab201-d278-4e2b-a192-045b58d842ac/volumes" Oct 30 19:13:56.065886 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:13:56.065849 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-6babca-6dc6645cb4-jw8l5" podUID="0c64d72e-b778-40b9-aeb7-6b7cfad39907" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:14:03.540749 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:03.540720 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-6babca-6dc6645cb4-jw8l5" Oct 30 19:14:03.608386 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:03.608343 4442 generic.go:356] "Generic (PLEG): container finished" podID="0c64d72e-b778-40b9-aeb7-6b7cfad39907" containerID="8c9c3cdd8114253edf129d148611d17da392b74056ec048786ac0ea25e4b49b0" exitCode=0 Oct 30 19:14:03.608583 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:03.608398 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-6babca-6dc6645cb4-jw8l5" Oct 30 19:14:03.608583 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:03.608422 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-6babca-6dc6645cb4-jw8l5" event={"ID":"0c64d72e-b778-40b9-aeb7-6b7cfad39907","Type":"ContainerDied","Data":"8c9c3cdd8114253edf129d148611d17da392b74056ec048786ac0ea25e4b49b0"} Oct 30 19:14:03.608583 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:03.608487 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-6babca-6dc6645cb4-jw8l5" event={"ID":"0c64d72e-b778-40b9-aeb7-6b7cfad39907","Type":"ContainerDied","Data":"410520d2e903439b28151a26e9dd24f27e27514d8e887d8ae4d070b5789ab7a9"} Oct 30 19:14:03.608583 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:03.608515 4442 scope.go:122] "RemoveContainer" containerID="8c9c3cdd8114253edf129d148611d17da392b74056ec048786ac0ea25e4b49b0" Oct 30 19:14:03.615001 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:03.614979 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c64d72e-b778-40b9-aeb7-6b7cfad39907-kube-api-access-s9qpt\" (UniqueName: \"kubernetes.io/projected/0c64d72e-b778-40b9-aeb7-6b7cfad39907-kube-api-access-s9qpt\") pod \"0c64d72e-b778-40b9-aeb7-6b7cfad39907\" (UID: \"0c64d72e-b778-40b9-aeb7-6b7cfad39907\") " Oct 30 19:14:03.615195 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:03.615027 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c64d72e-b778-40b9-aeb7-6b7cfad39907-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c64d72e-b778-40b9-aeb7-6b7cfad39907-plugins-config-volume\") pod \"0c64d72e-b778-40b9-aeb7-6b7cfad39907\" (UID: \"0c64d72e-b778-40b9-aeb7-6b7cfad39907\") " Oct 30 19:14:03.615445 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:03.615423 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c64d72e-b778-40b9-aeb7-6b7cfad39907-plugins-config-volume" pod "0c64d72e-b778-40b9-aeb7-6b7cfad39907" (UID: "0c64d72e-b778-40b9-aeb7-6b7cfad39907"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:14:03.615548 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:03.615534 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c64d72e-b778-40b9-aeb7-6b7cfad39907-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:03.617431 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:03.617389 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c64d72e-b778-40b9-aeb7-6b7cfad39907-kube-api-access-s9qpt" pod "0c64d72e-b778-40b9-aeb7-6b7cfad39907" (UID: "0c64d72e-b778-40b9-aeb7-6b7cfad39907"). InnerVolumeSpecName "kube-api-access-s9qpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:03.621953 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:03.621934 4442 scope.go:122] "RemoveContainer" containerID="8c9c3cdd8114253edf129d148611d17da392b74056ec048786ac0ea25e4b49b0" Oct 30 19:14:03.622285 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:14:03.622258 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c9c3cdd8114253edf129d148611d17da392b74056ec048786ac0ea25e4b49b0\": container with ID starting with 8c9c3cdd8114253edf129d148611d17da392b74056ec048786ac0ea25e4b49b0 not found: ID does not exist" containerID="8c9c3cdd8114253edf129d148611d17da392b74056ec048786ac0ea25e4b49b0" Oct 30 19:14:03.622285 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:03.622292 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c9c3cdd8114253edf129d148611d17da392b74056ec048786ac0ea25e4b49b0"} err="failed to get container status \"8c9c3cdd8114253edf129d148611d17da392b74056ec048786ac0ea25e4b49b0\": rpc error: code = NotFound desc = could not find container \"8c9c3cdd8114253edf129d148611d17da392b74056ec048786ac0ea25e4b49b0\": container with ID starting with 8c9c3cdd8114253edf129d148611d17da392b74056ec048786ac0ea25e4b49b0 not found: ID does not exist" Oct 30 19:14:03.716959 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:03.716928 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9qpt\" (UniqueName: \"kubernetes.io/projected/0c64d72e-b778-40b9-aeb7-6b7cfad39907-kube-api-access-s9qpt\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:03.955982 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:03.955947 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6babca-6dc6645cb4-jw8l5"] Oct 30 19:14:03.961430 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:03.961394 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6babca-6dc6645cb4-jw8l5"] Oct 30 19:14:05.064368 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:05.064291 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c64d72e-b778-40b9-aeb7-6b7cfad39907" path="/var/lib/kubelet/pods/0c64d72e-b778-40b9-aeb7-6b7cfad39907/volumes" Oct 30 19:14:08.278989 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:08.278955 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1a2436-7676957859-jxlmq"] Oct 30 19:14:09.534428 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:09.534385 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm"] Oct 30 19:14:09.534943 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:09.534663 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" podUID="57bae3b7-f320-45e8-bdff-e41abbb9f20c" containerName="istio-proxy" containerID="cri-o://ad46a731672326b4c762008c36d30c8e288327b2cb72e22e9d5b070da3842a39" gracePeriod=30 Oct 30 19:14:10.237359 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:10.237325 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-675070-n64xj"] Oct 30 19:14:10.237860 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:10.237847 4442 cpu_manager.go:414] "containerMap: removing container" podUID="721069e6-60e5-4a5f-b79f-6d8b47ec87aa" containerName="istio-proxy" Oct 30 19:14:10.237907 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:10.237862 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="721069e6-60e5-4a5f-b79f-6d8b47ec87aa" containerName="istio-proxy" Oct 30 19:14:10.237907 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:10.237878 4442 cpu_manager.go:414] "containerMap: removing container" podUID="58035120-0d0b-48c0-9096-c531d8b10f82" containerName="payload-processing" Oct 30 19:14:10.237907 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:10.237883 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58035120-0d0b-48c0-9096-c531d8b10f82" containerName="payload-processing" Oct 30 19:14:10.237907 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:10.237890 4442 cpu_manager.go:414] "containerMap: removing container" podUID="721069e6-60e5-4a5f-b79f-6d8b47ec87aa" containerName="istio-proxy" Oct 30 19:14:10.237907 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:10.237895 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="721069e6-60e5-4a5f-b79f-6d8b47ec87aa" containerName="istio-proxy" Oct 30 19:14:10.237907 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:10.237903 4442 cpu_manager.go:414] "containerMap: removing container" podUID="dd0f5b27-6383-4d55-a840-3938d11d9f6f" containerName="istio-proxy" Oct 30 19:14:10.237907 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:10.237908 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd0f5b27-6383-4d55-a840-3938d11d9f6f" containerName="istio-proxy" Oct 30 19:14:10.238146 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:10.237916 4442 cpu_manager.go:414] "containerMap: removing container" podUID="0c64d72e-b778-40b9-aeb7-6b7cfad39907" containerName="payload-processing" Oct 30 19:14:10.238146 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:10.237921 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c64d72e-b778-40b9-aeb7-6b7cfad39907" containerName="payload-processing" Oct 30 19:14:10.238146 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:10.237936 4442 cpu_manager.go:414] "containerMap: removing container" podUID="e6bab201-d278-4e2b-a192-045b58d842ac" containerName="revoke-keys" Oct 30 19:14:10.238146 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:10.237941 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6bab201-d278-4e2b-a192-045b58d842ac" containerName="revoke-keys" Oct 30 19:14:10.238146 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:10.238022 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="58035120-0d0b-48c0-9096-c531d8b10f82" containerName="payload-processing" Oct 30 19:14:10.238146 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:10.238029 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="721069e6-60e5-4a5f-b79f-6d8b47ec87aa" containerName="istio-proxy" Oct 30 19:14:10.238146 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:10.238038 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c64d72e-b778-40b9-aeb7-6b7cfad39907" containerName="payload-processing" Oct 30 19:14:10.238146 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:10.238046 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6bab201-d278-4e2b-a192-045b58d842ac" containerName="revoke-keys" Oct 30 19:14:10.243218 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:10.243193 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-675070-n64xj" Oct 30 19:14:10.261191 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:10.261161 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-675070-n64xj"] Oct 30 19:14:10.268905 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:10.268883 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:14:10.269190 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:10.269052 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:14:10.317094 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:10.317065 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6b36d92-196f-4a2b-b312-9fc0d6260e58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-675070-n64xj\" (UID: \"c6b36d92-196f-4a2b-b312-9fc0d6260e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-675070-n64xj" Oct 30 19:14:10.317330 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:10.317163 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6b36d92-196f-4a2b-b312-9fc0d6260e58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-675070-n64xj\" (UID: \"c6b36d92-196f-4a2b-b312-9fc0d6260e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-675070-n64xj" Oct 30 19:14:10.317330 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:10.317303 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rdfv\" (UniqueName: \"kubernetes.io/projected/c6b36d92-196f-4a2b-b312-9fc0d6260e58-kube-api-access-9rdfv\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-675070-n64xj\" (UID: \"c6b36d92-196f-4a2b-b312-9fc0d6260e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-675070-n64xj" Oct 30 19:14:10.418336 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:10.418298 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rdfv\" (UniqueName: \"kubernetes.io/projected/c6b36d92-196f-4a2b-b312-9fc0d6260e58-kube-api-access-9rdfv\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-675070-n64xj\" (UID: \"c6b36d92-196f-4a2b-b312-9fc0d6260e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-675070-n64xj" Oct 30 19:14:10.418518 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:10.418391 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6b36d92-196f-4a2b-b312-9fc0d6260e58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-675070-n64xj\" (UID: \"c6b36d92-196f-4a2b-b312-9fc0d6260e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-675070-n64xj" Oct 30 19:14:10.418518 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:10.418440 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6b36d92-196f-4a2b-b312-9fc0d6260e58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-675070-n64xj\" (UID: \"c6b36d92-196f-4a2b-b312-9fc0d6260e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-675070-n64xj" Oct 30 19:14:10.419055 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:10.419032 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6b36d92-196f-4a2b-b312-9fc0d6260e58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-675070-n64xj\" (UID: \"c6b36d92-196f-4a2b-b312-9fc0d6260e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-675070-n64xj" Oct 30 19:14:10.421126 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:10.421097 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6b36d92-196f-4a2b-b312-9fc0d6260e58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-675070-n64xj\" (UID: \"c6b36d92-196f-4a2b-b312-9fc0d6260e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-675070-n64xj" Oct 30 19:14:10.441154 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:10.441107 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rdfv\" (UniqueName: \"kubernetes.io/projected/c6b36d92-196f-4a2b-b312-9fc0d6260e58-kube-api-access-9rdfv\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-675070-n64xj\" (UID: \"c6b36d92-196f-4a2b-b312-9fc0d6260e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-675070-n64xj" Oct 30 19:14:10.558317 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:10.558236 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-675070-n64xj" Oct 30 19:14:10.862623 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:10.862599 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-675070-n64xj"] Oct 30 19:14:10.864065 ip-10-0-138-176 kubenswrapper[4442]: W1030 19:14:10.864040 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6b36d92_196f_4a2b_b312_9fc0d6260e58.slice/crio-bbd2e3545938fd150f5671c76fc3af6292672d89bbc8e9d1a306d14ea70461ac WatchSource:0}: Error finding container bbd2e3545938fd150f5671c76fc3af6292672d89bbc8e9d1a306d14ea70461ac: Status 404 returned error can't find the container with id bbd2e3545938fd150f5671c76fc3af6292672d89bbc8e9d1a306d14ea70461ac Oct 30 19:14:11.699122 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:11.699079 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-675070-n64xj" event={"ID":"c6b36d92-196f-4a2b-b312-9fc0d6260e58","Type":"ContainerStarted","Data":"eed1f500b35388b88f43b7abff030f8b2f7520c101ea7c0c3a09df20981103ef"} Oct 30 19:14:11.699574 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:11.699168 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-675070-n64xj" event={"ID":"c6b36d92-196f-4a2b-b312-9fc0d6260e58","Type":"ContainerStarted","Data":"bbd2e3545938fd150f5671c76fc3af6292672d89bbc8e9d1a306d14ea70461ac"} Oct 30 19:14:11.737415 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:11.737364 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-675070-n64xj" podStartSLOduration=1.737351106 podStartE2EDuration="1.737351106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:14:11.736662106 +0000 UTC m=+4548.826724445" watchObservedRunningTime="2026-10-30 19:14:11.737351106 +0000 UTC m=+4548.827413444" Oct 30 19:14:12.708504 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:12.708469 4442 generic.go:356] "Generic (PLEG): container finished" podID="c6b36d92-196f-4a2b-b312-9fc0d6260e58" containerID="eed1f500b35388b88f43b7abff030f8b2f7520c101ea7c0c3a09df20981103ef" exitCode=0 Oct 30 19:14:12.708896 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:12.708557 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-675070-n64xj" event={"ID":"c6b36d92-196f-4a2b-b312-9fc0d6260e58","Type":"ContainerDied","Data":"eed1f500b35388b88f43b7abff030f8b2f7520c101ea7c0c3a09df20981103ef"} Oct 30 19:14:13.280297 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:13.280243 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1a2436-7676957859-jxlmq" podUID="0249c5c5-6032-425a-9107-2526fd89627a" containerName="payload-pre-processing" containerID="cri-o://5b6d407866c8208622812345b02af5516e616c559b0fce5ccf46501bf7ee5d1f" gracePeriod=25 Oct 30 19:14:13.873402 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:13.873375 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-675070-n64xj" Oct 30 19:14:13.970562 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:13.970479 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6b36d92-196f-4a2b-b312-9fc0d6260e58-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6b36d92-196f-4a2b-b312-9fc0d6260e58-maas-api-xks-ca\") pod \"c6b36d92-196f-4a2b-b312-9fc0d6260e58\" (UID: \"c6b36d92-196f-4a2b-b312-9fc0d6260e58\") " Oct 30 19:14:13.970740 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:13.970614 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6b36d92-196f-4a2b-b312-9fc0d6260e58-kube-api-access-9rdfv\" (UniqueName: \"kubernetes.io/projected/c6b36d92-196f-4a2b-b312-9fc0d6260e58-kube-api-access-9rdfv\") pod \"c6b36d92-196f-4a2b-b312-9fc0d6260e58\" (UID: \"c6b36d92-196f-4a2b-b312-9fc0d6260e58\") " Oct 30 19:14:13.970740 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:13.970706 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6b36d92-196f-4a2b-b312-9fc0d6260e58-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6b36d92-196f-4a2b-b312-9fc0d6260e58-maas-api-openshift-ca\") pod \"c6b36d92-196f-4a2b-b312-9fc0d6260e58\" (UID: \"c6b36d92-196f-4a2b-b312-9fc0d6260e58\") " Oct 30 19:14:13.971372 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:13.971330 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6b36d92-196f-4a2b-b312-9fc0d6260e58-maas-api-openshift-ca" pod "c6b36d92-196f-4a2b-b312-9fc0d6260e58" (UID: "c6b36d92-196f-4a2b-b312-9fc0d6260e58"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:14:13.973515 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:13.973487 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6b36d92-196f-4a2b-b312-9fc0d6260e58-maas-api-xks-ca" pod "c6b36d92-196f-4a2b-b312-9fc0d6260e58" (UID: "c6b36d92-196f-4a2b-b312-9fc0d6260e58"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:14:13.974258 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:13.974234 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6b36d92-196f-4a2b-b312-9fc0d6260e58-kube-api-access-9rdfv" pod "c6b36d92-196f-4a2b-b312-9fc0d6260e58" (UID: "c6b36d92-196f-4a2b-b312-9fc0d6260e58"). InnerVolumeSpecName "kube-api-access-9rdfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:14.072288 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:14.072258 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rdfv\" (UniqueName: \"kubernetes.io/projected/c6b36d92-196f-4a2b-b312-9fc0d6260e58-kube-api-access-9rdfv\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:14.072288 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:14.072285 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6b36d92-196f-4a2b-b312-9fc0d6260e58-maas-api-openshift-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:14.072288 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:14.072295 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6b36d92-196f-4a2b-b312-9fc0d6260e58-maas-api-xks-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:14.489286 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:14.489251 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1a2436-7676957859-jxlmq" podUID="0249c5c5-6032-425a-9107-2526fd89627a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:14:14.725086 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:14.725060 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-675070-n64xj" Oct 30 19:14:14.725295 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:14.725101 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-675070-n64xj" event={"ID":"c6b36d92-196f-4a2b-b312-9fc0d6260e58","Type":"ContainerDied","Data":"bbd2e3545938fd150f5671c76fc3af6292672d89bbc8e9d1a306d14ea70461ac"} Oct 30 19:14:14.725295 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:14.725165 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbd2e3545938fd150f5671c76fc3af6292672d89bbc8e9d1a306d14ea70461ac" Oct 30 19:14:14.727801 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:14.727774 4442 generic.go:356] "Generic (PLEG): container finished" podID="57bae3b7-f320-45e8-bdff-e41abbb9f20c" containerID="ad46a731672326b4c762008c36d30c8e288327b2cb72e22e9d5b070da3842a39" exitCode=0 Oct 30 19:14:14.727906 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:14.727818 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" event={"ID":"57bae3b7-f320-45e8-bdff-e41abbb9f20c","Type":"ContainerDied","Data":"ad46a731672326b4c762008c36d30c8e288327b2cb72e22e9d5b070da3842a39"} Oct 30 19:14:14.727906 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:14.727871 4442 scope.go:122] "RemoveContainer" containerID="6d0627b0ed1bf6e7a5d4a27d1ff0aa10bae33940ce515c1bd38a78b1a8466a13" Oct 30 19:14:14.819888 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:14.819864 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" Oct 30 19:14:14.881637 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:14.881606 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57bae3b7-f320-45e8-bdff-e41abbb9f20c-istio-token\" (UniqueName: \"kubernetes.io/projected/57bae3b7-f320-45e8-bdff-e41abbb9f20c-istio-token\") pod \"57bae3b7-f320-45e8-bdff-e41abbb9f20c\" (UID: \"57bae3b7-f320-45e8-bdff-e41abbb9f20c\") " Oct 30 19:14:14.882217 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:14.881659 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57bae3b7-f320-45e8-bdff-e41abbb9f20c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57bae3b7-f320-45e8-bdff-e41abbb9f20c-istio-envoy\") pod \"57bae3b7-f320-45e8-bdff-e41abbb9f20c\" (UID: \"57bae3b7-f320-45e8-bdff-e41abbb9f20c\") " Oct 30 19:14:14.882217 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:14.881691 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57bae3b7-f320-45e8-bdff-e41abbb9f20c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bae3b7-f320-45e8-bdff-e41abbb9f20c-credential-socket\") pod \"57bae3b7-f320-45e8-bdff-e41abbb9f20c\" (UID: \"57bae3b7-f320-45e8-bdff-e41abbb9f20c\") " Oct 30 19:14:14.882217 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:14.881724 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57bae3b7-f320-45e8-bdff-e41abbb9f20c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57bae3b7-f320-45e8-bdff-e41abbb9f20c-istio-podinfo\") pod \"57bae3b7-f320-45e8-bdff-e41abbb9f20c\" (UID: \"57bae3b7-f320-45e8-bdff-e41abbb9f20c\") " Oct 30 19:14:14.882217 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:14.881790 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57bae3b7-f320-45e8-bdff-e41abbb9f20c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57bae3b7-f320-45e8-bdff-e41abbb9f20c-workload-certs\") pod \"57bae3b7-f320-45e8-bdff-e41abbb9f20c\" (UID: \"57bae3b7-f320-45e8-bdff-e41abbb9f20c\") " Oct 30 19:14:14.882217 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:14.881863 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57bae3b7-f320-45e8-bdff-e41abbb9f20c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bae3b7-f320-45e8-bdff-e41abbb9f20c-workload-socket\") pod \"57bae3b7-f320-45e8-bdff-e41abbb9f20c\" (UID: \"57bae3b7-f320-45e8-bdff-e41abbb9f20c\") " Oct 30 19:14:14.882217 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:14.882098 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57bae3b7-f320-45e8-bdff-e41abbb9f20c-credential-socket" pod "57bae3b7-f320-45e8-bdff-e41abbb9f20c" (UID: "57bae3b7-f320-45e8-bdff-e41abbb9f20c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:14.882217 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:14.882124 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57bae3b7-f320-45e8-bdff-e41abbb9f20c-workload-socket" pod "57bae3b7-f320-45e8-bdff-e41abbb9f20c" (UID: "57bae3b7-f320-45e8-bdff-e41abbb9f20c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:14.882819 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:14.882327 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57bae3b7-f320-45e8-bdff-e41abbb9f20c-workload-certs" pod "57bae3b7-f320-45e8-bdff-e41abbb9f20c" (UID: "57bae3b7-f320-45e8-bdff-e41abbb9f20c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:14.882819 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:14.882595 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57bae3b7-f320-45e8-bdff-e41abbb9f20c-istio-data" pod "57bae3b7-f320-45e8-bdff-e41abbb9f20c" (UID: "57bae3b7-f320-45e8-bdff-e41abbb9f20c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:14.884387 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:14.884350 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57bae3b7-f320-45e8-bdff-e41abbb9f20c-istio-envoy" pod "57bae3b7-f320-45e8-bdff-e41abbb9f20c" (UID: "57bae3b7-f320-45e8-bdff-e41abbb9f20c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:14.884588 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:14.884563 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57bae3b7-f320-45e8-bdff-e41abbb9f20c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57bae3b7-f320-45e8-bdff-e41abbb9f20c-istio-data\") pod \"57bae3b7-f320-45e8-bdff-e41abbb9f20c\" (UID: \"57bae3b7-f320-45e8-bdff-e41abbb9f20c\") " Oct 30 19:14:14.884769 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:14.884753 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57bae3b7-f320-45e8-bdff-e41abbb9f20c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57bae3b7-f320-45e8-bdff-e41abbb9f20c-istiod-ca-cert\") pod \"57bae3b7-f320-45e8-bdff-e41abbb9f20c\" (UID: \"57bae3b7-f320-45e8-bdff-e41abbb9f20c\") " Oct 30 19:14:14.884938 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:14.884915 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57bae3b7-f320-45e8-bdff-e41abbb9f20c-kube-api-access-68fhp\" (UniqueName: \"kubernetes.io/projected/57bae3b7-f320-45e8-bdff-e41abbb9f20c-kube-api-access-68fhp\") pod \"57bae3b7-f320-45e8-bdff-e41abbb9f20c\" (UID: \"57bae3b7-f320-45e8-bdff-e41abbb9f20c\") " Oct 30 19:14:14.885403 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:14.885046 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57bae3b7-f320-45e8-bdff-e41abbb9f20c-istio-token" pod "57bae3b7-f320-45e8-bdff-e41abbb9f20c" (UID: "57bae3b7-f320-45e8-bdff-e41abbb9f20c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:14.885403 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:14.885124 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57bae3b7-f320-45e8-bdff-e41abbb9f20c-istiod-ca-cert" pod "57bae3b7-f320-45e8-bdff-e41abbb9f20c" (UID: "57bae3b7-f320-45e8-bdff-e41abbb9f20c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:14:14.885544 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:14.885529 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bae3b7-f320-45e8-bdff-e41abbb9f20c-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:14.885644 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:14.885551 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57bae3b7-f320-45e8-bdff-e41abbb9f20c-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:14.885644 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:14.885565 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57bae3b7-f320-45e8-bdff-e41abbb9f20c-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:14.885644 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:14.885578 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57bae3b7-f320-45e8-bdff-e41abbb9f20c-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:14.885644 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:14.885590 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57bae3b7-f320-45e8-bdff-e41abbb9f20c-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:14.885644 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:14.885604 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bae3b7-f320-45e8-bdff-e41abbb9f20c-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:14.885644 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:14.885617 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57bae3b7-f320-45e8-bdff-e41abbb9f20c-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:14.886121 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:14.885707 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57bae3b7-f320-45e8-bdff-e41abbb9f20c-istio-podinfo" pod "57bae3b7-f320-45e8-bdff-e41abbb9f20c" (UID: "57bae3b7-f320-45e8-bdff-e41abbb9f20c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:14:14.887210 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:14.887185 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57bae3b7-f320-45e8-bdff-e41abbb9f20c-kube-api-access-68fhp" pod "57bae3b7-f320-45e8-bdff-e41abbb9f20c" (UID: "57bae3b7-f320-45e8-bdff-e41abbb9f20c"). InnerVolumeSpecName "kube-api-access-68fhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:14.986791 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:14.986757 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68fhp\" (UniqueName: \"kubernetes.io/projected/57bae3b7-f320-45e8-bdff-e41abbb9f20c-kube-api-access-68fhp\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:14.986791 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:14.986784 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57bae3b7-f320-45e8-bdff-e41abbb9f20c-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:15.043432 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:15.043343 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8eee7a8-v9kd6"] Oct 30 19:14:15.044409 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:15.044371 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8eee7a8-v9kd6"] Oct 30 19:14:15.063727 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:15.063694 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="855d59cb-e130-4f86-aba2-2f77b4e9fd2a" path="/var/lib/kubelet/pods/855d59cb-e130-4f86-aba2-2f77b4e9fd2a/volumes" Oct 30 19:14:15.737623 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:15.737472 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" event={"ID":"57bae3b7-f320-45e8-bdff-e41abbb9f20c","Type":"ContainerDied","Data":"f51a08592c66532ae445ae23464680004a7c713471644ffcc7cdba335e15e7eb"} Oct 30 19:14:15.737623 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:15.737497 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm" Oct 30 19:14:15.737623 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:15.737538 4442 scope.go:122] "RemoveContainer" containerID="ad46a731672326b4c762008c36d30c8e288327b2cb72e22e9d5b070da3842a39" Oct 30 19:14:15.802007 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:15.801970 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm"] Oct 30 19:14:15.813871 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:15.813839 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-6babca-openshift-default-cb56dc55c-v9rvm"] Oct 30 19:14:17.067630 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:17.067590 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57bae3b7-f320-45e8-bdff-e41abbb9f20c" path="/var/lib/kubelet/pods/57bae3b7-f320-45e8-bdff-e41abbb9f20c/volumes" Oct 30 19:14:17.561343 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:14:17.561308 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 19:14:21.344458 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:21.344418 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-236fc4-88f5755fb-fskbl"] Oct 30 19:14:21.376545 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:21.376509 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-236fc4-6c5fd847b7-xxv8l"] Oct 30 19:14:24.258259 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:24.258220 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c"] Oct 30 19:14:24.258908 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:24.258538 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" podUID="fb9a48e1-6d20-4f44-8dc6-eb0e6125276d" containerName="istio-proxy" containerID="cri-o://9d3eafd4d8bbded92b3012b35447c3b518dd880574e66189e06c655e423764d5" gracePeriod=30 Oct 30 19:14:24.489262 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:24.489227 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1a2436-7676957859-jxlmq" podUID="0249c5c5-6032-425a-9107-2526fd89627a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:14:26.068895 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:26.064188 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39a799a2-qmhd2"] Oct 30 19:14:26.070230 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:26.070195 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39a799a2-qmhd2"] Oct 30 19:14:26.345806 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:26.345697 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-236fc4-88f5755fb-fskbl" podUID="e4f4443c-7b80-47d0-8bbf-3e6ef11f437b" containerName="payload-processing" containerID="cri-o://07610df0022ae7c8f2f3cdddea980b64fac326104d01edae512e864f05bfd075" gracePeriod=25 Oct 30 19:14:26.386161 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:26.385354 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-236fc4-6c5fd847b7-xxv8l" podUID="97f2a51a-33c7-4869-8315-f5c266adac7a" containerName="payload-pre-processing" containerID="cri-o://d615caf88a30f4ad70b0e2c68c2b7c267d67ab7f45e8558f11be071a3f15430b" gracePeriod=25 Oct 30 19:14:26.424898 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:26.424862 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-675070-n64xj"] Oct 30 19:14:26.428078 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:26.428049 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-675070-n64xj"] Oct 30 19:14:27.065612 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:27.065579 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6b36d92-196f-4a2b-b312-9fc0d6260e58" path="/var/lib/kubelet/pods/c6b36d92-196f-4a2b-b312-9fc0d6260e58/volumes" Oct 30 19:14:27.066010 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:27.065993 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef052ec7-f935-4c53-bf21-6d1e2e9c90fc" path="/var/lib/kubelet/pods/ef052ec7-f935-4c53-bf21-6d1e2e9c90fc/volumes" Oct 30 19:14:29.534179 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:29.534077 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" Oct 30 19:14:29.599018 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:29.598988 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-istio-envoy\") pod \"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d\" (UID: \"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d\") " Oct 30 19:14:29.599018 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:29.599024 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-istiod-ca-cert\") pod \"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d\" (UID: \"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d\") " Oct 30 19:14:29.599274 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:29.599058 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-workload-certs\") pod \"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d\" (UID: \"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d\") " Oct 30 19:14:29.599274 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:29.599122 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-workload-socket\") pod \"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d\" (UID: \"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d\") " Oct 30 19:14:29.599274 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:29.599198 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-kube-api-access-gt9lb\" (UniqueName: \"kubernetes.io/projected/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-kube-api-access-gt9lb\") pod \"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d\" (UID: \"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d\") " Oct 30 19:14:29.599435 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:29.599276 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-istio-token\" (UniqueName: \"kubernetes.io/projected/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-istio-token\") pod \"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d\" (UID: \"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d\") " Oct 30 19:14:29.599435 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:29.599308 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-istio-podinfo\") pod \"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d\" (UID: \"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d\") " Oct 30 19:14:29.599435 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:29.599353 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-workload-certs" pod "fb9a48e1-6d20-4f44-8dc6-eb0e6125276d" (UID: "fb9a48e1-6d20-4f44-8dc6-eb0e6125276d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:29.599604 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:29.599447 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-istio-data\") pod \"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d\" (UID: \"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d\") " Oct 30 19:14:29.599604 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:29.599464 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-workload-socket" pod "fb9a48e1-6d20-4f44-8dc6-eb0e6125276d" (UID: "fb9a48e1-6d20-4f44-8dc6-eb0e6125276d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:29.599604 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:29.599473 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-istiod-ca-cert" pod "fb9a48e1-6d20-4f44-8dc6-eb0e6125276d" (UID: "fb9a48e1-6d20-4f44-8dc6-eb0e6125276d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:14:29.599604 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:29.599518 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-credential-socket\") pod \"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d\" (UID: \"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d\") " Oct 30 19:14:29.599825 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:29.599704 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-istio-data" pod "fb9a48e1-6d20-4f44-8dc6-eb0e6125276d" (UID: "fb9a48e1-6d20-4f44-8dc6-eb0e6125276d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:29.599883 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:29.599868 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-credential-socket" pod "fb9a48e1-6d20-4f44-8dc6-eb0e6125276d" (UID: "fb9a48e1-6d20-4f44-8dc6-eb0e6125276d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:29.600167 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:29.600044 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:29.600167 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:29.600068 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:29.600167 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:29.600086 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:29.600167 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:29.600099 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:29.600167 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:29.600155 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:29.602906 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:29.602883 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-istio-podinfo" pod "fb9a48e1-6d20-4f44-8dc6-eb0e6125276d" (UID: "fb9a48e1-6d20-4f44-8dc6-eb0e6125276d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:14:29.603010 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:29.602917 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-kube-api-access-gt9lb" pod "fb9a48e1-6d20-4f44-8dc6-eb0e6125276d" (UID: "fb9a48e1-6d20-4f44-8dc6-eb0e6125276d"). InnerVolumeSpecName "kube-api-access-gt9lb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:29.603010 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:29.602952 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-istio-token" pod "fb9a48e1-6d20-4f44-8dc6-eb0e6125276d" (UID: "fb9a48e1-6d20-4f44-8dc6-eb0e6125276d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:29.603160 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:29.603113 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-istio-envoy" pod "fb9a48e1-6d20-4f44-8dc6-eb0e6125276d" (UID: "fb9a48e1-6d20-4f44-8dc6-eb0e6125276d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:29.701331 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:29.701291 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:29.701331 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:29.701330 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:29.701537 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:29.701344 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:29.701537 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:29.701358 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt9lb\" (UniqueName: \"kubernetes.io/projected/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d-kube-api-access-gt9lb\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:29.932936 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:29.932902 4442 generic.go:356] "Generic (PLEG): container finished" podID="fb9a48e1-6d20-4f44-8dc6-eb0e6125276d" containerID="9d3eafd4d8bbded92b3012b35447c3b518dd880574e66189e06c655e423764d5" exitCode=0 Oct 30 19:14:29.933202 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:29.932970 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" Oct 30 19:14:29.933202 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:29.932992 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" event={"ID":"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d","Type":"ContainerDied","Data":"9d3eafd4d8bbded92b3012b35447c3b518dd880574e66189e06c655e423764d5"} Oct 30 19:14:29.933202 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:29.933055 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c" event={"ID":"fb9a48e1-6d20-4f44-8dc6-eb0e6125276d","Type":"ContainerDied","Data":"dc579ab93cffb95cd2697122cf4b4861b4b0b343f1e28ea4665f04e06dd42bd5"} Oct 30 19:14:29.933202 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:29.933090 4442 scope.go:122] "RemoveContainer" containerID="9d3eafd4d8bbded92b3012b35447c3b518dd880574e66189e06c655e423764d5" Oct 30 19:14:29.945000 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:29.944981 4442 scope.go:122] "RemoveContainer" containerID="6d4813a4d8e1241c4cecbf9c43bbf423127bd8b4f6fc0bff157e1e5dddc1868b" Oct 30 19:14:29.955475 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:29.955455 4442 scope.go:122] "RemoveContainer" containerID="9d3eafd4d8bbded92b3012b35447c3b518dd880574e66189e06c655e423764d5" Oct 30 19:14:29.955757 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:14:29.955736 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d3eafd4d8bbded92b3012b35447c3b518dd880574e66189e06c655e423764d5\": container with ID starting with 9d3eafd4d8bbded92b3012b35447c3b518dd880574e66189e06c655e423764d5 not found: ID does not exist" containerID="9d3eafd4d8bbded92b3012b35447c3b518dd880574e66189e06c655e423764d5" Oct 30 19:14:29.955830 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:29.955765 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d3eafd4d8bbded92b3012b35447c3b518dd880574e66189e06c655e423764d5"} err="failed to get container status \"9d3eafd4d8bbded92b3012b35447c3b518dd880574e66189e06c655e423764d5\": rpc error: code = NotFound desc = could not find container \"9d3eafd4d8bbded92b3012b35447c3b518dd880574e66189e06c655e423764d5\": container with ID starting with 9d3eafd4d8bbded92b3012b35447c3b518dd880574e66189e06c655e423764d5 not found: ID does not exist" Oct 30 19:14:29.955830 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:29.955786 4442 scope.go:122] "RemoveContainer" containerID="6d4813a4d8e1241c4cecbf9c43bbf423127bd8b4f6fc0bff157e1e5dddc1868b" Oct 30 19:14:29.956031 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:14:29.956010 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d4813a4d8e1241c4cecbf9c43bbf423127bd8b4f6fc0bff157e1e5dddc1868b\": container with ID starting with 6d4813a4d8e1241c4cecbf9c43bbf423127bd8b4f6fc0bff157e1e5dddc1868b not found: ID does not exist" containerID="6d4813a4d8e1241c4cecbf9c43bbf423127bd8b4f6fc0bff157e1e5dddc1868b" Oct 30 19:14:29.956104 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:29.956036 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d4813a4d8e1241c4cecbf9c43bbf423127bd8b4f6fc0bff157e1e5dddc1868b"} err="failed to get container status \"6d4813a4d8e1241c4cecbf9c43bbf423127bd8b4f6fc0bff157e1e5dddc1868b\": rpc error: code = NotFound desc = could not find container \"6d4813a4d8e1241c4cecbf9c43bbf423127bd8b4f6fc0bff157e1e5dddc1868b\": container with ID starting with 6d4813a4d8e1241c4cecbf9c43bbf423127bd8b4f6fc0bff157e1e5dddc1868b not found: ID does not exist" Oct 30 19:14:29.986877 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:29.986841 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c"] Oct 30 19:14:29.997220 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:29.997194 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-kqc5c"] Oct 30 19:14:31.065128 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:31.065091 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb9a48e1-6d20-4f44-8dc6-eb0e6125276d" path="/var/lib/kubelet/pods/fb9a48e1-6d20-4f44-8dc6-eb0e6125276d/volumes" Oct 30 19:14:31.624972 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:31.624932 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-236fc4-c8b87c5ff-9knfc"] Oct 30 19:14:31.625249 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:31.625193 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-236fc4-c8b87c5ff-9knfc" podUID="320da9e0-4338-4b7e-b950-7e3cc76b2faa" containerName="maas-api" containerID="cri-o://66e6d4aaaef585f8861ad78a159705b7349c285543bf79410e0e852ecf041e47" gracePeriod=30 Oct 30 19:14:31.931557 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:31.931535 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-236fc4-c8b87c5ff-9knfc" Oct 30 19:14:31.949952 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:31.949922 4442 generic.go:356] "Generic (PLEG): container finished" podID="320da9e0-4338-4b7e-b950-7e3cc76b2faa" containerID="66e6d4aaaef585f8861ad78a159705b7349c285543bf79410e0e852ecf041e47" exitCode=0 Oct 30 19:14:31.950100 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:31.949965 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-236fc4-c8b87c5ff-9knfc" Oct 30 19:14:31.950100 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:31.950003 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-236fc4-c8b87c5ff-9knfc" event={"ID":"320da9e0-4338-4b7e-b950-7e3cc76b2faa","Type":"ContainerDied","Data":"66e6d4aaaef585f8861ad78a159705b7349c285543bf79410e0e852ecf041e47"} Oct 30 19:14:31.950100 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:31.950063 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-236fc4-c8b87c5ff-9knfc" event={"ID":"320da9e0-4338-4b7e-b950-7e3cc76b2faa","Type":"ContainerDied","Data":"503ba738d9cc793c5704f6d3c3c8ad71a95e7a320178751138ff6b9759191a21"} Oct 30 19:14:31.950100 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:31.950093 4442 scope.go:122] "RemoveContainer" containerID="66e6d4aaaef585f8861ad78a159705b7349c285543bf79410e0e852ecf041e47" Oct 30 19:14:31.959990 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:31.959973 4442 scope.go:122] "RemoveContainer" containerID="66e6d4aaaef585f8861ad78a159705b7349c285543bf79410e0e852ecf041e47" Oct 30 19:14:31.960280 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:14:31.960260 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66e6d4aaaef585f8861ad78a159705b7349c285543bf79410e0e852ecf041e47\": container with ID starting with 66e6d4aaaef585f8861ad78a159705b7349c285543bf79410e0e852ecf041e47 not found: ID does not exist" containerID="66e6d4aaaef585f8861ad78a159705b7349c285543bf79410e0e852ecf041e47" Oct 30 19:14:31.960364 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:31.960290 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66e6d4aaaef585f8861ad78a159705b7349c285543bf79410e0e852ecf041e47"} err="failed to get container status \"66e6d4aaaef585f8861ad78a159705b7349c285543bf79410e0e852ecf041e47\": rpc error: code = NotFound desc = could not find container \"66e6d4aaaef585f8861ad78a159705b7349c285543bf79410e0e852ecf041e47\": container with ID starting with 66e6d4aaaef585f8861ad78a159705b7349c285543bf79410e0e852ecf041e47 not found: ID does not exist" Oct 30 19:14:32.028636 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:32.028609 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/320da9e0-4338-4b7e-b950-7e3cc76b2faa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/320da9e0-4338-4b7e-b950-7e3cc76b2faa-metrics-certs\") pod \"320da9e0-4338-4b7e-b950-7e3cc76b2faa\" (UID: \"320da9e0-4338-4b7e-b950-7e3cc76b2faa\") " Oct 30 19:14:32.028850 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:32.028673 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/320da9e0-4338-4b7e-b950-7e3cc76b2faa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/320da9e0-4338-4b7e-b950-7e3cc76b2faa-maas-api-tls\") pod \"320da9e0-4338-4b7e-b950-7e3cc76b2faa\" (UID: \"320da9e0-4338-4b7e-b950-7e3cc76b2faa\") " Oct 30 19:14:32.028850 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:32.028783 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/320da9e0-4338-4b7e-b950-7e3cc76b2faa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/320da9e0-4338-4b7e-b950-7e3cc76b2faa-openshift-service-ca\") pod \"320da9e0-4338-4b7e-b950-7e3cc76b2faa\" (UID: \"320da9e0-4338-4b7e-b950-7e3cc76b2faa\") " Oct 30 19:14:32.028982 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:32.028938 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/320da9e0-4338-4b7e-b950-7e3cc76b2faa-kube-api-access-6dfjq\" (UniqueName: \"kubernetes.io/projected/320da9e0-4338-4b7e-b950-7e3cc76b2faa-kube-api-access-6dfjq\") pod \"320da9e0-4338-4b7e-b950-7e3cc76b2faa\" (UID: \"320da9e0-4338-4b7e-b950-7e3cc76b2faa\") " Oct 30 19:14:32.029315 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:32.029282 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/320da9e0-4338-4b7e-b950-7e3cc76b2faa-openshift-service-ca" pod "320da9e0-4338-4b7e-b950-7e3cc76b2faa" (UID: "320da9e0-4338-4b7e-b950-7e3cc76b2faa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:14:32.029498 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:32.029466 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/320da9e0-4338-4b7e-b950-7e3cc76b2faa-openshift-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:32.031408 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:32.031387 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/320da9e0-4338-4b7e-b950-7e3cc76b2faa-metrics-certs" pod "320da9e0-4338-4b7e-b950-7e3cc76b2faa" (UID: "320da9e0-4338-4b7e-b950-7e3cc76b2faa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:14:32.031627 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:32.031611 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/320da9e0-4338-4b7e-b950-7e3cc76b2faa-maas-api-tls" pod "320da9e0-4338-4b7e-b950-7e3cc76b2faa" (UID: "320da9e0-4338-4b7e-b950-7e3cc76b2faa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:14:32.034084 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:32.034054 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/320da9e0-4338-4b7e-b950-7e3cc76b2faa-kube-api-access-6dfjq" pod "320da9e0-4338-4b7e-b950-7e3cc76b2faa" (UID: "320da9e0-4338-4b7e-b950-7e3cc76b2faa"). InnerVolumeSpecName "kube-api-access-6dfjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:32.142892 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:32.141817 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dfjq\" (UniqueName: \"kubernetes.io/projected/320da9e0-4338-4b7e-b950-7e3cc76b2faa-kube-api-access-6dfjq\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:32.142892 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:32.141859 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/320da9e0-4338-4b7e-b950-7e3cc76b2faa-metrics-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:32.142892 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:32.141876 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/320da9e0-4338-4b7e-b950-7e3cc76b2faa-maas-api-tls\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:32.296614 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:32.296581 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-236fc4-c8b87c5ff-9knfc"] Oct 30 19:14:32.308794 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:32.308765 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-236fc4-c8b87c5ff-9knfc"] Oct 30 19:14:33.064712 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:33.064680 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="320da9e0-4338-4b7e-b950-7e3cc76b2faa" path="/var/lib/kubelet/pods/320da9e0-4338-4b7e-b950-7e3cc76b2faa/volumes" Oct 30 19:14:33.231847 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:33.231805 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-236fc4-6c5fd847b7-xxv8l" podUID="97f2a51a-33c7-4869-8315-f5c266adac7a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:14:33.232313 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:33.231843 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-236fc4-88f5755fb-fskbl" podUID="e4f4443c-7b80-47d0-8bbf-3e6ef11f437b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:14:33.553018 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:33.552995 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1a2436-7676957859-jxlmq" Oct 30 19:14:33.665765 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:33.665684 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0249c5c5-6032-425a-9107-2526fd89627a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0249c5c5-6032-425a-9107-2526fd89627a-plugins-config-volume\") pod \"0249c5c5-6032-425a-9107-2526fd89627a\" (UID: \"0249c5c5-6032-425a-9107-2526fd89627a\") " Oct 30 19:14:33.665765 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:33.665755 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0249c5c5-6032-425a-9107-2526fd89627a-kube-api-access-fdc8z\" (UniqueName: \"kubernetes.io/projected/0249c5c5-6032-425a-9107-2526fd89627a-kube-api-access-fdc8z\") pod \"0249c5c5-6032-425a-9107-2526fd89627a\" (UID: \"0249c5c5-6032-425a-9107-2526fd89627a\") " Oct 30 19:14:33.666071 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:33.666045 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0249c5c5-6032-425a-9107-2526fd89627a-plugins-config-volume" pod "0249c5c5-6032-425a-9107-2526fd89627a" (UID: "0249c5c5-6032-425a-9107-2526fd89627a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:14:33.668149 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:33.668112 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0249c5c5-6032-425a-9107-2526fd89627a-kube-api-access-fdc8z" pod "0249c5c5-6032-425a-9107-2526fd89627a" (UID: "0249c5c5-6032-425a-9107-2526fd89627a"). InnerVolumeSpecName "kube-api-access-fdc8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:33.767645 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:33.767596 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0249c5c5-6032-425a-9107-2526fd89627a-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:33.767645 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:33.767643 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdc8z\" (UniqueName: \"kubernetes.io/projected/0249c5c5-6032-425a-9107-2526fd89627a-kube-api-access-fdc8z\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:33.915576 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:33.915536 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4"] Oct 30 19:14:33.915920 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:33.915854 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" podUID="4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54" containerName="istio-proxy" containerID="cri-o://44aa1e3f40f6b9c728b09f0d8f4db7b7e8e92c838e962d13bd77546b0fab7e06" gracePeriod=30 Oct 30 19:14:33.968900 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:33.968855 4442 generic.go:356] "Generic (PLEG): container finished" podID="0249c5c5-6032-425a-9107-2526fd89627a" containerID="5b6d407866c8208622812345b02af5516e616c559b0fce5ccf46501bf7ee5d1f" exitCode=0 Oct 30 19:14:33.969065 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:33.968941 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1a2436-7676957859-jxlmq" Oct 30 19:14:33.969065 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:33.968956 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1a2436-7676957859-jxlmq" event={"ID":"0249c5c5-6032-425a-9107-2526fd89627a","Type":"ContainerDied","Data":"5b6d407866c8208622812345b02af5516e616c559b0fce5ccf46501bf7ee5d1f"} Oct 30 19:14:33.969065 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:33.969021 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1a2436-7676957859-jxlmq" event={"ID":"0249c5c5-6032-425a-9107-2526fd89627a","Type":"ContainerDied","Data":"607e2f4eca7aaf4974e3607e9b458afc6fb516a375da6467ce57fe058c1d92ee"} Oct 30 19:14:33.969065 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:33.969047 4442 scope.go:122] "RemoveContainer" containerID="5b6d407866c8208622812345b02af5516e616c559b0fce5ccf46501bf7ee5d1f" Oct 30 19:14:33.980278 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:33.980261 4442 scope.go:122] "RemoveContainer" containerID="5b6d407866c8208622812345b02af5516e616c559b0fce5ccf46501bf7ee5d1f" Oct 30 19:14:33.980629 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:14:33.980595 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b6d407866c8208622812345b02af5516e616c559b0fce5ccf46501bf7ee5d1f\": container with ID starting with 5b6d407866c8208622812345b02af5516e616c559b0fce5ccf46501bf7ee5d1f not found: ID does not exist" containerID="5b6d407866c8208622812345b02af5516e616c559b0fce5ccf46501bf7ee5d1f" Oct 30 19:14:33.980711 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:33.980638 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b6d407866c8208622812345b02af5516e616c559b0fce5ccf46501bf7ee5d1f"} err="failed to get container status \"5b6d407866c8208622812345b02af5516e616c559b0fce5ccf46501bf7ee5d1f\": rpc error: code = NotFound desc = could not find container \"5b6d407866c8208622812345b02af5516e616c559b0fce5ccf46501bf7ee5d1f\": container with ID starting with 5b6d407866c8208622812345b02af5516e616c559b0fce5ccf46501bf7ee5d1f not found: ID does not exist" Oct 30 19:14:34.037221 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:34.031824 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1a2436-7676957859-jxlmq"] Oct 30 19:14:34.074237 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:34.073112 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1a2436-7676957859-jxlmq"] Oct 30 19:14:34.505370 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:34.505339 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-09ef1d-4fdhh"] Oct 30 19:14:34.505841 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:34.505828 4442 cpu_manager.go:414] "containerMap: removing container" podUID="c6b36d92-196f-4a2b-b312-9fc0d6260e58" containerName="revoke-keys" Oct 30 19:14:34.505883 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:34.505842 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6b36d92-196f-4a2b-b312-9fc0d6260e58" containerName="revoke-keys" Oct 30 19:14:34.505883 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:34.505856 4442 cpu_manager.go:414] "containerMap: removing container" podUID="57bae3b7-f320-45e8-bdff-e41abbb9f20c" containerName="istio-proxy" Oct 30 19:14:34.505883 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:34.505861 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57bae3b7-f320-45e8-bdff-e41abbb9f20c" containerName="istio-proxy" Oct 30 19:14:34.505883 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:34.505872 4442 cpu_manager.go:414] "containerMap: removing container" podUID="fb9a48e1-6d20-4f44-8dc6-eb0e6125276d" containerName="istio-proxy" Oct 30 19:14:34.505883 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:34.505878 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb9a48e1-6d20-4f44-8dc6-eb0e6125276d" containerName="istio-proxy" Oct 30 19:14:34.506049 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:34.505899 4442 cpu_manager.go:414] "containerMap: removing container" podUID="fb9a48e1-6d20-4f44-8dc6-eb0e6125276d" containerName="istio-proxy" Oct 30 19:14:34.506049 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:34.505904 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb9a48e1-6d20-4f44-8dc6-eb0e6125276d" containerName="istio-proxy" Oct 30 19:14:34.506049 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:34.505918 4442 cpu_manager.go:414] "containerMap: removing container" podUID="0249c5c5-6032-425a-9107-2526fd89627a" containerName="payload-pre-processing" Oct 30 19:14:34.506049 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:34.505922 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0249c5c5-6032-425a-9107-2526fd89627a" containerName="payload-pre-processing" Oct 30 19:14:34.506049 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:34.505929 4442 cpu_manager.go:414] "containerMap: removing container" podUID="320da9e0-4338-4b7e-b950-7e3cc76b2faa" containerName="maas-api" Oct 30 19:14:34.506049 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:34.505934 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="320da9e0-4338-4b7e-b950-7e3cc76b2faa" containerName="maas-api" Oct 30 19:14:34.506049 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:34.506003 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="57bae3b7-f320-45e8-bdff-e41abbb9f20c" containerName="istio-proxy" Oct 30 19:14:34.506049 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:34.506012 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6b36d92-196f-4a2b-b312-9fc0d6260e58" containerName="revoke-keys" Oct 30 19:14:34.506049 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:34.506018 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="57bae3b7-f320-45e8-bdff-e41abbb9f20c" containerName="istio-proxy" Oct 30 19:14:34.506049 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:34.506023 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb9a48e1-6d20-4f44-8dc6-eb0e6125276d" containerName="istio-proxy" Oct 30 19:14:34.506049 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:34.506031 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="721069e6-60e5-4a5f-b79f-6d8b47ec87aa" containerName="istio-proxy" Oct 30 19:14:34.506049 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:34.506037 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb9a48e1-6d20-4f44-8dc6-eb0e6125276d" containerName="istio-proxy" Oct 30 19:14:34.506049 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:34.506045 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="0249c5c5-6032-425a-9107-2526fd89627a" containerName="payload-pre-processing" Oct 30 19:14:34.506049 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:34.506050 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="320da9e0-4338-4b7e-b950-7e3cc76b2faa" containerName="maas-api" Oct 30 19:14:34.510405 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:34.510378 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-09ef1d-4fdhh" Oct 30 19:14:34.525343 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:34.525320 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:14:34.525615 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:34.525588 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:14:34.528239 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:34.528214 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-09ef1d-4fdhh"] Oct 30 19:14:34.576035 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:34.575986 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd074006-b670-47f8-a498-758e1dbe6991-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-09ef1d-4fdhh\" (UID: \"dd074006-b670-47f8-a498-758e1dbe6991\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-09ef1d-4fdhh" Oct 30 19:14:34.576310 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:34.576287 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgt5j\" (UniqueName: \"kubernetes.io/projected/dd074006-b670-47f8-a498-758e1dbe6991-kube-api-access-kgt5j\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-09ef1d-4fdhh\" (UID: \"dd074006-b670-47f8-a498-758e1dbe6991\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-09ef1d-4fdhh" Oct 30 19:14:34.576439 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:34.576413 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd074006-b670-47f8-a498-758e1dbe6991-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-09ef1d-4fdhh\" (UID: \"dd074006-b670-47f8-a498-758e1dbe6991\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-09ef1d-4fdhh" Oct 30 19:14:34.679756 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:34.679727 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgt5j\" (UniqueName: \"kubernetes.io/projected/dd074006-b670-47f8-a498-758e1dbe6991-kube-api-access-kgt5j\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-09ef1d-4fdhh\" (UID: \"dd074006-b670-47f8-a498-758e1dbe6991\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-09ef1d-4fdhh" Oct 30 19:14:34.679930 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:34.679781 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd074006-b670-47f8-a498-758e1dbe6991-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-09ef1d-4fdhh\" (UID: \"dd074006-b670-47f8-a498-758e1dbe6991\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-09ef1d-4fdhh" Oct 30 19:14:34.679930 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:34.679805 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd074006-b670-47f8-a498-758e1dbe6991-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-09ef1d-4fdhh\" (UID: \"dd074006-b670-47f8-a498-758e1dbe6991\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-09ef1d-4fdhh" Oct 30 19:14:34.680519 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:34.680496 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd074006-b670-47f8-a498-758e1dbe6991-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-09ef1d-4fdhh\" (UID: \"dd074006-b670-47f8-a498-758e1dbe6991\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-09ef1d-4fdhh" Oct 30 19:14:34.682511 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:34.682478 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd074006-b670-47f8-a498-758e1dbe6991-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-09ef1d-4fdhh\" (UID: \"dd074006-b670-47f8-a498-758e1dbe6991\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-09ef1d-4fdhh" Oct 30 19:14:34.699907 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:34.699878 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgt5j\" (UniqueName: \"kubernetes.io/projected/dd074006-b670-47f8-a498-758e1dbe6991-kube-api-access-kgt5j\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-09ef1d-4fdhh\" (UID: \"dd074006-b670-47f8-a498-758e1dbe6991\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-09ef1d-4fdhh" Oct 30 19:14:34.824848 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:34.824763 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-09ef1d-4fdhh" Oct 30 19:14:35.064835 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:35.064784 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0249c5c5-6032-425a-9107-2526fd89627a" path="/var/lib/kubelet/pods/0249c5c5-6032-425a-9107-2526fd89627a/volumes" Oct 30 19:14:35.131205 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:35.131176 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-09ef1d-4fdhh"] Oct 30 19:14:35.131594 ip-10-0-138-176 kubenswrapper[4442]: W1030 19:14:35.131569 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd074006_b670_47f8_a498_758e1dbe6991.slice/crio-72231778b8a6f819c89e664b9b3b02dc529386d6886797a0bf9c180bfb3ac5a7 WatchSource:0}: Error finding container 72231778b8a6f819c89e664b9b3b02dc529386d6886797a0bf9c180bfb3ac5a7: Status 404 returned error can't find the container with id 72231778b8a6f819c89e664b9b3b02dc529386d6886797a0bf9c180bfb3ac5a7 Oct 30 19:14:35.988182 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:35.988127 4442 generic.go:356] "Generic (PLEG): container finished" podID="dd074006-b670-47f8-a498-758e1dbe6991" containerID="8871870f21f262018e7eb2b1185acc07a7f28cffa2631728dd5d635be7af4b0f" exitCode=0 Oct 30 19:14:35.988182 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:35.988175 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-09ef1d-4fdhh" event={"ID":"dd074006-b670-47f8-a498-758e1dbe6991","Type":"ContainerDied","Data":"8871870f21f262018e7eb2b1185acc07a7f28cffa2631728dd5d635be7af4b0f"} Oct 30 19:14:35.988614 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:35.988228 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-09ef1d-4fdhh" event={"ID":"dd074006-b670-47f8-a498-758e1dbe6991","Type":"ContainerStarted","Data":"72231778b8a6f819c89e664b9b3b02dc529386d6886797a0bf9c180bfb3ac5a7"} Oct 30 19:14:37.015587 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:37.015544 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:14:37.016051 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:37.015601 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" podUID="4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:14:37.153649 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:37.153620 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-09ef1d-4fdhh" Oct 30 19:14:37.313565 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:37.313462 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd074006-b670-47f8-a498-758e1dbe6991-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd074006-b670-47f8-a498-758e1dbe6991-maas-api-openshift-ca\") pod \"dd074006-b670-47f8-a498-758e1dbe6991\" (UID: \"dd074006-b670-47f8-a498-758e1dbe6991\") " Oct 30 19:14:37.313746 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:37.313603 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd074006-b670-47f8-a498-758e1dbe6991-kube-api-access-kgt5j\" (UniqueName: \"kubernetes.io/projected/dd074006-b670-47f8-a498-758e1dbe6991-kube-api-access-kgt5j\") pod \"dd074006-b670-47f8-a498-758e1dbe6991\" (UID: \"dd074006-b670-47f8-a498-758e1dbe6991\") " Oct 30 19:14:37.313746 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:37.313637 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd074006-b670-47f8-a498-758e1dbe6991-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd074006-b670-47f8-a498-758e1dbe6991-maas-api-xks-ca\") pod \"dd074006-b670-47f8-a498-758e1dbe6991\" (UID: \"dd074006-b670-47f8-a498-758e1dbe6991\") " Oct 30 19:14:37.313910 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:37.313878 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd074006-b670-47f8-a498-758e1dbe6991-maas-api-openshift-ca" pod "dd074006-b670-47f8-a498-758e1dbe6991" (UID: "dd074006-b670-47f8-a498-758e1dbe6991"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:14:37.314071 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:37.314053 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd074006-b670-47f8-a498-758e1dbe6991-maas-api-openshift-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:37.315939 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:37.315913 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd074006-b670-47f8-a498-758e1dbe6991-maas-api-xks-ca" pod "dd074006-b670-47f8-a498-758e1dbe6991" (UID: "dd074006-b670-47f8-a498-758e1dbe6991"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:14:37.316052 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:37.315941 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd074006-b670-47f8-a498-758e1dbe6991-kube-api-access-kgt5j" pod "dd074006-b670-47f8-a498-758e1dbe6991" (UID: "dd074006-b670-47f8-a498-758e1dbe6991"). InnerVolumeSpecName "kube-api-access-kgt5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:37.414958 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:37.414916 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgt5j\" (UniqueName: \"kubernetes.io/projected/dd074006-b670-47f8-a498-758e1dbe6991-kube-api-access-kgt5j\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:37.414958 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:37.414958 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd074006-b670-47f8-a498-758e1dbe6991-maas-api-xks-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:38.004717 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:38.004687 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-09ef1d-4fdhh" Oct 30 19:14:38.004899 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:38.004721 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-09ef1d-4fdhh" event={"ID":"dd074006-b670-47f8-a498-758e1dbe6991","Type":"ContainerDied","Data":"72231778b8a6f819c89e664b9b3b02dc529386d6886797a0bf9c180bfb3ac5a7"} Oct 30 19:14:38.004899 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:38.004765 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72231778b8a6f819c89e664b9b3b02dc529386d6886797a0bf9c180bfb3ac5a7" Oct 30 19:14:39.197874 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:39.197810 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" Oct 30 19:14:39.348610 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:39.348567 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-istio-envoy\") pod \"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54\" (UID: \"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54\") " Oct 30 19:14:39.348779 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:39.348669 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-istiod-ca-cert\") pod \"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54\" (UID: \"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54\") " Oct 30 19:14:39.348779 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:39.348704 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-workload-certs\") pod \"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54\" (UID: \"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54\") " Oct 30 19:14:39.348779 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:39.348739 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-credential-socket\") pod \"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54\" (UID: \"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54\") " Oct 30 19:14:39.348926 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:39.348826 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-kube-api-access-qv4gk\" (UniqueName: \"kubernetes.io/projected/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-kube-api-access-qv4gk\") pod \"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54\" (UID: \"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54\") " Oct 30 19:14:39.348926 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:39.348881 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-istio-data\") pod \"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54\" (UID: \"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54\") " Oct 30 19:14:39.348926 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:39.348914 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-istio-token\" (UniqueName: \"kubernetes.io/projected/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-istio-token\") pod \"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54\" (UID: \"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54\") " Oct 30 19:14:39.349469 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:39.349001 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-istio-podinfo\") pod \"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54\" (UID: \"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54\") " Oct 30 19:14:39.349469 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:39.349069 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-workload-socket\") pod \"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54\" (UID: \"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54\") " Oct 30 19:14:39.349469 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:39.349278 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-credential-socket" pod "4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54" (UID: "4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:39.349640 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:39.349517 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-workload-socket" pod "4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54" (UID: "4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:39.349705 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:39.349664 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:39.349705 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:39.349681 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:39.349817 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:39.349696 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-istio-data" pod "4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54" (UID: "4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:39.350532 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:39.350174 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-istiod-ca-cert" pod "4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54" (UID: "4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:14:39.350532 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:39.350474 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-workload-certs" pod "4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54" (UID: "4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:39.351582 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:39.351555 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-istio-envoy" pod "4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54" (UID: "4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:39.352334 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:39.352304 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-istio-token" pod "4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54" (UID: "4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:39.352838 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:39.352819 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-istio-podinfo" pod "4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54" (UID: "4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:14:39.353184 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:39.353167 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-kube-api-access-qv4gk" pod "4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54" (UID: "4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54"). InnerVolumeSpecName "kube-api-access-qv4gk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:39.451083 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:39.450981 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:39.451083 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:39.451018 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:39.451083 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:39.451033 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:39.451083 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:39.451046 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:39.451083 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:39.451060 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv4gk\" (UniqueName: \"kubernetes.io/projected/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-kube-api-access-qv4gk\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:39.451083 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:39.451073 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:39.451083 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:39.451087 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:39.719600 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:39.719517 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs"] Oct 30 19:14:39.719781 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:39.719755 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" podUID="bfbb6b47-1274-4d3d-8845-627898246065" containerName="istio-proxy" containerID="cri-o://1ff4a0cd501cafc0d3c80ef1c5a352e444fdb8bd53b65bda59ea6122c68c33b9" gracePeriod=30 Oct 30 19:14:40.022447 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:40.022365 4442 generic.go:356] "Generic (PLEG): container finished" podID="4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54" containerID="44aa1e3f40f6b9c728b09f0d8f4db7b7e8e92c838e962d13bd77546b0fab7e06" exitCode=0 Oct 30 19:14:40.022447 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:40.022423 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" event={"ID":"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54","Type":"ContainerDied","Data":"44aa1e3f40f6b9c728b09f0d8f4db7b7e8e92c838e962d13bd77546b0fab7e06"} Oct 30 19:14:40.022674 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:40.022487 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" event={"ID":"4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54","Type":"ContainerDied","Data":"d10f6dcfe04e75e9d975292928cee936a1e3abe930acc4c85cc311e5e9613eef"} Oct 30 19:14:40.022674 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:40.022499 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4" Oct 30 19:14:40.022674 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:40.022520 4442 scope.go:122] "RemoveContainer" containerID="44aa1e3f40f6b9c728b09f0d8f4db7b7e8e92c838e962d13bd77546b0fab7e06" Oct 30 19:14:40.041897 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:40.041855 4442 scope.go:122] "RemoveContainer" containerID="428227e6c5a7d68f2d063fd873665ee979ec1ec52df3f48dd50ed5b36d64cd24" Oct 30 19:14:40.052057 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:40.052032 4442 scope.go:122] "RemoveContainer" containerID="44aa1e3f40f6b9c728b09f0d8f4db7b7e8e92c838e962d13bd77546b0fab7e06" Oct 30 19:14:40.052433 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:14:40.052403 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44aa1e3f40f6b9c728b09f0d8f4db7b7e8e92c838e962d13bd77546b0fab7e06\": container with ID starting with 44aa1e3f40f6b9c728b09f0d8f4db7b7e8e92c838e962d13bd77546b0fab7e06 not found: ID does not exist" containerID="44aa1e3f40f6b9c728b09f0d8f4db7b7e8e92c838e962d13bd77546b0fab7e06" Oct 30 19:14:40.052518 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:40.052445 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44aa1e3f40f6b9c728b09f0d8f4db7b7e8e92c838e962d13bd77546b0fab7e06"} err="failed to get container status \"44aa1e3f40f6b9c728b09f0d8f4db7b7e8e92c838e962d13bd77546b0fab7e06\": rpc error: code = NotFound desc = could not find container \"44aa1e3f40f6b9c728b09f0d8f4db7b7e8e92c838e962d13bd77546b0fab7e06\": container with ID starting with 44aa1e3f40f6b9c728b09f0d8f4db7b7e8e92c838e962d13bd77546b0fab7e06 not found: ID does not exist" Oct 30 19:14:40.052518 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:40.052471 4442 scope.go:122] "RemoveContainer" containerID="428227e6c5a7d68f2d063fd873665ee979ec1ec52df3f48dd50ed5b36d64cd24" Oct 30 19:14:40.052735 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:14:40.052716 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"428227e6c5a7d68f2d063fd873665ee979ec1ec52df3f48dd50ed5b36d64cd24\": container with ID starting with 428227e6c5a7d68f2d063fd873665ee979ec1ec52df3f48dd50ed5b36d64cd24 not found: ID does not exist" containerID="428227e6c5a7d68f2d063fd873665ee979ec1ec52df3f48dd50ed5b36d64cd24" Oct 30 19:14:40.052796 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:40.052741 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"428227e6c5a7d68f2d063fd873665ee979ec1ec52df3f48dd50ed5b36d64cd24"} err="failed to get container status \"428227e6c5a7d68f2d063fd873665ee979ec1ec52df3f48dd50ed5b36d64cd24\": rpc error: code = NotFound desc = could not find container \"428227e6c5a7d68f2d063fd873665ee979ec1ec52df3f48dd50ed5b36d64cd24\": container with ID starting with 428227e6c5a7d68f2d063fd873665ee979ec1ec52df3f48dd50ed5b36d64cd24 not found: ID does not exist" Oct 30 19:14:40.090814 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:40.090786 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4"] Oct 30 19:14:40.109170 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:40.109123 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-675070-openshift-default-8b45bc97-jxjr4"] Oct 30 19:14:40.329522 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:40.329440 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c095fa-p6zcn"] Oct 30 19:14:40.329948 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:40.329934 4442 cpu_manager.go:414] "containerMap: removing container" podUID="4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54" containerName="istio-proxy" Oct 30 19:14:40.329948 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:40.329948 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54" containerName="istio-proxy" Oct 30 19:14:40.330027 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:40.329971 4442 cpu_manager.go:414] "containerMap: removing container" podUID="57bae3b7-f320-45e8-bdff-e41abbb9f20c" containerName="istio-proxy" Oct 30 19:14:40.330027 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:40.329976 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57bae3b7-f320-45e8-bdff-e41abbb9f20c" containerName="istio-proxy" Oct 30 19:14:40.330027 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:40.329989 4442 cpu_manager.go:414] "containerMap: removing container" podUID="4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54" containerName="istio-proxy" Oct 30 19:14:40.330027 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:40.329994 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54" containerName="istio-proxy" Oct 30 19:14:40.330027 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:40.330009 4442 cpu_manager.go:414] "containerMap: removing container" podUID="dd074006-b670-47f8-a498-758e1dbe6991" containerName="revoke-keys" Oct 30 19:14:40.330027 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:40.330013 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd074006-b670-47f8-a498-758e1dbe6991" containerName="revoke-keys" Oct 30 19:14:40.330258 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:40.330107 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54" containerName="istio-proxy" Oct 30 19:14:40.330258 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:40.330116 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54" containerName="istio-proxy" Oct 30 19:14:40.330258 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:40.330125 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd074006-b670-47f8-a498-758e1dbe6991" containerName="revoke-keys" Oct 30 19:14:40.334915 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:40.334899 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c095fa-p6zcn" Oct 30 19:14:40.361987 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:40.361963 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c095fa-p6zcn"] Oct 30 19:14:40.385802 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:40.385759 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:14:40.385802 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:40.385779 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:14:40.463394 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:40.463343 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/512ab6d5-f111-4957-bb85-b9373f3cb13d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c095fa-p6zcn\" (UID: \"512ab6d5-f111-4957-bb85-b9373f3cb13d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c095fa-p6zcn" Oct 30 19:14:40.463620 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:40.463584 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/512ab6d5-f111-4957-bb85-b9373f3cb13d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c095fa-p6zcn\" (UID: \"512ab6d5-f111-4957-bb85-b9373f3cb13d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c095fa-p6zcn" Oct 30 19:14:40.463700 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:40.463628 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qxwj\" (UniqueName: \"kubernetes.io/projected/512ab6d5-f111-4957-bb85-b9373f3cb13d-kube-api-access-9qxwj\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c095fa-p6zcn\" (UID: \"512ab6d5-f111-4957-bb85-b9373f3cb13d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c095fa-p6zcn" Oct 30 19:14:40.564671 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:40.564634 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/512ab6d5-f111-4957-bb85-b9373f3cb13d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c095fa-p6zcn\" (UID: \"512ab6d5-f111-4957-bb85-b9373f3cb13d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c095fa-p6zcn" Oct 30 19:14:40.564671 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:40.564678 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qxwj\" (UniqueName: \"kubernetes.io/projected/512ab6d5-f111-4957-bb85-b9373f3cb13d-kube-api-access-9qxwj\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c095fa-p6zcn\" (UID: \"512ab6d5-f111-4957-bb85-b9373f3cb13d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c095fa-p6zcn" Oct 30 19:14:40.564890 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:40.564757 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/512ab6d5-f111-4957-bb85-b9373f3cb13d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c095fa-p6zcn\" (UID: \"512ab6d5-f111-4957-bb85-b9373f3cb13d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c095fa-p6zcn" Oct 30 19:14:40.565537 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:40.565510 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/512ab6d5-f111-4957-bb85-b9373f3cb13d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c095fa-p6zcn\" (UID: \"512ab6d5-f111-4957-bb85-b9373f3cb13d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c095fa-p6zcn" Oct 30 19:14:40.567360 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:40.567331 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/512ab6d5-f111-4957-bb85-b9373f3cb13d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c095fa-p6zcn\" (UID: \"512ab6d5-f111-4957-bb85-b9373f3cb13d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c095fa-p6zcn" Oct 30 19:14:40.583560 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:40.583492 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qxwj\" (UniqueName: \"kubernetes.io/projected/512ab6d5-f111-4957-bb85-b9373f3cb13d-kube-api-access-9qxwj\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c095fa-p6zcn\" (UID: \"512ab6d5-f111-4957-bb85-b9373f3cb13d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c095fa-p6zcn" Oct 30 19:14:40.649514 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:40.649480 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c095fa-p6zcn" Oct 30 19:14:40.950509 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:40.950331 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c095fa-p6zcn"] Oct 30 19:14:40.951490 ip-10-0-138-176 kubenswrapper[4442]: W1030 19:14:40.951446 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod512ab6d5_f111_4957_bb85_b9373f3cb13d.slice/crio-0c9948d63bb6eaf6863d85bf3b978d153bf19bb8df378bdcd878ab7adc333e0b WatchSource:0}: Error finding container 0c9948d63bb6eaf6863d85bf3b978d153bf19bb8df378bdcd878ab7adc333e0b: Status 404 returned error can't find the container with id 0c9948d63bb6eaf6863d85bf3b978d153bf19bb8df378bdcd878ab7adc333e0b Oct 30 19:14:41.035105 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:41.035065 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c095fa-p6zcn" event={"ID":"512ab6d5-f111-4957-bb85-b9373f3cb13d","Type":"ContainerStarted","Data":"0c9948d63bb6eaf6863d85bf3b978d153bf19bb8df378bdcd878ab7adc333e0b"} Oct 30 19:14:41.064950 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:41.064920 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54" path="/var/lib/kubelet/pods/4bf25ed4-7f2c-4b1b-ad43-e38b37d9ab54/volumes" Oct 30 19:14:42.046615 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:42.046578 4442 generic.go:356] "Generic (PLEG): container finished" podID="512ab6d5-f111-4957-bb85-b9373f3cb13d" containerID="e3b3a91048fd5dcd56308d375b40162cc8917c530a9bd8ef67489a1d557a7070" exitCode=0 Oct 30 19:14:42.047169 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:42.046664 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c095fa-p6zcn" event={"ID":"512ab6d5-f111-4957-bb85-b9373f3cb13d","Type":"ContainerDied","Data":"e3b3a91048fd5dcd56308d375b40162cc8917c530a9bd8ef67489a1d557a7070"} Oct 30 19:14:43.208531 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:43.208502 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c095fa-p6zcn" Oct 30 19:14:43.231369 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:43.231332 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-236fc4-6c5fd847b7-xxv8l" podUID="97f2a51a-33c7-4869-8315-f5c266adac7a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:14:43.231551 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:43.231462 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-236fc4-88f5755fb-fskbl" podUID="e4f4443c-7b80-47d0-8bbf-3e6ef11f437b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:14:43.296093 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:43.296064 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/512ab6d5-f111-4957-bb85-b9373f3cb13d-kube-api-access-9qxwj\" (UniqueName: \"kubernetes.io/projected/512ab6d5-f111-4957-bb85-b9373f3cb13d-kube-api-access-9qxwj\") pod \"512ab6d5-f111-4957-bb85-b9373f3cb13d\" (UID: \"512ab6d5-f111-4957-bb85-b9373f3cb13d\") " Oct 30 19:14:43.296323 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:43.296188 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/512ab6d5-f111-4957-bb85-b9373f3cb13d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/512ab6d5-f111-4957-bb85-b9373f3cb13d-maas-api-openshift-ca\") pod \"512ab6d5-f111-4957-bb85-b9373f3cb13d\" (UID: \"512ab6d5-f111-4957-bb85-b9373f3cb13d\") " Oct 30 19:14:43.296323 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:43.296266 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/512ab6d5-f111-4957-bb85-b9373f3cb13d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/512ab6d5-f111-4957-bb85-b9373f3cb13d-maas-api-xks-ca\") pod \"512ab6d5-f111-4957-bb85-b9373f3cb13d\" (UID: \"512ab6d5-f111-4957-bb85-b9373f3cb13d\") " Oct 30 19:14:43.296715 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:43.296689 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/512ab6d5-f111-4957-bb85-b9373f3cb13d-maas-api-openshift-ca" pod "512ab6d5-f111-4957-bb85-b9373f3cb13d" (UID: "512ab6d5-f111-4957-bb85-b9373f3cb13d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:14:43.298735 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:43.298699 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/512ab6d5-f111-4957-bb85-b9373f3cb13d-maas-api-xks-ca" pod "512ab6d5-f111-4957-bb85-b9373f3cb13d" (UID: "512ab6d5-f111-4957-bb85-b9373f3cb13d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:14:43.298854 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:43.298833 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/512ab6d5-f111-4957-bb85-b9373f3cb13d-kube-api-access-9qxwj" pod "512ab6d5-f111-4957-bb85-b9373f3cb13d" (UID: "512ab6d5-f111-4957-bb85-b9373f3cb13d"). InnerVolumeSpecName "kube-api-access-9qxwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:43.397821 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:43.397791 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/512ab6d5-f111-4957-bb85-b9373f3cb13d-maas-api-openshift-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:43.397821 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:43.397820 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/512ab6d5-f111-4957-bb85-b9373f3cb13d-maas-api-xks-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:43.397821 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:43.397829 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qxwj\" (UniqueName: \"kubernetes.io/projected/512ab6d5-f111-4957-bb85-b9373f3cb13d-kube-api-access-9qxwj\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:44.065181 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:44.065145 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c095fa-p6zcn" Oct 30 19:14:44.065374 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:44.065151 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c095fa-p6zcn" event={"ID":"512ab6d5-f111-4957-bb85-b9373f3cb13d","Type":"ContainerDied","Data":"0c9948d63bb6eaf6863d85bf3b978d153bf19bb8df378bdcd878ab7adc333e0b"} Oct 30 19:14:44.065374 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:44.065294 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c9948d63bb6eaf6863d85bf3b978d153bf19bb8df378bdcd878ab7adc333e0b" Oct 30 19:14:44.132781 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:44.132752 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:14:44.132944 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:44.132806 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" podUID="bfbb6b47-1274-4d3d-8845-627898246065" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:14:44.979785 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:44.979760 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" Oct 30 19:14:45.091255 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:45.091110 4442 generic.go:356] "Generic (PLEG): container finished" podID="bfbb6b47-1274-4d3d-8845-627898246065" containerID="1ff4a0cd501cafc0d3c80ef1c5a352e444fdb8bd53b65bda59ea6122c68c33b9" exitCode=0 Oct 30 19:14:45.091255 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:45.091187 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" Oct 30 19:14:45.091255 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:45.091189 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" event={"ID":"bfbb6b47-1274-4d3d-8845-627898246065","Type":"ContainerDied","Data":"1ff4a0cd501cafc0d3c80ef1c5a352e444fdb8bd53b65bda59ea6122c68c33b9"} Oct 30 19:14:45.091255 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:45.091252 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs" event={"ID":"bfbb6b47-1274-4d3d-8845-627898246065","Type":"ContainerDied","Data":"6984700f5380d2c175c5f8f7bbc26284ab4671adcb3988d48fa2b302d7589364"} Oct 30 19:14:45.091649 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:45.091290 4442 scope.go:122] "RemoveContainer" containerID="1ff4a0cd501cafc0d3c80ef1c5a352e444fdb8bd53b65bda59ea6122c68c33b9" Oct 30 19:14:45.104978 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:45.104953 4442 scope.go:122] "RemoveContainer" containerID="09643b155ae61d3db212a12fe01e8b94e6d8211a3180916351b902bcce2dad97" Oct 30 19:14:45.117084 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:45.117045 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfbb6b47-1274-4d3d-8845-627898246065-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfbb6b47-1274-4d3d-8845-627898246065-istiod-ca-cert\") pod \"bfbb6b47-1274-4d3d-8845-627898246065\" (UID: \"bfbb6b47-1274-4d3d-8845-627898246065\") " Oct 30 19:14:45.117760 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:45.117724 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfbb6b47-1274-4d3d-8845-627898246065-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfbb6b47-1274-4d3d-8845-627898246065-istio-data\") pod \"bfbb6b47-1274-4d3d-8845-627898246065\" (UID: \"bfbb6b47-1274-4d3d-8845-627898246065\") " Oct 30 19:14:45.118030 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:45.117634 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfbb6b47-1274-4d3d-8845-627898246065-istiod-ca-cert" pod "bfbb6b47-1274-4d3d-8845-627898246065" (UID: "bfbb6b47-1274-4d3d-8845-627898246065"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:14:45.118553 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:45.118527 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfbb6b47-1274-4d3d-8845-627898246065-credential-socket" pod "bfbb6b47-1274-4d3d-8845-627898246065" (UID: "bfbb6b47-1274-4d3d-8845-627898246065"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:45.118639 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:45.118588 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfbb6b47-1274-4d3d-8845-627898246065-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfbb6b47-1274-4d3d-8845-627898246065-credential-socket\") pod \"bfbb6b47-1274-4d3d-8845-627898246065\" (UID: \"bfbb6b47-1274-4d3d-8845-627898246065\") " Oct 30 19:14:45.118796 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:45.118776 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfbb6b47-1274-4d3d-8845-627898246065-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfbb6b47-1274-4d3d-8845-627898246065-workload-socket\") pod \"bfbb6b47-1274-4d3d-8845-627898246065\" (UID: \"bfbb6b47-1274-4d3d-8845-627898246065\") " Oct 30 19:14:45.118868 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:45.118843 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfbb6b47-1274-4d3d-8845-627898246065-istio-data" pod "bfbb6b47-1274-4d3d-8845-627898246065" (UID: "bfbb6b47-1274-4d3d-8845-627898246065"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:45.119177 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:45.119126 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfbb6b47-1274-4d3d-8845-627898246065-workload-socket" pod "bfbb6b47-1274-4d3d-8845-627898246065" (UID: "bfbb6b47-1274-4d3d-8845-627898246065"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:45.119285 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:45.119267 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfbb6b47-1274-4d3d-8845-627898246065-kube-api-access-g5jhm\" (UniqueName: \"kubernetes.io/projected/bfbb6b47-1274-4d3d-8845-627898246065-kube-api-access-g5jhm\") pod \"bfbb6b47-1274-4d3d-8845-627898246065\" (UID: \"bfbb6b47-1274-4d3d-8845-627898246065\") " Oct 30 19:14:45.119362 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:45.119345 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfbb6b47-1274-4d3d-8845-627898246065-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfbb6b47-1274-4d3d-8845-627898246065-istio-envoy\") pod \"bfbb6b47-1274-4d3d-8845-627898246065\" (UID: \"bfbb6b47-1274-4d3d-8845-627898246065\") " Oct 30 19:14:45.120272 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:45.120236 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfbb6b47-1274-4d3d-8845-627898246065-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfbb6b47-1274-4d3d-8845-627898246065-workload-certs\") pod \"bfbb6b47-1274-4d3d-8845-627898246065\" (UID: \"bfbb6b47-1274-4d3d-8845-627898246065\") " Oct 30 19:14:45.120846 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:45.120386 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfbb6b47-1274-4d3d-8845-627898246065-istio-token\" (UniqueName: \"kubernetes.io/projected/bfbb6b47-1274-4d3d-8845-627898246065-istio-token\") pod \"bfbb6b47-1274-4d3d-8845-627898246065\" (UID: \"bfbb6b47-1274-4d3d-8845-627898246065\") " Oct 30 19:14:45.120846 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:45.120493 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bfbb6b47-1274-4d3d-8845-627898246065-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfbb6b47-1274-4d3d-8845-627898246065-istio-podinfo\") pod \"bfbb6b47-1274-4d3d-8845-627898246065\" (UID: \"bfbb6b47-1274-4d3d-8845-627898246065\") " Oct 30 19:14:45.121606 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:45.121084 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfbb6b47-1274-4d3d-8845-627898246065-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:45.121606 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:45.121113 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfbb6b47-1274-4d3d-8845-627898246065-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:45.121606 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:45.121148 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfbb6b47-1274-4d3d-8845-627898246065-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:45.121606 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:45.121165 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfbb6b47-1274-4d3d-8845-627898246065-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:45.121606 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:45.121401 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfbb6b47-1274-4d3d-8845-627898246065-workload-certs" pod "bfbb6b47-1274-4d3d-8845-627898246065" (UID: "bfbb6b47-1274-4d3d-8845-627898246065"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:45.122321 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:45.122206 4442 scope.go:122] "RemoveContainer" containerID="1ff4a0cd501cafc0d3c80ef1c5a352e444fdb8bd53b65bda59ea6122c68c33b9" Oct 30 19:14:45.122743 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:14:45.122563 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ff4a0cd501cafc0d3c80ef1c5a352e444fdb8bd53b65bda59ea6122c68c33b9\": container with ID starting with 1ff4a0cd501cafc0d3c80ef1c5a352e444fdb8bd53b65bda59ea6122c68c33b9 not found: ID does not exist" containerID="1ff4a0cd501cafc0d3c80ef1c5a352e444fdb8bd53b65bda59ea6122c68c33b9" Oct 30 19:14:45.122743 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:45.122604 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ff4a0cd501cafc0d3c80ef1c5a352e444fdb8bd53b65bda59ea6122c68c33b9"} err="failed to get container status \"1ff4a0cd501cafc0d3c80ef1c5a352e444fdb8bd53b65bda59ea6122c68c33b9\": rpc error: code = NotFound desc = could not find container \"1ff4a0cd501cafc0d3c80ef1c5a352e444fdb8bd53b65bda59ea6122c68c33b9\": container with ID starting with 1ff4a0cd501cafc0d3c80ef1c5a352e444fdb8bd53b65bda59ea6122c68c33b9 not found: ID does not exist" Oct 30 19:14:45.122743 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:45.122632 4442 scope.go:122] "RemoveContainer" containerID="09643b155ae61d3db212a12fe01e8b94e6d8211a3180916351b902bcce2dad97" Oct 30 19:14:45.123456 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:14:45.123386 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09643b155ae61d3db212a12fe01e8b94e6d8211a3180916351b902bcce2dad97\": container with ID starting with 09643b155ae61d3db212a12fe01e8b94e6d8211a3180916351b902bcce2dad97 not found: ID does not exist" containerID="09643b155ae61d3db212a12fe01e8b94e6d8211a3180916351b902bcce2dad97" Oct 30 19:14:45.123456 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:45.123418 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09643b155ae61d3db212a12fe01e8b94e6d8211a3180916351b902bcce2dad97"} err="failed to get container status \"09643b155ae61d3db212a12fe01e8b94e6d8211a3180916351b902bcce2dad97\": rpc error: code = NotFound desc = could not find container \"09643b155ae61d3db212a12fe01e8b94e6d8211a3180916351b902bcce2dad97\": container with ID starting with 09643b155ae61d3db212a12fe01e8b94e6d8211a3180916351b902bcce2dad97 not found: ID does not exist" Oct 30 19:14:45.125085 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:45.124932 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfbb6b47-1274-4d3d-8845-627898246065-kube-api-access-g5jhm" pod "bfbb6b47-1274-4d3d-8845-627898246065" (UID: "bfbb6b47-1274-4d3d-8845-627898246065"). InnerVolumeSpecName "kube-api-access-g5jhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:45.125838 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:45.125804 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfbb6b47-1274-4d3d-8845-627898246065-istio-envoy" pod "bfbb6b47-1274-4d3d-8845-627898246065" (UID: "bfbb6b47-1274-4d3d-8845-627898246065"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:45.126694 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:45.126662 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfbb6b47-1274-4d3d-8845-627898246065-istio-token" pod "bfbb6b47-1274-4d3d-8845-627898246065" (UID: "bfbb6b47-1274-4d3d-8845-627898246065"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:45.126839 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:45.126758 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfbb6b47-1274-4d3d-8845-627898246065-istio-podinfo" pod "bfbb6b47-1274-4d3d-8845-627898246065" (UID: "bfbb6b47-1274-4d3d-8845-627898246065"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:14:45.227106 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:45.226070 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5jhm\" (UniqueName: \"kubernetes.io/projected/bfbb6b47-1274-4d3d-8845-627898246065-kube-api-access-g5jhm\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:45.227106 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:45.226109 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfbb6b47-1274-4d3d-8845-627898246065-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:45.227106 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:45.226124 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfbb6b47-1274-4d3d-8845-627898246065-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:45.227106 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:45.226160 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfbb6b47-1274-4d3d-8845-627898246065-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:45.227106 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:45.226172 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfbb6b47-1274-4d3d-8845-627898246065-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:45.439832 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:45.439794 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs"] Oct 30 19:14:45.444317 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:45.444290 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-236fc4-openshift-default-7f5d4cd5f9-mx2gs"] Oct 30 19:14:46.719970 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:46.719946 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-236fc4-88f5755fb-fskbl" Oct 30 19:14:46.765547 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:46.765520 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-236fc4-6c5fd847b7-xxv8l" Oct 30 19:14:46.870000 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:46.869970 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97f2a51a-33c7-4869-8315-f5c266adac7a-kube-api-access-g9gtn\" (UniqueName: \"kubernetes.io/projected/97f2a51a-33c7-4869-8315-f5c266adac7a-kube-api-access-g9gtn\") pod \"97f2a51a-33c7-4869-8315-f5c266adac7a\" (UID: \"97f2a51a-33c7-4869-8315-f5c266adac7a\") " Oct 30 19:14:46.870175 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:46.870053 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4f4443c-7b80-47d0-8bbf-3e6ef11f437b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4f4443c-7b80-47d0-8bbf-3e6ef11f437b-plugins-config-volume\") pod \"e4f4443c-7b80-47d0-8bbf-3e6ef11f437b\" (UID: \"e4f4443c-7b80-47d0-8bbf-3e6ef11f437b\") " Oct 30 19:14:46.870175 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:46.870106 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4f4443c-7b80-47d0-8bbf-3e6ef11f437b-kube-api-access-r5c22\" (UniqueName: \"kubernetes.io/projected/e4f4443c-7b80-47d0-8bbf-3e6ef11f437b-kube-api-access-r5c22\") pod \"e4f4443c-7b80-47d0-8bbf-3e6ef11f437b\" (UID: \"e4f4443c-7b80-47d0-8bbf-3e6ef11f437b\") " Oct 30 19:14:46.870293 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:46.870172 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97f2a51a-33c7-4869-8315-f5c266adac7a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97f2a51a-33c7-4869-8315-f5c266adac7a-plugins-config-volume\") pod \"97f2a51a-33c7-4869-8315-f5c266adac7a\" (UID: \"97f2a51a-33c7-4869-8315-f5c266adac7a\") " Oct 30 19:14:46.870594 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:46.870437 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4f4443c-7b80-47d0-8bbf-3e6ef11f437b-plugins-config-volume" pod "e4f4443c-7b80-47d0-8bbf-3e6ef11f437b" (UID: "e4f4443c-7b80-47d0-8bbf-3e6ef11f437b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:14:46.870731 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:46.870618 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97f2a51a-33c7-4869-8315-f5c266adac7a-plugins-config-volume" pod "97f2a51a-33c7-4869-8315-f5c266adac7a" (UID: "97f2a51a-33c7-4869-8315-f5c266adac7a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:14:46.872482 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:46.872457 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f2a51a-33c7-4869-8315-f5c266adac7a-kube-api-access-g9gtn" pod "97f2a51a-33c7-4869-8315-f5c266adac7a" (UID: "97f2a51a-33c7-4869-8315-f5c266adac7a"). InnerVolumeSpecName "kube-api-access-g9gtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:46.872482 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:46.872476 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4f4443c-7b80-47d0-8bbf-3e6ef11f437b-kube-api-access-r5c22" pod "e4f4443c-7b80-47d0-8bbf-3e6ef11f437b" (UID: "e4f4443c-7b80-47d0-8bbf-3e6ef11f437b"). InnerVolumeSpecName "kube-api-access-r5c22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:46.971802 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:46.971725 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5c22\" (UniqueName: \"kubernetes.io/projected/e4f4443c-7b80-47d0-8bbf-3e6ef11f437b-kube-api-access-r5c22\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:46.971802 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:46.971752 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97f2a51a-33c7-4869-8315-f5c266adac7a-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:46.971802 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:46.971765 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9gtn\" (UniqueName: \"kubernetes.io/projected/97f2a51a-33c7-4869-8315-f5c266adac7a-kube-api-access-g9gtn\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:46.971802 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:46.971775 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4f4443c-7b80-47d0-8bbf-3e6ef11f437b-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:47.064749 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:47.064712 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfbb6b47-1274-4d3d-8845-627898246065" path="/var/lib/kubelet/pods/bfbb6b47-1274-4d3d-8845-627898246065/volumes" Oct 30 19:14:47.065384 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:47.065357 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cecb19a3-wjgv7"] Oct 30 19:14:47.074974 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:47.074949 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cecb19a3-wjgv7"] Oct 30 19:14:47.108581 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:47.108546 4442 generic.go:356] "Generic (PLEG): container finished" podID="97f2a51a-33c7-4869-8315-f5c266adac7a" containerID="d615caf88a30f4ad70b0e2c68c2b7c267d67ab7f45e8558f11be071a3f15430b" exitCode=0 Oct 30 19:14:47.108751 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:47.108597 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-236fc4-6c5fd847b7-xxv8l" Oct 30 19:14:47.108751 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:47.108634 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-236fc4-6c5fd847b7-xxv8l" event={"ID":"97f2a51a-33c7-4869-8315-f5c266adac7a","Type":"ContainerDied","Data":"d615caf88a30f4ad70b0e2c68c2b7c267d67ab7f45e8558f11be071a3f15430b"} Oct 30 19:14:47.108751 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:47.108690 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-236fc4-6c5fd847b7-xxv8l" event={"ID":"97f2a51a-33c7-4869-8315-f5c266adac7a","Type":"ContainerDied","Data":"fc03a88338214590050a5b2be26a0bd429ebf3936716a8c2c5d43996bff5eda3"} Oct 30 19:14:47.108751 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:47.108722 4442 scope.go:122] "RemoveContainer" containerID="d615caf88a30f4ad70b0e2c68c2b7c267d67ab7f45e8558f11be071a3f15430b" Oct 30 19:14:47.110746 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:47.110712 4442 generic.go:356] "Generic (PLEG): container finished" podID="e4f4443c-7b80-47d0-8bbf-3e6ef11f437b" containerID="07610df0022ae7c8f2f3cdddea980b64fac326104d01edae512e864f05bfd075" exitCode=0 Oct 30 19:14:47.110872 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:47.110781 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-236fc4-88f5755fb-fskbl" Oct 30 19:14:47.110872 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:47.110852 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-236fc4-88f5755fb-fskbl" event={"ID":"e4f4443c-7b80-47d0-8bbf-3e6ef11f437b","Type":"ContainerDied","Data":"07610df0022ae7c8f2f3cdddea980b64fac326104d01edae512e864f05bfd075"} Oct 30 19:14:47.111004 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:47.110905 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-236fc4-88f5755fb-fskbl" event={"ID":"e4f4443c-7b80-47d0-8bbf-3e6ef11f437b","Type":"ContainerDied","Data":"5c81304d529d9687312b2f6a25a169a5c11e549d683f944d07b7bcceddad696e"} Oct 30 19:14:47.119626 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:47.119604 4442 scope.go:122] "RemoveContainer" containerID="d615caf88a30f4ad70b0e2c68c2b7c267d67ab7f45e8558f11be071a3f15430b" Oct 30 19:14:47.119962 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:14:47.119936 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d615caf88a30f4ad70b0e2c68c2b7c267d67ab7f45e8558f11be071a3f15430b\": container with ID starting with d615caf88a30f4ad70b0e2c68c2b7c267d67ab7f45e8558f11be071a3f15430b not found: ID does not exist" containerID="d615caf88a30f4ad70b0e2c68c2b7c267d67ab7f45e8558f11be071a3f15430b" Oct 30 19:14:47.120072 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:47.119970 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d615caf88a30f4ad70b0e2c68c2b7c267d67ab7f45e8558f11be071a3f15430b"} err="failed to get container status \"d615caf88a30f4ad70b0e2c68c2b7c267d67ab7f45e8558f11be071a3f15430b\": rpc error: code = NotFound desc = could not find container \"d615caf88a30f4ad70b0e2c68c2b7c267d67ab7f45e8558f11be071a3f15430b\": container with ID starting with d615caf88a30f4ad70b0e2c68c2b7c267d67ab7f45e8558f11be071a3f15430b not found: ID does not exist" Oct 30 19:14:47.120072 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:47.119988 4442 scope.go:122] "RemoveContainer" containerID="07610df0022ae7c8f2f3cdddea980b64fac326104d01edae512e864f05bfd075" Oct 30 19:14:47.131419 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:47.131396 4442 scope.go:122] "RemoveContainer" containerID="07610df0022ae7c8f2f3cdddea980b64fac326104d01edae512e864f05bfd075" Oct 30 19:14:47.131783 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:14:47.131758 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07610df0022ae7c8f2f3cdddea980b64fac326104d01edae512e864f05bfd075\": container with ID starting with 07610df0022ae7c8f2f3cdddea980b64fac326104d01edae512e864f05bfd075 not found: ID does not exist" containerID="07610df0022ae7c8f2f3cdddea980b64fac326104d01edae512e864f05bfd075" Oct 30 19:14:47.131883 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:47.131795 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07610df0022ae7c8f2f3cdddea980b64fac326104d01edae512e864f05bfd075"} err="failed to get container status \"07610df0022ae7c8f2f3cdddea980b64fac326104d01edae512e864f05bfd075\": rpc error: code = NotFound desc = could not find container \"07610df0022ae7c8f2f3cdddea980b64fac326104d01edae512e864f05bfd075\": container with ID starting with 07610df0022ae7c8f2f3cdddea980b64fac326104d01edae512e864f05bfd075 not found: ID does not exist" Oct 30 19:14:47.175248 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:47.175104 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-236fc4-88f5755fb-fskbl"] Oct 30 19:14:47.183180 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:47.183151 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-236fc4-88f5755fb-fskbl"] Oct 30 19:14:47.231198 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:47.231065 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-236fc4-6c5fd847b7-xxv8l"] Oct 30 19:14:47.234602 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:47.234567 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-236fc4-6c5fd847b7-xxv8l"] Oct 30 19:14:47.560848 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:14:47.560752 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 19:14:49.064095 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:49.064061 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97f2a51a-33c7-4869-8315-f5c266adac7a" path="/var/lib/kubelet/pods/97f2a51a-33c7-4869-8315-f5c266adac7a/volumes" Oct 30 19:14:49.064570 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:49.064551 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aff128e4-d366-47f4-bcf9-ba1c133122f4" path="/var/lib/kubelet/pods/aff128e4-d366-47f4-bcf9-ba1c133122f4/volumes" Oct 30 19:14:49.064894 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:49.064881 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4f4443c-7b80-47d0-8bbf-3e6ef11f437b" path="/var/lib/kubelet/pods/e4f4443c-7b80-47d0-8bbf-3e6ef11f437b/volumes" Oct 30 19:14:50.788297 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:50.788268 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c095fa-7ff45cdc78-lvpxv"] Oct 30 19:14:50.828049 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:50.828013 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c095fa-5d479bd6f9-rgj6w"] Oct 30 19:14:54.176966 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:54.176931 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t"] Oct 30 19:14:54.177689 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:54.177213 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" podUID="6cbde8cc-5ae5-4133-9c52-cb30a81b91b1" containerName="istio-proxy" containerID="cri-o://3d88c64e63c7aeedd0ed117f78b4cc8caa5d73d92f12d22edd5add46abb0e473" gracePeriod=30 Oct 30 19:14:55.789572 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:55.789528 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c095fa-7ff45cdc78-lvpxv" podUID="979a6818-4b68-4229-86e5-c11fdd1586bd" containerName="payload-processing" containerID="cri-o://4e28fba4ba4a8187c95d2d0115df7b6eef83539475cce4a4d812515bd5a4f43f" gracePeriod=25 Oct 30 19:14:55.828644 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:55.828595 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c095fa-5d479bd6f9-rgj6w" podUID="b9e66197-f5b9-4ea3-8fa6-d52647e45327" containerName="payload-pre-processing" containerID="cri-o://b8439f6f2605eed5e5db2887dfbddd74528a4337fe7ac304ea75cf9b7bea2e30" gracePeriod=25 Oct 30 19:14:56.567763 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:56.567725 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-09ef1d-4fdhh"] Oct 30 19:14:56.578901 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:56.578859 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-09ef1d-4fdhh"] Oct 30 19:14:57.064960 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:57.064920 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd074006-b670-47f8-a498-758e1dbe6991" path="/var/lib/kubelet/pods/dd074006-b670-47f8-a498-758e1dbe6991/volumes" Oct 30 19:14:57.108891 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:57.108862 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c095fa-p6zcn"] Oct 30 19:14:57.120467 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:57.120440 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c095fa-p6zcn"] Oct 30 19:14:59.064565 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:59.064530 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="512ab6d5-f111-4957-bb85-b9373f3cb13d" path="/var/lib/kubelet/pods/512ab6d5-f111-4957-bb85-b9373f3cb13d/volumes" Oct 30 19:14:59.444242 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:59.444213 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" Oct 30 19:14:59.539171 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:59.539126 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-istio-podinfo\") pod \"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1\" (UID: \"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1\") " Oct 30 19:14:59.539370 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:59.539185 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-istio-envoy\") pod \"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1\" (UID: \"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1\") " Oct 30 19:14:59.539370 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:59.539237 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-istio-token\" (UniqueName: \"kubernetes.io/projected/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-istio-token\") pod \"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1\" (UID: \"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1\") " Oct 30 19:14:59.539370 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:59.539322 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-istiod-ca-cert\") pod \"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1\" (UID: \"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1\") " Oct 30 19:14:59.539370 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:59.539361 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-workload-certs\") pod \"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1\" (UID: \"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1\") " Oct 30 19:14:59.539598 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:59.539405 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-kube-api-access-whlcf\" (UniqueName: \"kubernetes.io/projected/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-kube-api-access-whlcf\") pod \"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1\" (UID: \"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1\") " Oct 30 19:14:59.539598 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:59.539458 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-credential-socket\") pod \"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1\" (UID: \"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1\") " Oct 30 19:14:59.539598 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:59.539488 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-istio-data\") pod \"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1\" (UID: \"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1\") " Oct 30 19:14:59.539598 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:59.539570 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-workload-socket\") pod \"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1\" (UID: \"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1\") " Oct 30 19:14:59.539810 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:59.539713 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-workload-certs" pod "6cbde8cc-5ae5-4133-9c52-cb30a81b91b1" (UID: "6cbde8cc-5ae5-4133-9c52-cb30a81b91b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:59.539878 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:59.539829 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-credential-socket" pod "6cbde8cc-5ae5-4133-9c52-cb30a81b91b1" (UID: "6cbde8cc-5ae5-4133-9c52-cb30a81b91b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:59.540222 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:59.539981 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-workload-socket" pod "6cbde8cc-5ae5-4133-9c52-cb30a81b91b1" (UID: "6cbde8cc-5ae5-4133-9c52-cb30a81b91b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:59.540222 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:59.540180 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-istio-data" pod "6cbde8cc-5ae5-4133-9c52-cb30a81b91b1" (UID: "6cbde8cc-5ae5-4133-9c52-cb30a81b91b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:59.540222 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:59.540194 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-istiod-ca-cert" pod "6cbde8cc-5ae5-4133-9c52-cb30a81b91b1" (UID: "6cbde8cc-5ae5-4133-9c52-cb30a81b91b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:14:59.540462 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:59.540243 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:59.540462 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:59.540265 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:59.540462 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:59.540280 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:59.542445 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:59.542409 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-kube-api-access-whlcf" pod "6cbde8cc-5ae5-4133-9c52-cb30a81b91b1" (UID: "6cbde8cc-5ae5-4133-9c52-cb30a81b91b1"). InnerVolumeSpecName "kube-api-access-whlcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:59.542910 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:59.542887 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-istio-token" pod "6cbde8cc-5ae5-4133-9c52-cb30a81b91b1" (UID: "6cbde8cc-5ae5-4133-9c52-cb30a81b91b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:59.543299 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:59.543279 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-istio-podinfo" pod "6cbde8cc-5ae5-4133-9c52-cb30a81b91b1" (UID: "6cbde8cc-5ae5-4133-9c52-cb30a81b91b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:14:59.544293 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:59.544273 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-istio-envoy" pod "6cbde8cc-5ae5-4133-9c52-cb30a81b91b1" (UID: "6cbde8cc-5ae5-4133-9c52-cb30a81b91b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:59.631643 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:59.631603 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c095fa-7ff45cdc78-lvpxv" podUID="979a6818-4b68-4229-86e5-c11fdd1586bd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:14:59.632255 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:59.632209 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c095fa-5d479bd6f9-rgj6w" podUID="b9e66197-f5b9-4ea3-8fa6-d52647e45327" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:14:59.641630 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:59.641598 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:59.641630 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:59.641625 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:59.641630 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:59.641636 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whlcf\" (UniqueName: \"kubernetes.io/projected/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-kube-api-access-whlcf\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:59.641858 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:59.641645 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:59.641858 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:59.641654 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:14:59.641858 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:14:59.641662 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:00.223314 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:00.223277 4442 generic.go:356] "Generic (PLEG): container finished" podID="6cbde8cc-5ae5-4133-9c52-cb30a81b91b1" containerID="3d88c64e63c7aeedd0ed117f78b4cc8caa5d73d92f12d22edd5add46abb0e473" exitCode=0 Oct 30 19:15:00.223791 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:00.223331 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" Oct 30 19:15:00.223791 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:00.223362 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" event={"ID":"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1","Type":"ContainerDied","Data":"3d88c64e63c7aeedd0ed117f78b4cc8caa5d73d92f12d22edd5add46abb0e473"} Oct 30 19:15:00.223791 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:00.223428 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t" event={"ID":"6cbde8cc-5ae5-4133-9c52-cb30a81b91b1","Type":"ContainerDied","Data":"ffb396db5245ceb20545af738d074ce0e96559fb4ae16efb450fc1035a3c89ae"} Oct 30 19:15:00.223791 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:00.223491 4442 scope.go:122] "RemoveContainer" containerID="3d88c64e63c7aeedd0ed117f78b4cc8caa5d73d92f12d22edd5add46abb0e473" Oct 30 19:15:00.237744 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:00.237631 4442 scope.go:122] "RemoveContainer" containerID="c888cfd579cfb4cbc1e6dfe502cd0317d0e214dfc0a411ed6737b1d2ce023bc1" Oct 30 19:15:00.238895 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:00.238757 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889795-ljkwb"] Oct 30 19:15:00.239749 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:00.239728 4442 cpu_manager.go:414] "containerMap: removing container" podUID="e4f4443c-7b80-47d0-8bbf-3e6ef11f437b" containerName="payload-processing" Oct 30 19:15:00.239900 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:00.239884 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4f4443c-7b80-47d0-8bbf-3e6ef11f437b" containerName="payload-processing" Oct 30 19:15:00.240019 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:00.240003 4442 cpu_manager.go:414] "containerMap: removing container" podUID="bfbb6b47-1274-4d3d-8845-627898246065" containerName="istio-proxy" Oct 30 19:15:00.240090 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:00.240018 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfbb6b47-1274-4d3d-8845-627898246065" containerName="istio-proxy" Oct 30 19:15:00.240090 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:00.240054 4442 cpu_manager.go:414] "containerMap: removing container" podUID="6cbde8cc-5ae5-4133-9c52-cb30a81b91b1" containerName="istio-proxy" Oct 30 19:15:00.240090 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:00.240062 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cbde8cc-5ae5-4133-9c52-cb30a81b91b1" containerName="istio-proxy" Oct 30 19:15:00.240090 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:00.240076 4442 cpu_manager.go:414] "containerMap: removing container" podUID="bfbb6b47-1274-4d3d-8845-627898246065" containerName="istio-proxy" Oct 30 19:15:00.240090 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:00.240083 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfbb6b47-1274-4d3d-8845-627898246065" containerName="istio-proxy" Oct 30 19:15:00.240389 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:00.240119 4442 cpu_manager.go:414] "containerMap: removing container" podUID="512ab6d5-f111-4957-bb85-b9373f3cb13d" containerName="revoke-keys" Oct 30 19:15:00.240389 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:00.240143 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="512ab6d5-f111-4957-bb85-b9373f3cb13d" containerName="revoke-keys" Oct 30 19:15:00.240389 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:00.240174 4442 cpu_manager.go:414] "containerMap: removing container" podUID="97f2a51a-33c7-4869-8315-f5c266adac7a" containerName="payload-pre-processing" Oct 30 19:15:00.240389 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:00.240181 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97f2a51a-33c7-4869-8315-f5c266adac7a" containerName="payload-pre-processing" Oct 30 19:15:00.240389 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:00.240314 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cbde8cc-5ae5-4133-9c52-cb30a81b91b1" containerName="istio-proxy" Oct 30 19:15:00.240389 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:00.240326 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfbb6b47-1274-4d3d-8845-627898246065" containerName="istio-proxy" Oct 30 19:15:00.240389 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:00.240341 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="512ab6d5-f111-4957-bb85-b9373f3cb13d" containerName="revoke-keys" Oct 30 19:15:00.240389 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:00.240356 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="97f2a51a-33c7-4869-8315-f5c266adac7a" containerName="payload-pre-processing" Oct 30 19:15:00.240389 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:00.240369 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4f4443c-7b80-47d0-8bbf-3e6ef11f437b" containerName="payload-processing" Oct 30 19:15:00.245186 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:00.245167 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889795-ljkwb" Oct 30 19:15:00.254549 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:00.252327 4442 scope.go:122] "RemoveContainer" containerID="3d88c64e63c7aeedd0ed117f78b4cc8caa5d73d92f12d22edd5add46abb0e473" Oct 30 19:15:00.255965 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:15:00.255228 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d88c64e63c7aeedd0ed117f78b4cc8caa5d73d92f12d22edd5add46abb0e473\": container with ID starting with 3d88c64e63c7aeedd0ed117f78b4cc8caa5d73d92f12d22edd5add46abb0e473 not found: ID does not exist" containerID="3d88c64e63c7aeedd0ed117f78b4cc8caa5d73d92f12d22edd5add46abb0e473" Oct 30 19:15:00.255965 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:00.255302 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d88c64e63c7aeedd0ed117f78b4cc8caa5d73d92f12d22edd5add46abb0e473"} err="failed to get container status \"3d88c64e63c7aeedd0ed117f78b4cc8caa5d73d92f12d22edd5add46abb0e473\": rpc error: code = NotFound desc = could not find container \"3d88c64e63c7aeedd0ed117f78b4cc8caa5d73d92f12d22edd5add46abb0e473\": container with ID starting with 3d88c64e63c7aeedd0ed117f78b4cc8caa5d73d92f12d22edd5add46abb0e473 not found: ID does not exist" Oct 30 19:15:00.255965 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:00.255328 4442 scope.go:122] "RemoveContainer" containerID="c888cfd579cfb4cbc1e6dfe502cd0317d0e214dfc0a411ed6737b1d2ce023bc1" Oct 30 19:15:00.255965 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:15:00.255620 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c888cfd579cfb4cbc1e6dfe502cd0317d0e214dfc0a411ed6737b1d2ce023bc1\": container with ID starting with c888cfd579cfb4cbc1e6dfe502cd0317d0e214dfc0a411ed6737b1d2ce023bc1 not found: ID does not exist" containerID="c888cfd579cfb4cbc1e6dfe502cd0317d0e214dfc0a411ed6737b1d2ce023bc1" Oct 30 19:15:00.255965 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:00.255652 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c888cfd579cfb4cbc1e6dfe502cd0317d0e214dfc0a411ed6737b1d2ce023bc1"} err="failed to get container status \"c888cfd579cfb4cbc1e6dfe502cd0317d0e214dfc0a411ed6737b1d2ce023bc1\": rpc error: code = NotFound desc = could not find container \"c888cfd579cfb4cbc1e6dfe502cd0317d0e214dfc0a411ed6737b1d2ce023bc1\": container with ID starting with c888cfd579cfb4cbc1e6dfe502cd0317d0e214dfc0a411ed6737b1d2ce023bc1 not found: ID does not exist" Oct 30 19:15:00.269261 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:00.269240 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889795-ljkwb"] Oct 30 19:15:00.298561 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:00.298530 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t"] Oct 30 19:15:00.301939 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:00.301915 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-kmj4t"] Oct 30 19:15:00.323805 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:00.323775 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:15:00.348759 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:00.348720 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07aa8798-3d58-4d0d-b33e-715d8871c259-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-45dc34-29889795-ljkwb\" (UID: \"07aa8798-3d58-4d0d-b33e-715d8871c259\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889795-ljkwb" Oct 30 19:15:00.348948 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:00.348896 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt2ms\" (UniqueName: \"kubernetes.io/projected/07aa8798-3d58-4d0d-b33e-715d8871c259-kube-api-access-rt2ms\") pod \"maas-api-key-cleanup-e2e-worker-w2-45dc34-29889795-ljkwb\" (UID: \"07aa8798-3d58-4d0d-b33e-715d8871c259\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889795-ljkwb" Oct 30 19:15:00.450057 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:00.450014 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07aa8798-3d58-4d0d-b33e-715d8871c259-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-45dc34-29889795-ljkwb\" (UID: \"07aa8798-3d58-4d0d-b33e-715d8871c259\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889795-ljkwb" Oct 30 19:15:00.450303 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:00.450122 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt2ms\" (UniqueName: \"kubernetes.io/projected/07aa8798-3d58-4d0d-b33e-715d8871c259-kube-api-access-rt2ms\") pod \"maas-api-key-cleanup-e2e-worker-w2-45dc34-29889795-ljkwb\" (UID: \"07aa8798-3d58-4d0d-b33e-715d8871c259\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889795-ljkwb" Oct 30 19:15:00.450746 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:00.450715 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07aa8798-3d58-4d0d-b33e-715d8871c259-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-45dc34-29889795-ljkwb\" (UID: \"07aa8798-3d58-4d0d-b33e-715d8871c259\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889795-ljkwb" Oct 30 19:15:00.472201 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:00.472172 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt2ms\" (UniqueName: \"kubernetes.io/projected/07aa8798-3d58-4d0d-b33e-715d8871c259-kube-api-access-rt2ms\") pod \"maas-api-key-cleanup-e2e-worker-w2-45dc34-29889795-ljkwb\" (UID: \"07aa8798-3d58-4d0d-b33e-715d8871c259\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889795-ljkwb" Oct 30 19:15:00.565751 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:00.565661 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889795-ljkwb" Oct 30 19:15:00.866552 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:00.866521 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889795-ljkwb"] Oct 30 19:15:00.871888 ip-10-0-138-176 kubenswrapper[4442]: W1030 19:15:00.871858 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07aa8798_3d58_4d0d_b33e_715d8871c259.slice/crio-20963b357e4aca5f38b0556e901a9bcd0c5c91fe08b35c2fbadf648dea16bd78 WatchSource:0}: Error finding container 20963b357e4aca5f38b0556e901a9bcd0c5c91fe08b35c2fbadf648dea16bd78: Status 404 returned error can't find the container with id 20963b357e4aca5f38b0556e901a9bcd0c5c91fe08b35c2fbadf648dea16bd78 Oct 30 19:15:01.065046 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:01.065015 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cbde8cc-5ae5-4133-9c52-cb30a81b91b1" path="/var/lib/kubelet/pods/6cbde8cc-5ae5-4133-9c52-cb30a81b91b1/volumes" Oct 30 19:15:01.234071 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:01.234033 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889795-ljkwb" event={"ID":"07aa8798-3d58-4d0d-b33e-715d8871c259","Type":"ContainerStarted","Data":"4b00969fd254e57ed9139978df56030432b0b5382735d64d82d7a6b7e155a4cc"} Oct 30 19:15:01.234618 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:01.234099 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889795-ljkwb" event={"ID":"07aa8798-3d58-4d0d-b33e-715d8871c259","Type":"ContainerStarted","Data":"20963b357e4aca5f38b0556e901a9bcd0c5c91fe08b35c2fbadf648dea16bd78"} Oct 30 19:15:01.272819 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:01.272762 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889795-ljkwb" podStartSLOduration=1.272746441 podStartE2EDuration="1.272746441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:01.271840155 +0000 UTC m=+4598.361902494" watchObservedRunningTime="2026-10-30 19:15:01.272746441 +0000 UTC m=+4598.362808828" Oct 30 19:15:02.246080 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:02.246039 4442 generic.go:356] "Generic (PLEG): container finished" podID="07aa8798-3d58-4d0d-b33e-715d8871c259" containerID="4b00969fd254e57ed9139978df56030432b0b5382735d64d82d7a6b7e155a4cc" exitCode=60 Oct 30 19:15:02.246548 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:02.246127 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889795-ljkwb" event={"ID":"07aa8798-3d58-4d0d-b33e-715d8871c259","Type":"ContainerDied","Data":"4b00969fd254e57ed9139978df56030432b0b5382735d64d82d7a6b7e155a4cc"} Oct 30 19:15:02.246548 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:02.246444 4442 scope.go:122] "RemoveContainer" containerID="4b00969fd254e57ed9139978df56030432b0b5382735d64d82d7a6b7e155a4cc" Oct 30 19:15:03.036513 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:03.036483 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b49e554a-btjhm"] Oct 30 19:15:03.048621 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:03.048588 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b49e554a-btjhm"] Oct 30 19:15:03.075181 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:03.075123 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f89327e4-347c-4656-89eb-5674c31f9ec8" path="/var/lib/kubelet/pods/f89327e4-347c-4656-89eb-5674c31f9ec8/volumes" Oct 30 19:15:03.256305 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:03.256264 4442 generic.go:356] "Generic (PLEG): container finished" podID="07aa8798-3d58-4d0d-b33e-715d8871c259" containerID="e7935d73730c5e418da03ca355810516221ac6e2da3213b6320a306b391e2666" exitCode=60 Oct 30 19:15:03.256770 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:03.256351 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889795-ljkwb" event={"ID":"07aa8798-3d58-4d0d-b33e-715d8871c259","Type":"ContainerDied","Data":"e7935d73730c5e418da03ca355810516221ac6e2da3213b6320a306b391e2666"} Oct 30 19:15:03.256770 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:03.256411 4442 scope.go:122] "RemoveContainer" containerID="4b00969fd254e57ed9139978df56030432b0b5382735d64d82d7a6b7e155a4cc" Oct 30 19:15:03.256770 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:03.256664 4442 scope.go:122] "RemoveContainer" containerID="e7935d73730c5e418da03ca355810516221ac6e2da3213b6320a306b391e2666" Oct 30 19:15:03.256995 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:15:03.256967 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-45dc34-29889795-ljkwb_odh-ai-gateway-infra(07aa8798-3d58-4d0d-b33e-715d8871c259)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889795-ljkwb" podUID="07aa8798-3d58-4d0d-b33e-715d8871c259" Oct 30 19:15:04.266302 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:04.266271 4442 scope.go:122] "RemoveContainer" containerID="e7935d73730c5e418da03ca355810516221ac6e2da3213b6320a306b391e2666" Oct 30 19:15:04.266729 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:15:04.266565 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-45dc34-29889795-ljkwb_odh-ai-gateway-infra(07aa8798-3d58-4d0d-b33e-715d8871c259)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889795-ljkwb" podUID="07aa8798-3d58-4d0d-b33e-715d8871c259" Oct 30 19:15:04.399105 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:04.399076 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk"] Oct 30 19:15:04.399355 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:04.399331 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" podUID="2c46403e-a9d6-48ca-ad0e-34533d5b22ab" containerName="istio-proxy" containerID="cri-o://74aba9be06e1f2a0300e6db926ed683c3f9f22280513bee037595b8d6e8f8481" gracePeriod=30 Oct 30 19:15:05.487879 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:05.487847 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp"] Oct 30 19:15:05.488399 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:05.488126 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" podUID="425bc56e-e6f2-400d-a6dc-a277201acb8f" containerName="istio-proxy" containerID="cri-o://4f65edc89f7838267e2c63af4430fda749a05285dc4b8bfc9c0576835efb71bc" gracePeriod=30 Oct 30 19:15:06.219099 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:06.219056 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-921450-mtrn6"] Oct 30 19:15:06.219637 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:06.219618 4442 cpu_manager.go:414] "containerMap: removing container" podUID="6cbde8cc-5ae5-4133-9c52-cb30a81b91b1" containerName="istio-proxy" Oct 30 19:15:06.219745 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:06.219638 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cbde8cc-5ae5-4133-9c52-cb30a81b91b1" containerName="istio-proxy" Oct 30 19:15:06.219812 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:06.219798 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cbde8cc-5ae5-4133-9c52-cb30a81b91b1" containerName="istio-proxy" Oct 30 19:15:06.219868 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:06.219812 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfbb6b47-1274-4d3d-8845-627898246065" containerName="istio-proxy" Oct 30 19:15:06.223346 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:06.223323 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-921450-mtrn6" Oct 30 19:15:06.258005 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:06.257971 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-921450-mtrn6"] Oct 30 19:15:06.272156 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:06.272098 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:15:06.315788 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:06.315751 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4c7339b-0d56-4483-8323-2b9d0059f21d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-921450-mtrn6\" (UID: \"e4c7339b-0d56-4483-8323-2b9d0059f21d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-921450-mtrn6" Oct 30 19:15:06.315983 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:06.315896 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4c7339b-0d56-4483-8323-2b9d0059f21d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-921450-mtrn6\" (UID: \"e4c7339b-0d56-4483-8323-2b9d0059f21d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-921450-mtrn6" Oct 30 19:15:06.316041 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:06.316026 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whbf8\" (UniqueName: \"kubernetes.io/projected/e4c7339b-0d56-4483-8323-2b9d0059f21d-kube-api-access-whbf8\") pod \"maas-api-revoke-keys-e2e-worker-w0-921450-mtrn6\" (UID: \"e4c7339b-0d56-4483-8323-2b9d0059f21d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-921450-mtrn6" Oct 30 19:15:06.417504 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:06.417457 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whbf8\" (UniqueName: \"kubernetes.io/projected/e4c7339b-0d56-4483-8323-2b9d0059f21d-kube-api-access-whbf8\") pod \"maas-api-revoke-keys-e2e-worker-w0-921450-mtrn6\" (UID: \"e4c7339b-0d56-4483-8323-2b9d0059f21d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-921450-mtrn6" Oct 30 19:15:06.417728 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:06.417546 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4c7339b-0d56-4483-8323-2b9d0059f21d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-921450-mtrn6\" (UID: \"e4c7339b-0d56-4483-8323-2b9d0059f21d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-921450-mtrn6" Oct 30 19:15:06.417728 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:06.417651 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4c7339b-0d56-4483-8323-2b9d0059f21d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-921450-mtrn6\" (UID: \"e4c7339b-0d56-4483-8323-2b9d0059f21d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-921450-mtrn6" Oct 30 19:15:06.418250 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:06.418227 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4c7339b-0d56-4483-8323-2b9d0059f21d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-921450-mtrn6\" (UID: \"e4c7339b-0d56-4483-8323-2b9d0059f21d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-921450-mtrn6" Oct 30 19:15:06.420357 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:06.420331 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4c7339b-0d56-4483-8323-2b9d0059f21d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-921450-mtrn6\" (UID: \"e4c7339b-0d56-4483-8323-2b9d0059f21d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-921450-mtrn6" Oct 30 19:15:06.436946 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:06.436923 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whbf8\" (UniqueName: \"kubernetes.io/projected/e4c7339b-0d56-4483-8323-2b9d0059f21d-kube-api-access-whbf8\") pod \"maas-api-revoke-keys-e2e-worker-w0-921450-mtrn6\" (UID: \"e4c7339b-0d56-4483-8323-2b9d0059f21d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-921450-mtrn6" Oct 30 19:15:06.541674 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:06.541568 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-921450-mtrn6" Oct 30 19:15:06.842820 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:06.842791 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-921450-mtrn6"] Oct 30 19:15:06.845423 ip-10-0-138-176 kubenswrapper[4442]: W1030 19:15:06.845387 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4c7339b_0d56_4483_8323_2b9d0059f21d.slice/crio-294d39e0557bb321b60e13a76d3c495b0702f3be2b68827462391fcae6fd6b52 WatchSource:0}: Error finding container 294d39e0557bb321b60e13a76d3c495b0702f3be2b68827462391fcae6fd6b52: Status 404 returned error can't find the container with id 294d39e0557bb321b60e13a76d3c495b0702f3be2b68827462391fcae6fd6b52 Oct 30 19:15:07.298160 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:07.292401 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-921450-mtrn6" event={"ID":"e4c7339b-0d56-4483-8323-2b9d0059f21d","Type":"ContainerStarted","Data":"4389019fae212a9c82b2695f23e6ef4b6c85d9db5586a3352e4633b1c2ccc071"} Oct 30 19:15:07.298160 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:07.292508 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-921450-mtrn6" event={"ID":"e4c7339b-0d56-4483-8323-2b9d0059f21d","Type":"ContainerStarted","Data":"294d39e0557bb321b60e13a76d3c495b0702f3be2b68827462391fcae6fd6b52"} Oct 30 19:15:07.331670 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:07.331599 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-921450-mtrn6" podStartSLOduration=1.331580188 podStartE2EDuration="1.331580188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:07.330562102 +0000 UTC m=+4604.420624441" watchObservedRunningTime="2026-10-30 19:15:07.331580188 +0000 UTC m=+4604.421642528" Oct 30 19:15:08.301624 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:08.301590 4442 generic.go:356] "Generic (PLEG): container finished" podID="e4c7339b-0d56-4483-8323-2b9d0059f21d" containerID="4389019fae212a9c82b2695f23e6ef4b6c85d9db5586a3352e4633b1c2ccc071" exitCode=0 Oct 30 19:15:08.302087 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:08.301730 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-921450-mtrn6" event={"ID":"e4c7339b-0d56-4483-8323-2b9d0059f21d","Type":"ContainerDied","Data":"4389019fae212a9c82b2695f23e6ef4b6c85d9db5586a3352e4633b1c2ccc071"} Oct 30 19:15:09.528350 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:09.528320 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-921450-mtrn6" Oct 30 19:15:09.631185 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:09.631112 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c095fa-7ff45cdc78-lvpxv" podUID="979a6818-4b68-4229-86e5-c11fdd1586bd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:15:09.631614 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:09.631586 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c095fa-5d479bd6f9-rgj6w" podUID="b9e66197-f5b9-4ea3-8fa6-d52647e45327" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:15:09.661256 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:09.661231 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" Oct 30 19:15:09.661402 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:09.661364 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4c7339b-0d56-4483-8323-2b9d0059f21d-kube-api-access-whbf8\" (UniqueName: \"kubernetes.io/projected/e4c7339b-0d56-4483-8323-2b9d0059f21d-kube-api-access-whbf8\") pod \"e4c7339b-0d56-4483-8323-2b9d0059f21d\" (UID: \"e4c7339b-0d56-4483-8323-2b9d0059f21d\") " Oct 30 19:15:09.661469 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:09.661453 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4c7339b-0d56-4483-8323-2b9d0059f21d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4c7339b-0d56-4483-8323-2b9d0059f21d-maas-api-xks-ca\") pod \"e4c7339b-0d56-4483-8323-2b9d0059f21d\" (UID: \"e4c7339b-0d56-4483-8323-2b9d0059f21d\") " Oct 30 19:15:09.661512 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:09.661496 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4c7339b-0d56-4483-8323-2b9d0059f21d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4c7339b-0d56-4483-8323-2b9d0059f21d-maas-api-openshift-ca\") pod \"e4c7339b-0d56-4483-8323-2b9d0059f21d\" (UID: \"e4c7339b-0d56-4483-8323-2b9d0059f21d\") " Oct 30 19:15:09.661897 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:09.661876 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4c7339b-0d56-4483-8323-2b9d0059f21d-maas-api-openshift-ca" pod "e4c7339b-0d56-4483-8323-2b9d0059f21d" (UID: "e4c7339b-0d56-4483-8323-2b9d0059f21d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:09.663786 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:09.663752 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4c7339b-0d56-4483-8323-2b9d0059f21d-maas-api-xks-ca" pod "e4c7339b-0d56-4483-8323-2b9d0059f21d" (UID: "e4c7339b-0d56-4483-8323-2b9d0059f21d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:15:09.663884 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:09.663785 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c7339b-0d56-4483-8323-2b9d0059f21d-kube-api-access-whbf8" pod "e4c7339b-0d56-4483-8323-2b9d0059f21d" (UID: "e4c7339b-0d56-4483-8323-2b9d0059f21d"). InnerVolumeSpecName "kube-api-access-whbf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:09.762660 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:09.762612 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-workload-socket\") pod \"2c46403e-a9d6-48ca-ad0e-34533d5b22ab\" (UID: \"2c46403e-a9d6-48ca-ad0e-34533d5b22ab\") " Oct 30 19:15:09.762863 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:09.762679 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-credential-socket\") pod \"2c46403e-a9d6-48ca-ad0e-34533d5b22ab\" (UID: \"2c46403e-a9d6-48ca-ad0e-34533d5b22ab\") " Oct 30 19:15:09.762863 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:09.762739 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-kube-api-access-ll4nq\" (UniqueName: \"kubernetes.io/projected/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-kube-api-access-ll4nq\") pod \"2c46403e-a9d6-48ca-ad0e-34533d5b22ab\" (UID: \"2c46403e-a9d6-48ca-ad0e-34533d5b22ab\") " Oct 30 19:15:09.762863 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:09.762795 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-istiod-ca-cert\") pod \"2c46403e-a9d6-48ca-ad0e-34533d5b22ab\" (UID: \"2c46403e-a9d6-48ca-ad0e-34533d5b22ab\") " Oct 30 19:15:09.762863 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:09.762839 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-istio-token\" (UniqueName: \"kubernetes.io/projected/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-istio-token\") pod \"2c46403e-a9d6-48ca-ad0e-34533d5b22ab\" (UID: \"2c46403e-a9d6-48ca-ad0e-34533d5b22ab\") " Oct 30 19:15:09.763153 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:09.762959 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-istio-envoy\") pod \"2c46403e-a9d6-48ca-ad0e-34533d5b22ab\" (UID: \"2c46403e-a9d6-48ca-ad0e-34533d5b22ab\") " Oct 30 19:15:09.763153 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:09.762999 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-istio-podinfo\") pod \"2c46403e-a9d6-48ca-ad0e-34533d5b22ab\" (UID: \"2c46403e-a9d6-48ca-ad0e-34533d5b22ab\") " Oct 30 19:15:09.763153 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:09.763042 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-workload-certs\") pod \"2c46403e-a9d6-48ca-ad0e-34533d5b22ab\" (UID: \"2c46403e-a9d6-48ca-ad0e-34533d5b22ab\") " Oct 30 19:15:09.763153 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:09.763060 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-credential-socket" pod "2c46403e-a9d6-48ca-ad0e-34533d5b22ab" (UID: "2c46403e-a9d6-48ca-ad0e-34533d5b22ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:09.763153 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:09.763091 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-istio-data\") pod \"2c46403e-a9d6-48ca-ad0e-34533d5b22ab\" (UID: \"2c46403e-a9d6-48ca-ad0e-34533d5b22ab\") " Oct 30 19:15:09.763768 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:09.763495 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whbf8\" (UniqueName: \"kubernetes.io/projected/e4c7339b-0d56-4483-8323-2b9d0059f21d-kube-api-access-whbf8\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:09.763768 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:09.763521 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:09.763768 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:09.763536 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4c7339b-0d56-4483-8323-2b9d0059f21d-maas-api-xks-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:09.763768 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:09.763552 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4c7339b-0d56-4483-8323-2b9d0059f21d-maas-api-openshift-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:09.763768 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:09.763571 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-istio-data" pod "2c46403e-a9d6-48ca-ad0e-34533d5b22ab" (UID: "2c46403e-a9d6-48ca-ad0e-34533d5b22ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:09.763768 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:09.763722 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-workload-certs" pod "2c46403e-a9d6-48ca-ad0e-34533d5b22ab" (UID: "2c46403e-a9d6-48ca-ad0e-34533d5b22ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:09.764168 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:09.763774 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-istiod-ca-cert" pod "2c46403e-a9d6-48ca-ad0e-34533d5b22ab" (UID: "2c46403e-a9d6-48ca-ad0e-34533d5b22ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:09.764168 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:09.763817 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-workload-socket" pod "2c46403e-a9d6-48ca-ad0e-34533d5b22ab" (UID: "2c46403e-a9d6-48ca-ad0e-34533d5b22ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:09.765894 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:09.765863 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-istio-envoy" pod "2c46403e-a9d6-48ca-ad0e-34533d5b22ab" (UID: "2c46403e-a9d6-48ca-ad0e-34533d5b22ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:09.766016 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:09.765873 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-kube-api-access-ll4nq" pod "2c46403e-a9d6-48ca-ad0e-34533d5b22ab" (UID: "2c46403e-a9d6-48ca-ad0e-34533d5b22ab"). InnerVolumeSpecName "kube-api-access-ll4nq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:09.766455 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:09.766434 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-istio-token" pod "2c46403e-a9d6-48ca-ad0e-34533d5b22ab" (UID: "2c46403e-a9d6-48ca-ad0e-34533d5b22ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:09.766532 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:09.766486 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-istio-podinfo" pod "2c46403e-a9d6-48ca-ad0e-34533d5b22ab" (UID: "2c46403e-a9d6-48ca-ad0e-34533d5b22ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:15:09.865187 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:09.865155 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:09.865187 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:09.865186 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:09.865396 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:09.865195 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:09.865396 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:09.865205 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:09.865396 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:09.865214 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:09.865396 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:09.865222 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:09.865396 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:09.865231 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll4nq\" (UniqueName: \"kubernetes.io/projected/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-kube-api-access-ll4nq\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:09.865396 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:09.865241 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c46403e-a9d6-48ca-ad0e-34533d5b22ab-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:10.318639 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:10.318548 4442 generic.go:356] "Generic (PLEG): container finished" podID="2c46403e-a9d6-48ca-ad0e-34533d5b22ab" containerID="74aba9be06e1f2a0300e6db926ed683c3f9f22280513bee037595b8d6e8f8481" exitCode=0 Oct 30 19:15:10.318639 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:10.318604 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" Oct 30 19:15:10.318966 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:10.318642 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" event={"ID":"2c46403e-a9d6-48ca-ad0e-34533d5b22ab","Type":"ContainerDied","Data":"74aba9be06e1f2a0300e6db926ed683c3f9f22280513bee037595b8d6e8f8481"} Oct 30 19:15:10.318966 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:10.318697 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk" event={"ID":"2c46403e-a9d6-48ca-ad0e-34533d5b22ab","Type":"ContainerDied","Data":"627572e798908d4873fbc32c64572c625a1985edd0c7783f3246f6cd676f7262"} Oct 30 19:15:10.318966 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:10.318741 4442 scope.go:122] "RemoveContainer" containerID="74aba9be06e1f2a0300e6db926ed683c3f9f22280513bee037595b8d6e8f8481" Oct 30 19:15:10.320309 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:10.320287 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-921450-mtrn6" event={"ID":"e4c7339b-0d56-4483-8323-2b9d0059f21d","Type":"ContainerDied","Data":"294d39e0557bb321b60e13a76d3c495b0702f3be2b68827462391fcae6fd6b52"} Oct 30 19:15:10.320403 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:10.320331 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="294d39e0557bb321b60e13a76d3c495b0702f3be2b68827462391fcae6fd6b52" Oct 30 19:15:10.320403 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:10.320355 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-921450-mtrn6" Oct 30 19:15:10.332430 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:10.331940 4442 scope.go:122] "RemoveContainer" containerID="b2d680892430d282d2b2ad6922e0aba3e381d7d7b0de2ccde62ab1ff08d5bf03" Oct 30 19:15:10.344967 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:10.344936 4442 scope.go:122] "RemoveContainer" containerID="74aba9be06e1f2a0300e6db926ed683c3f9f22280513bee037595b8d6e8f8481" Oct 30 19:15:10.345332 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:15:10.345308 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74aba9be06e1f2a0300e6db926ed683c3f9f22280513bee037595b8d6e8f8481\": container with ID starting with 74aba9be06e1f2a0300e6db926ed683c3f9f22280513bee037595b8d6e8f8481 not found: ID does not exist" containerID="74aba9be06e1f2a0300e6db926ed683c3f9f22280513bee037595b8d6e8f8481" Oct 30 19:15:10.345438 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:10.345347 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74aba9be06e1f2a0300e6db926ed683c3f9f22280513bee037595b8d6e8f8481"} err="failed to get container status \"74aba9be06e1f2a0300e6db926ed683c3f9f22280513bee037595b8d6e8f8481\": rpc error: code = NotFound desc = could not find container \"74aba9be06e1f2a0300e6db926ed683c3f9f22280513bee037595b8d6e8f8481\": container with ID starting with 74aba9be06e1f2a0300e6db926ed683c3f9f22280513bee037595b8d6e8f8481 not found: ID does not exist" Oct 30 19:15:10.345438 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:10.345372 4442 scope.go:122] "RemoveContainer" containerID="b2d680892430d282d2b2ad6922e0aba3e381d7d7b0de2ccde62ab1ff08d5bf03" Oct 30 19:15:10.345699 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:15:10.345681 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2d680892430d282d2b2ad6922e0aba3e381d7d7b0de2ccde62ab1ff08d5bf03\": container with ID starting with b2d680892430d282d2b2ad6922e0aba3e381d7d7b0de2ccde62ab1ff08d5bf03 not found: ID does not exist" containerID="b2d680892430d282d2b2ad6922e0aba3e381d7d7b0de2ccde62ab1ff08d5bf03" Oct 30 19:15:10.345770 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:10.345713 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2d680892430d282d2b2ad6922e0aba3e381d7d7b0de2ccde62ab1ff08d5bf03"} err="failed to get container status \"b2d680892430d282d2b2ad6922e0aba3e381d7d7b0de2ccde62ab1ff08d5bf03\": rpc error: code = NotFound desc = could not find container \"b2d680892430d282d2b2ad6922e0aba3e381d7d7b0de2ccde62ab1ff08d5bf03\": container with ID starting with b2d680892430d282d2b2ad6922e0aba3e381d7d7b0de2ccde62ab1ff08d5bf03 not found: ID does not exist" Oct 30 19:15:10.372614 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:10.372584 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk"] Oct 30 19:15:10.384178 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:10.383797 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-mh2fk"] Oct 30 19:15:10.748220 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:10.748195 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" Oct 30 19:15:10.773679 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:10.773644 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/425bc56e-e6f2-400d-a6dc-a277201acb8f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/425bc56e-e6f2-400d-a6dc-a277201acb8f-credential-socket\") pod \"425bc56e-e6f2-400d-a6dc-a277201acb8f\" (UID: \"425bc56e-e6f2-400d-a6dc-a277201acb8f\") " Oct 30 19:15:10.773846 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:10.773701 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/425bc56e-e6f2-400d-a6dc-a277201acb8f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/425bc56e-e6f2-400d-a6dc-a277201acb8f-istio-data\") pod \"425bc56e-e6f2-400d-a6dc-a277201acb8f\" (UID: \"425bc56e-e6f2-400d-a6dc-a277201acb8f\") " Oct 30 19:15:10.773846 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:10.773755 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/425bc56e-e6f2-400d-a6dc-a277201acb8f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/425bc56e-e6f2-400d-a6dc-a277201acb8f-workload-certs\") pod \"425bc56e-e6f2-400d-a6dc-a277201acb8f\" (UID: \"425bc56e-e6f2-400d-a6dc-a277201acb8f\") " Oct 30 19:15:10.773846 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:10.773805 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/425bc56e-e6f2-400d-a6dc-a277201acb8f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/425bc56e-e6f2-400d-a6dc-a277201acb8f-istio-envoy\") pod \"425bc56e-e6f2-400d-a6dc-a277201acb8f\" (UID: \"425bc56e-e6f2-400d-a6dc-a277201acb8f\") " Oct 30 19:15:10.774056 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:10.773848 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/425bc56e-e6f2-400d-a6dc-a277201acb8f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/425bc56e-e6f2-400d-a6dc-a277201acb8f-workload-socket\") pod \"425bc56e-e6f2-400d-a6dc-a277201acb8f\" (UID: \"425bc56e-e6f2-400d-a6dc-a277201acb8f\") " Oct 30 19:15:10.774056 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:10.773963 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/425bc56e-e6f2-400d-a6dc-a277201acb8f-kube-api-access-n85w5\" (UniqueName: \"kubernetes.io/projected/425bc56e-e6f2-400d-a6dc-a277201acb8f-kube-api-access-n85w5\") pod \"425bc56e-e6f2-400d-a6dc-a277201acb8f\" (UID: \"425bc56e-e6f2-400d-a6dc-a277201acb8f\") " Oct 30 19:15:10.774056 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:10.773965 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/425bc56e-e6f2-400d-a6dc-a277201acb8f-credential-socket" pod "425bc56e-e6f2-400d-a6dc-a277201acb8f" (UID: "425bc56e-e6f2-400d-a6dc-a277201acb8f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:10.774056 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:10.774010 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/425bc56e-e6f2-400d-a6dc-a277201acb8f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/425bc56e-e6f2-400d-a6dc-a277201acb8f-istio-podinfo\") pod \"425bc56e-e6f2-400d-a6dc-a277201acb8f\" (UID: \"425bc56e-e6f2-400d-a6dc-a277201acb8f\") " Oct 30 19:15:10.774056 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:10.774024 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/425bc56e-e6f2-400d-a6dc-a277201acb8f-istio-data" pod "425bc56e-e6f2-400d-a6dc-a277201acb8f" (UID: "425bc56e-e6f2-400d-a6dc-a277201acb8f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:10.774363 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:10.774061 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/425bc56e-e6f2-400d-a6dc-a277201acb8f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/425bc56e-e6f2-400d-a6dc-a277201acb8f-istiod-ca-cert\") pod \"425bc56e-e6f2-400d-a6dc-a277201acb8f\" (UID: \"425bc56e-e6f2-400d-a6dc-a277201acb8f\") " Oct 30 19:15:10.774363 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:10.774165 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/425bc56e-e6f2-400d-a6dc-a277201acb8f-istio-token\" (UniqueName: \"kubernetes.io/projected/425bc56e-e6f2-400d-a6dc-a277201acb8f-istio-token\") pod \"425bc56e-e6f2-400d-a6dc-a277201acb8f\" (UID: \"425bc56e-e6f2-400d-a6dc-a277201acb8f\") " Oct 30 19:15:10.774363 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:10.774170 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/425bc56e-e6f2-400d-a6dc-a277201acb8f-workload-certs" pod "425bc56e-e6f2-400d-a6dc-a277201acb8f" (UID: "425bc56e-e6f2-400d-a6dc-a277201acb8f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:10.774363 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:10.774258 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/425bc56e-e6f2-400d-a6dc-a277201acb8f-workload-socket" pod "425bc56e-e6f2-400d-a6dc-a277201acb8f" (UID: "425bc56e-e6f2-400d-a6dc-a277201acb8f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:10.774570 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:10.774556 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/425bc56e-e6f2-400d-a6dc-a277201acb8f-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:10.774634 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:10.774578 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/425bc56e-e6f2-400d-a6dc-a277201acb8f-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:10.774634 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:10.774591 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/425bc56e-e6f2-400d-a6dc-a277201acb8f-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:10.774634 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:10.774603 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/425bc56e-e6f2-400d-a6dc-a277201acb8f-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:10.774782 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:10.774637 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/425bc56e-e6f2-400d-a6dc-a277201acb8f-istiod-ca-cert" pod "425bc56e-e6f2-400d-a6dc-a277201acb8f" (UID: "425bc56e-e6f2-400d-a6dc-a277201acb8f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:10.776430 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:10.776402 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/425bc56e-e6f2-400d-a6dc-a277201acb8f-kube-api-access-n85w5" pod "425bc56e-e6f2-400d-a6dc-a277201acb8f" (UID: "425bc56e-e6f2-400d-a6dc-a277201acb8f"). InnerVolumeSpecName "kube-api-access-n85w5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:10.776792 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:10.776769 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/425bc56e-e6f2-400d-a6dc-a277201acb8f-istio-token" pod "425bc56e-e6f2-400d-a6dc-a277201acb8f" (UID: "425bc56e-e6f2-400d-a6dc-a277201acb8f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:10.776919 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:10.776824 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/425bc56e-e6f2-400d-a6dc-a277201acb8f-istio-envoy" pod "425bc56e-e6f2-400d-a6dc-a277201acb8f" (UID: "425bc56e-e6f2-400d-a6dc-a277201acb8f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:10.776919 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:10.776873 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/425bc56e-e6f2-400d-a6dc-a277201acb8f-istio-podinfo" pod "425bc56e-e6f2-400d-a6dc-a277201acb8f" (UID: "425bc56e-e6f2-400d-a6dc-a277201acb8f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:15:10.876090 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:10.876057 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/425bc56e-e6f2-400d-a6dc-a277201acb8f-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:10.876090 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:10.876086 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/425bc56e-e6f2-400d-a6dc-a277201acb8f-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:10.876090 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:10.876096 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n85w5\" (UniqueName: \"kubernetes.io/projected/425bc56e-e6f2-400d-a6dc-a277201acb8f-kube-api-access-n85w5\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:10.876397 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:10.876104 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/425bc56e-e6f2-400d-a6dc-a277201acb8f-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:10.876397 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:10.876114 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/425bc56e-e6f2-400d-a6dc-a277201acb8f-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:10.944961 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:10.944922 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4"] Oct 30 19:15:11.076210 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:11.070076 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c46403e-a9d6-48ca-ad0e-34533d5b22ab" path="/var/lib/kubelet/pods/2c46403e-a9d6-48ca-ad0e-34533d5b22ab/volumes" Oct 30 19:15:11.332698 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:11.332604 4442 generic.go:356] "Generic (PLEG): container finished" podID="425bc56e-e6f2-400d-a6dc-a277201acb8f" containerID="4f65edc89f7838267e2c63af4430fda749a05285dc4b8bfc9c0576835efb71bc" exitCode=0 Oct 30 19:15:11.332698 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:11.332660 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" event={"ID":"425bc56e-e6f2-400d-a6dc-a277201acb8f","Type":"ContainerDied","Data":"4f65edc89f7838267e2c63af4430fda749a05285dc4b8bfc9c0576835efb71bc"} Oct 30 19:15:11.332698 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:11.332691 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" Oct 30 19:15:11.333004 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:11.332717 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp" event={"ID":"425bc56e-e6f2-400d-a6dc-a277201acb8f","Type":"ContainerDied","Data":"e5cc57b12a6798a437e0d328c53f352deb7f2b933ba0c0a788a938e4606bcc2f"} Oct 30 19:15:11.333004 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:11.332742 4442 scope.go:122] "RemoveContainer" containerID="4f65edc89f7838267e2c63af4430fda749a05285dc4b8bfc9c0576835efb71bc" Oct 30 19:15:11.345886 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:11.345861 4442 scope.go:122] "RemoveContainer" containerID="529b13f2ac933b33601c939e4f23541c0eb541e41c6d2fa2fefc6ff9bcc9924f" Oct 30 19:15:11.362647 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:11.362620 4442 scope.go:122] "RemoveContainer" containerID="4f65edc89f7838267e2c63af4430fda749a05285dc4b8bfc9c0576835efb71bc" Oct 30 19:15:11.362971 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:15:11.362933 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f65edc89f7838267e2c63af4430fda749a05285dc4b8bfc9c0576835efb71bc\": container with ID starting with 4f65edc89f7838267e2c63af4430fda749a05285dc4b8bfc9c0576835efb71bc not found: ID does not exist" containerID="4f65edc89f7838267e2c63af4430fda749a05285dc4b8bfc9c0576835efb71bc" Oct 30 19:15:11.363059 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:11.362984 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f65edc89f7838267e2c63af4430fda749a05285dc4b8bfc9c0576835efb71bc"} err="failed to get container status \"4f65edc89f7838267e2c63af4430fda749a05285dc4b8bfc9c0576835efb71bc\": rpc error: code = NotFound desc = could not find container \"4f65edc89f7838267e2c63af4430fda749a05285dc4b8bfc9c0576835efb71bc\": container with ID starting with 4f65edc89f7838267e2c63af4430fda749a05285dc4b8bfc9c0576835efb71bc not found: ID does not exist" Oct 30 19:15:11.363059 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:11.363005 4442 scope.go:122] "RemoveContainer" containerID="529b13f2ac933b33601c939e4f23541c0eb541e41c6d2fa2fefc6ff9bcc9924f" Oct 30 19:15:11.363342 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:15:11.363320 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"529b13f2ac933b33601c939e4f23541c0eb541e41c6d2fa2fefc6ff9bcc9924f\": container with ID starting with 529b13f2ac933b33601c939e4f23541c0eb541e41c6d2fa2fefc6ff9bcc9924f not found: ID does not exist" containerID="529b13f2ac933b33601c939e4f23541c0eb541e41c6d2fa2fefc6ff9bcc9924f" Oct 30 19:15:11.363446 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:11.363348 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"529b13f2ac933b33601c939e4f23541c0eb541e41c6d2fa2fefc6ff9bcc9924f"} err="failed to get container status \"529b13f2ac933b33601c939e4f23541c0eb541e41c6d2fa2fefc6ff9bcc9924f\": rpc error: code = NotFound desc = could not find container \"529b13f2ac933b33601c939e4f23541c0eb541e41c6d2fa2fefc6ff9bcc9924f\": container with ID starting with 529b13f2ac933b33601c939e4f23541c0eb541e41c6d2fa2fefc6ff9bcc9924f not found: ID does not exist" Oct 30 19:15:11.389187 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:11.389146 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp"] Oct 30 19:15:11.393055 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:11.393031 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jddrp"] Oct 30 19:15:13.063746 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:13.063706 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="425bc56e-e6f2-400d-a6dc-a277201acb8f" path="/var/lib/kubelet/pods/425bc56e-e6f2-400d-a6dc-a277201acb8f/volumes" Oct 30 19:15:14.057162 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:14.057104 4442 scope.go:122] "RemoveContainer" containerID="e7935d73730c5e418da03ca355810516221ac6e2da3213b6320a306b391e2666" Oct 30 19:15:14.357464 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:14.357429 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889795-ljkwb" event={"ID":"07aa8798-3d58-4d0d-b33e-715d8871c259","Type":"ContainerStarted","Data":"c9446ed2ef98fd7a0adccc950b1ab6b5899a78fcf6128a7fc6a83e8f20b3d6e5"} Oct 30 19:15:15.370217 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:15.370180 4442 generic.go:356] "Generic (PLEG): container finished" podID="07aa8798-3d58-4d0d-b33e-715d8871c259" containerID="c9446ed2ef98fd7a0adccc950b1ab6b5899a78fcf6128a7fc6a83e8f20b3d6e5" exitCode=60 Oct 30 19:15:15.370746 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:15.370460 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889795-ljkwb" event={"ID":"07aa8798-3d58-4d0d-b33e-715d8871c259","Type":"ContainerDied","Data":"c9446ed2ef98fd7a0adccc950b1ab6b5899a78fcf6128a7fc6a83e8f20b3d6e5"} Oct 30 19:15:15.370746 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:15.370528 4442 scope.go:122] "RemoveContainer" containerID="e7935d73730c5e418da03ca355810516221ac6e2da3213b6320a306b391e2666" Oct 30 19:15:15.371008 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:15.370987 4442 scope.go:122] "RemoveContainer" containerID="c9446ed2ef98fd7a0adccc950b1ab6b5899a78fcf6128a7fc6a83e8f20b3d6e5" Oct 30 19:15:15.371365 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:15:15.371338 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-45dc34-29889795-ljkwb_odh-ai-gateway-infra(07aa8798-3d58-4d0d-b33e-715d8871c259)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889795-ljkwb" podUID="07aa8798-3d58-4d0d-b33e-715d8871c259" Oct 30 19:15:15.379585 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:15.377582 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-45dc34-79f9556b56-st7qr"] Oct 30 19:15:15.407300 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:15.407268 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-45dc34-d7f945cff-rqxz2"] Oct 30 19:15:15.410569 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:15.410545 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889795-ljkwb"] Oct 30 19:15:16.121554 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:16.121528 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c095fa-7ff45cdc78-lvpxv" Oct 30 19:15:16.156473 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:16.156443 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c095fa-5d479bd6f9-rgj6w" Oct 30 19:15:16.203328 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:16.201010 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-fe1374-5c8465bf67-bl75c"] Oct 30 19:15:16.241894 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:16.241856 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/979a6818-4b68-4229-86e5-c11fdd1586bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/979a6818-4b68-4229-86e5-c11fdd1586bd-plugins-config-volume\") pod \"979a6818-4b68-4229-86e5-c11fdd1586bd\" (UID: \"979a6818-4b68-4229-86e5-c11fdd1586bd\") " Oct 30 19:15:16.242110 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:16.241979 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9e66197-f5b9-4ea3-8fa6-d52647e45327-kube-api-access-79fng\" (UniqueName: \"kubernetes.io/projected/b9e66197-f5b9-4ea3-8fa6-d52647e45327-kube-api-access-79fng\") pod \"b9e66197-f5b9-4ea3-8fa6-d52647e45327\" (UID: \"b9e66197-f5b9-4ea3-8fa6-d52647e45327\") " Oct 30 19:15:16.242110 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:16.242080 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9e66197-f5b9-4ea3-8fa6-d52647e45327-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9e66197-f5b9-4ea3-8fa6-d52647e45327-plugins-config-volume\") pod \"b9e66197-f5b9-4ea3-8fa6-d52647e45327\" (UID: \"b9e66197-f5b9-4ea3-8fa6-d52647e45327\") " Oct 30 19:15:16.242316 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:16.242215 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/979a6818-4b68-4229-86e5-c11fdd1586bd-kube-api-access-w7l8k\" (UniqueName: \"kubernetes.io/projected/979a6818-4b68-4229-86e5-c11fdd1586bd-kube-api-access-w7l8k\") pod \"979a6818-4b68-4229-86e5-c11fdd1586bd\" (UID: \"979a6818-4b68-4229-86e5-c11fdd1586bd\") " Oct 30 19:15:16.242402 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:16.242374 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/979a6818-4b68-4229-86e5-c11fdd1586bd-plugins-config-volume" pod "979a6818-4b68-4229-86e5-c11fdd1586bd" (UID: "979a6818-4b68-4229-86e5-c11fdd1586bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:16.242715 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:16.242692 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/979a6818-4b68-4229-86e5-c11fdd1586bd-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:16.243251 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:16.243225 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9e66197-f5b9-4ea3-8fa6-d52647e45327-plugins-config-volume" pod "b9e66197-f5b9-4ea3-8fa6-d52647e45327" (UID: "b9e66197-f5b9-4ea3-8fa6-d52647e45327"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:16.245473 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:16.245446 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9e66197-f5b9-4ea3-8fa6-d52647e45327-kube-api-access-79fng" pod "b9e66197-f5b9-4ea3-8fa6-d52647e45327" (UID: "b9e66197-f5b9-4ea3-8fa6-d52647e45327"). InnerVolumeSpecName "kube-api-access-79fng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:16.245633 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:16.245485 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/979a6818-4b68-4229-86e5-c11fdd1586bd-kube-api-access-w7l8k" pod "979a6818-4b68-4229-86e5-c11fdd1586bd" (UID: "979a6818-4b68-4229-86e5-c11fdd1586bd"). InnerVolumeSpecName "kube-api-access-w7l8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:16.343639 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:16.343556 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79fng\" (UniqueName: \"kubernetes.io/projected/b9e66197-f5b9-4ea3-8fa6-d52647e45327-kube-api-access-79fng\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:16.343639 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:16.343583 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9e66197-f5b9-4ea3-8fa6-d52647e45327-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:16.343639 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:16.343593 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7l8k\" (UniqueName: \"kubernetes.io/projected/979a6818-4b68-4229-86e5-c11fdd1586bd-kube-api-access-w7l8k\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:16.389228 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:16.389184 4442 generic.go:356] "Generic (PLEG): container finished" podID="979a6818-4b68-4229-86e5-c11fdd1586bd" containerID="4e28fba4ba4a8187c95d2d0115df7b6eef83539475cce4a4d812515bd5a4f43f" exitCode=0 Oct 30 19:15:16.389228 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:16.389230 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c095fa-7ff45cdc78-lvpxv" Oct 30 19:15:16.389793 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:16.389187 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c095fa-7ff45cdc78-lvpxv" event={"ID":"979a6818-4b68-4229-86e5-c11fdd1586bd","Type":"ContainerDied","Data":"4e28fba4ba4a8187c95d2d0115df7b6eef83539475cce4a4d812515bd5a4f43f"} Oct 30 19:15:16.389793 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:16.389392 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c095fa-7ff45cdc78-lvpxv" event={"ID":"979a6818-4b68-4229-86e5-c11fdd1586bd","Type":"ContainerDied","Data":"0db5ed6135a93b7c247f96ee4126ec30453e3874f80cb595a2da2e5d51767f68"} Oct 30 19:15:16.389793 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:16.389427 4442 scope.go:122] "RemoveContainer" containerID="4e28fba4ba4a8187c95d2d0115df7b6eef83539475cce4a4d812515bd5a4f43f" Oct 30 19:15:16.391921 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:16.391705 4442 generic.go:356] "Generic (PLEG): container finished" podID="b9e66197-f5b9-4ea3-8fa6-d52647e45327" containerID="b8439f6f2605eed5e5db2887dfbddd74528a4337fe7ac304ea75cf9b7bea2e30" exitCode=0 Oct 30 19:15:16.391921 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:16.391771 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c095fa-5d479bd6f9-rgj6w" Oct 30 19:15:16.391921 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:16.391798 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c095fa-5d479bd6f9-rgj6w" event={"ID":"b9e66197-f5b9-4ea3-8fa6-d52647e45327","Type":"ContainerDied","Data":"b8439f6f2605eed5e5db2887dfbddd74528a4337fe7ac304ea75cf9b7bea2e30"} Oct 30 19:15:16.391921 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:16.391841 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c095fa-5d479bd6f9-rgj6w" event={"ID":"b9e66197-f5b9-4ea3-8fa6-d52647e45327","Type":"ContainerDied","Data":"1c0744bdeea36e128c76abd3c346e7177f14d4e00da6e69218243d1435017c99"} Oct 30 19:15:16.405500 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:16.405475 4442 scope.go:122] "RemoveContainer" containerID="4e28fba4ba4a8187c95d2d0115df7b6eef83539475cce4a4d812515bd5a4f43f" Oct 30 19:15:16.407526 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:15:16.407359 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e28fba4ba4a8187c95d2d0115df7b6eef83539475cce4a4d812515bd5a4f43f\": container with ID starting with 4e28fba4ba4a8187c95d2d0115df7b6eef83539475cce4a4d812515bd5a4f43f not found: ID does not exist" containerID="4e28fba4ba4a8187c95d2d0115df7b6eef83539475cce4a4d812515bd5a4f43f" Oct 30 19:15:16.407526 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:16.407401 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e28fba4ba4a8187c95d2d0115df7b6eef83539475cce4a4d812515bd5a4f43f"} err="failed to get container status \"4e28fba4ba4a8187c95d2d0115df7b6eef83539475cce4a4d812515bd5a4f43f\": rpc error: code = NotFound desc = could not find container \"4e28fba4ba4a8187c95d2d0115df7b6eef83539475cce4a4d812515bd5a4f43f\": container with ID starting with 4e28fba4ba4a8187c95d2d0115df7b6eef83539475cce4a4d812515bd5a4f43f not found: ID does not exist" Oct 30 19:15:16.407526 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:16.407425 4442 scope.go:122] "RemoveContainer" containerID="b8439f6f2605eed5e5db2887dfbddd74528a4337fe7ac304ea75cf9b7bea2e30" Oct 30 19:15:16.431888 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:16.431865 4442 scope.go:122] "RemoveContainer" containerID="b8439f6f2605eed5e5db2887dfbddd74528a4337fe7ac304ea75cf9b7bea2e30" Oct 30 19:15:16.432249 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:15:16.432224 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8439f6f2605eed5e5db2887dfbddd74528a4337fe7ac304ea75cf9b7bea2e30\": container with ID starting with b8439f6f2605eed5e5db2887dfbddd74528a4337fe7ac304ea75cf9b7bea2e30 not found: ID does not exist" containerID="b8439f6f2605eed5e5db2887dfbddd74528a4337fe7ac304ea75cf9b7bea2e30" Oct 30 19:15:16.432351 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:16.432261 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8439f6f2605eed5e5db2887dfbddd74528a4337fe7ac304ea75cf9b7bea2e30"} err="failed to get container status \"b8439f6f2605eed5e5db2887dfbddd74528a4337fe7ac304ea75cf9b7bea2e30\": rpc error: code = NotFound desc = could not find container \"b8439f6f2605eed5e5db2887dfbddd74528a4337fe7ac304ea75cf9b7bea2e30\": container with ID starting with b8439f6f2605eed5e5db2887dfbddd74528a4337fe7ac304ea75cf9b7bea2e30 not found: ID does not exist" Oct 30 19:15:16.458013 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:16.457983 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c095fa-7ff45cdc78-lvpxv"] Oct 30 19:15:16.461713 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:16.461684 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c095fa-7ff45cdc78-lvpxv"] Oct 30 19:15:16.499189 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:16.499103 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c095fa-5d479bd6f9-rgj6w"] Oct 30 19:15:16.502587 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:16.502558 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c095fa-5d479bd6f9-rgj6w"] Oct 30 19:15:16.553863 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:16.553841 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889795-ljkwb" Oct 30 19:15:16.645933 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:16.645902 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07aa8798-3d58-4d0d-b33e-715d8871c259-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07aa8798-3d58-4d0d-b33e-715d8871c259-maas-api-service-ca\") pod \"07aa8798-3d58-4d0d-b33e-715d8871c259\" (UID: \"07aa8798-3d58-4d0d-b33e-715d8871c259\") " Oct 30 19:15:16.645933 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:16.645949 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07aa8798-3d58-4d0d-b33e-715d8871c259-kube-api-access-rt2ms\" (UniqueName: \"kubernetes.io/projected/07aa8798-3d58-4d0d-b33e-715d8871c259-kube-api-access-rt2ms\") pod \"07aa8798-3d58-4d0d-b33e-715d8871c259\" (UID: \"07aa8798-3d58-4d0d-b33e-715d8871c259\") " Oct 30 19:15:16.646423 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:16.646396 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07aa8798-3d58-4d0d-b33e-715d8871c259-maas-api-service-ca" pod "07aa8798-3d58-4d0d-b33e-715d8871c259" (UID: "07aa8798-3d58-4d0d-b33e-715d8871c259"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:16.648315 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:16.648291 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07aa8798-3d58-4d0d-b33e-715d8871c259-kube-api-access-rt2ms" pod "07aa8798-3d58-4d0d-b33e-715d8871c259" (UID: "07aa8798-3d58-4d0d-b33e-715d8871c259"). InnerVolumeSpecName "kube-api-access-rt2ms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:16.747153 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:16.747109 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07aa8798-3d58-4d0d-b33e-715d8871c259-maas-api-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:16.747153 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:16.747157 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt2ms\" (UniqueName: \"kubernetes.io/projected/07aa8798-3d58-4d0d-b33e-715d8871c259-kube-api-access-rt2ms\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:16.929411 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:16.929321 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-921450-8d6546c9b-5pxxc"] Oct 30 19:15:16.961239 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:16.961209 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-921450-7f64fb4c79-xs76b"] Oct 30 19:15:17.066602 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.066379 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="979a6818-4b68-4229-86e5-c11fdd1586bd" path="/var/lib/kubelet/pods/979a6818-4b68-4229-86e5-c11fdd1586bd/volumes" Oct 30 19:15:17.066821 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.066778 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9e66197-f5b9-4ea3-8fa6-d52647e45327" path="/var/lib/kubelet/pods/b9e66197-f5b9-4ea3-8fa6-d52647e45327/volumes" Oct 30 19:15:17.353023 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.352929 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b963c8b4-j8jgz"] Oct 30 19:15:17.353430 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.353416 4442 cpu_manager.go:414] "containerMap: removing container" podUID="2c46403e-a9d6-48ca-ad0e-34533d5b22ab" containerName="istio-proxy" Oct 30 19:15:17.353518 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.353431 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c46403e-a9d6-48ca-ad0e-34533d5b22ab" containerName="istio-proxy" Oct 30 19:15:17.353518 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.353444 4442 cpu_manager.go:414] "containerMap: removing container" podUID="b9e66197-f5b9-4ea3-8fa6-d52647e45327" containerName="payload-pre-processing" Oct 30 19:15:17.353518 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.353453 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9e66197-f5b9-4ea3-8fa6-d52647e45327" containerName="payload-pre-processing" Oct 30 19:15:17.353518 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.353470 4442 cpu_manager.go:414] "containerMap: removing container" podUID="07aa8798-3d58-4d0d-b33e-715d8871c259" containerName="cleanup" Oct 30 19:15:17.353518 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.353475 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07aa8798-3d58-4d0d-b33e-715d8871c259" containerName="cleanup" Oct 30 19:15:17.353518 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.353481 4442 cpu_manager.go:414] "containerMap: removing container" podUID="e4c7339b-0d56-4483-8323-2b9d0059f21d" containerName="revoke-keys" Oct 30 19:15:17.353518 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.353486 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4c7339b-0d56-4483-8323-2b9d0059f21d" containerName="revoke-keys" Oct 30 19:15:17.353518 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.353497 4442 cpu_manager.go:414] "containerMap: removing container" podUID="979a6818-4b68-4229-86e5-c11fdd1586bd" containerName="payload-processing" Oct 30 19:15:17.353518 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.353501 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="979a6818-4b68-4229-86e5-c11fdd1586bd" containerName="payload-processing" Oct 30 19:15:17.353518 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.353511 4442 cpu_manager.go:414] "containerMap: removing container" podUID="07aa8798-3d58-4d0d-b33e-715d8871c259" containerName="cleanup" Oct 30 19:15:17.353518 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.353517 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07aa8798-3d58-4d0d-b33e-715d8871c259" containerName="cleanup" Oct 30 19:15:17.353868 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.353533 4442 cpu_manager.go:414] "containerMap: removing container" podUID="425bc56e-e6f2-400d-a6dc-a277201acb8f" containerName="istio-proxy" Oct 30 19:15:17.353868 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.353540 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="425bc56e-e6f2-400d-a6dc-a277201acb8f" containerName="istio-proxy" Oct 30 19:15:17.353868 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.353606 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="07aa8798-3d58-4d0d-b33e-715d8871c259" containerName="cleanup" Oct 30 19:15:17.353868 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.353616 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="425bc56e-e6f2-400d-a6dc-a277201acb8f" containerName="istio-proxy" Oct 30 19:15:17.353868 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.353623 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c46403e-a9d6-48ca-ad0e-34533d5b22ab" containerName="istio-proxy" Oct 30 19:15:17.353868 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.353630 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9e66197-f5b9-4ea3-8fa6-d52647e45327" containerName="payload-pre-processing" Oct 30 19:15:17.353868 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.353637 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="07aa8798-3d58-4d0d-b33e-715d8871c259" containerName="cleanup" Oct 30 19:15:17.353868 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.353641 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4c7339b-0d56-4483-8323-2b9d0059f21d" containerName="revoke-keys" Oct 30 19:15:17.353868 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.353647 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="979a6818-4b68-4229-86e5-c11fdd1586bd" containerName="payload-processing" Oct 30 19:15:17.353868 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.353651 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c46403e-a9d6-48ca-ad0e-34533d5b22ab" containerName="istio-proxy" Oct 30 19:15:17.353868 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.353658 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="07aa8798-3d58-4d0d-b33e-715d8871c259" containerName="cleanup" Oct 30 19:15:17.353868 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.353665 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="425bc56e-e6f2-400d-a6dc-a277201acb8f" containerName="istio-proxy" Oct 30 19:15:17.358104 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.358084 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b963c8b4-j8jgz" Oct 30 19:15:17.380472 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.379006 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b963c8b4-j8jgz"] Oct 30 19:15:17.383008 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.381601 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:15:17.434216 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.434185 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1fba6e3-ld59t"] Oct 30 19:15:17.434713 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.434635 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889795-ljkwb" Oct 30 19:15:17.435948 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.435439 4442 cpu_manager.go:414] "containerMap: removing container" podUID="2c46403e-a9d6-48ca-ad0e-34533d5b22ab" containerName="istio-proxy" Oct 30 19:15:17.435948 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.435460 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c46403e-a9d6-48ca-ad0e-34533d5b22ab" containerName="istio-proxy" Oct 30 19:15:17.435948 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.435502 4442 cpu_manager.go:414] "containerMap: removing container" podUID="07aa8798-3d58-4d0d-b33e-715d8871c259" containerName="cleanup" Oct 30 19:15:17.435948 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.435510 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07aa8798-3d58-4d0d-b33e-715d8871c259" containerName="cleanup" Oct 30 19:15:17.435948 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.435531 4442 cpu_manager.go:414] "containerMap: removing container" podUID="425bc56e-e6f2-400d-a6dc-a277201acb8f" containerName="istio-proxy" Oct 30 19:15:17.435948 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.435539 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="425bc56e-e6f2-400d-a6dc-a277201acb8f" containerName="istio-proxy" Oct 30 19:15:17.440212 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.440184 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889795-ljkwb" event={"ID":"07aa8798-3d58-4d0d-b33e-715d8871c259","Type":"ContainerDied","Data":"20963b357e4aca5f38b0556e901a9bcd0c5c91fe08b35c2fbadf648dea16bd78"} Oct 30 19:15:17.440376 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.440342 4442 scope.go:122] "RemoveContainer" containerID="c9446ed2ef98fd7a0adccc950b1ab6b5899a78fcf6128a7fc6a83e8f20b3d6e5" Oct 30 19:15:17.440492 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.440302 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1fba6e3-ld59t" Oct 30 19:15:17.455851 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.455546 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkbtp\" (UniqueName: \"kubernetes.io/projected/05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e-kube-api-access-kkbtp\") pod \"maas-api-revoke-sub-b963c8b4-j8jgz\" (UID: \"05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b963c8b4-j8jgz" Oct 30 19:15:17.456318 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.455994 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b963c8b4-j8jgz\" (UID: \"05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b963c8b4-j8jgz" Oct 30 19:15:17.456318 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.456160 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b963c8b4-j8jgz\" (UID: \"05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b963c8b4-j8jgz" Oct 30 19:15:17.456886 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.456850 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1fba6e3-ld59t"] Oct 30 19:15:17.532414 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.532107 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889795-ljkwb"] Oct 30 19:15:17.535489 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.535460 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889795-ljkwb"] Oct 30 19:15:17.557597 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.557567 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dab3a37-4bad-464c-b98f-5f6cc0592e9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1fba6e3-ld59t\" (UID: \"6dab3a37-4bad-464c-b98f-5f6cc0592e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1fba6e3-ld59t" Oct 30 19:15:17.557775 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.557616 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkbtp\" (UniqueName: \"kubernetes.io/projected/05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e-kube-api-access-kkbtp\") pod \"maas-api-revoke-sub-b963c8b4-j8jgz\" (UID: \"05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b963c8b4-j8jgz" Oct 30 19:15:17.557775 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.557728 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mjs9\" (UniqueName: \"kubernetes.io/projected/6dab3a37-4bad-464c-b98f-5f6cc0592e9f-kube-api-access-9mjs9\") pod \"maas-api-revoke-sub-e1fba6e3-ld59t\" (UID: \"6dab3a37-4bad-464c-b98f-5f6cc0592e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1fba6e3-ld59t" Oct 30 19:15:17.557904 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.557784 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b963c8b4-j8jgz\" (UID: \"05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b963c8b4-j8jgz" Oct 30 19:15:17.557904 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.557862 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b963c8b4-j8jgz\" (UID: \"05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b963c8b4-j8jgz" Oct 30 19:15:17.558120 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.558079 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dab3a37-4bad-464c-b98f-5f6cc0592e9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1fba6e3-ld59t\" (UID: \"6dab3a37-4bad-464c-b98f-5f6cc0592e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1fba6e3-ld59t" Oct 30 19:15:17.559476 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.559450 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b963c8b4-j8jgz\" (UID: \"05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b963c8b4-j8jgz" Oct 30 19:15:17.561059 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.561037 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b963c8b4-j8jgz\" (UID: \"05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b963c8b4-j8jgz" Oct 30 19:15:17.567507 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:15:17.567483 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 19:15:17.576915 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.576888 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkbtp\" (UniqueName: \"kubernetes.io/projected/05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e-kube-api-access-kkbtp\") pod \"maas-api-revoke-sub-b963c8b4-j8jgz\" (UID: \"05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b963c8b4-j8jgz" Oct 30 19:15:17.659490 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.659455 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dab3a37-4bad-464c-b98f-5f6cc0592e9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1fba6e3-ld59t\" (UID: \"6dab3a37-4bad-464c-b98f-5f6cc0592e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1fba6e3-ld59t" Oct 30 19:15:17.659688 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.659510 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mjs9\" (UniqueName: \"kubernetes.io/projected/6dab3a37-4bad-464c-b98f-5f6cc0592e9f-kube-api-access-9mjs9\") pod \"maas-api-revoke-sub-e1fba6e3-ld59t\" (UID: \"6dab3a37-4bad-464c-b98f-5f6cc0592e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1fba6e3-ld59t" Oct 30 19:15:17.659688 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.659604 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dab3a37-4bad-464c-b98f-5f6cc0592e9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1fba6e3-ld59t\" (UID: \"6dab3a37-4bad-464c-b98f-5f6cc0592e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1fba6e3-ld59t" Oct 30 19:15:17.660329 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.660299 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dab3a37-4bad-464c-b98f-5f6cc0592e9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1fba6e3-ld59t\" (UID: \"6dab3a37-4bad-464c-b98f-5f6cc0592e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1fba6e3-ld59t" Oct 30 19:15:17.661935 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.661911 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dab3a37-4bad-464c-b98f-5f6cc0592e9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1fba6e3-ld59t\" (UID: \"6dab3a37-4bad-464c-b98f-5f6cc0592e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1fba6e3-ld59t" Oct 30 19:15:17.674907 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.674886 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b963c8b4-j8jgz" Oct 30 19:15:17.678192 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.678170 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mjs9\" (UniqueName: \"kubernetes.io/projected/6dab3a37-4bad-464c-b98f-5f6cc0592e9f-kube-api-access-9mjs9\") pod \"maas-api-revoke-sub-e1fba6e3-ld59t\" (UID: \"6dab3a37-4bad-464c-b98f-5f6cc0592e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1fba6e3-ld59t" Oct 30 19:15:17.770544 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.770507 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1fba6e3-ld59t" Oct 30 19:15:17.974905 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:17.974881 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b963c8b4-j8jgz"] Oct 30 19:15:17.976989 ip-10-0-138-176 kubenswrapper[4442]: W1030 19:15:17.976949 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05a3ac2b_77cc_41ce_bc8e_8e88460d5c3e.slice/crio-34000c86908b13fd54fef2fcbb86ddb1fc1ce19b4464aea3f92d03a3d93b6b48 WatchSource:0}: Error finding container 34000c86908b13fd54fef2fcbb86ddb1fc1ce19b4464aea3f92d03a3d93b6b48: Status 404 returned error can't find the container with id 34000c86908b13fd54fef2fcbb86ddb1fc1ce19b4464aea3f92d03a3d93b6b48 Oct 30 19:15:18.077053 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:18.077021 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1fba6e3-ld59t"] Oct 30 19:15:18.126426 ip-10-0-138-176 kubenswrapper[4442]: W1030 19:15:18.126383 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dab3a37_4bad_464c_b98f_5f6cc0592e9f.slice/crio-23bcfad1a396c46988882f9e1c5edf53c62e42c44120602b24ce3faa330001e6 WatchSource:0}: Error finding container 23bcfad1a396c46988882f9e1c5edf53c62e42c44120602b24ce3faa330001e6: Status 404 returned error can't find the container with id 23bcfad1a396c46988882f9e1c5edf53c62e42c44120602b24ce3faa330001e6 Oct 30 19:15:18.447710 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:18.447670 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1fba6e3-ld59t" event={"ID":"6dab3a37-4bad-464c-b98f-5f6cc0592e9f","Type":"ContainerStarted","Data":"c1d6fee18f8be26ccfa6de4fe616bd30c753177597d6f69570c41b151e8ace2b"} Oct 30 19:15:18.448110 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:18.447728 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1fba6e3-ld59t" event={"ID":"6dab3a37-4bad-464c-b98f-5f6cc0592e9f","Type":"ContainerStarted","Data":"23bcfad1a396c46988882f9e1c5edf53c62e42c44120602b24ce3faa330001e6"} Oct 30 19:15:18.449333 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:18.449310 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b963c8b4-j8jgz" event={"ID":"05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e","Type":"ContainerStarted","Data":"bfeff1ac9c2da9957069c6e5d5af9d2ee181ca80c7672132f00d0c792fab9f03"} Oct 30 19:15:18.449433 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:18.449353 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b963c8b4-j8jgz" event={"ID":"05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e","Type":"ContainerStarted","Data":"34000c86908b13fd54fef2fcbb86ddb1fc1ce19b4464aea3f92d03a3d93b6b48"} Oct 30 19:15:18.488075 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:18.487979 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1fba6e3-ld59t" podStartSLOduration=1.487966486 podStartE2EDuration="1.487966486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:18.487313175 +0000 UTC m=+4615.577375514" watchObservedRunningTime="2026-10-30 19:15:18.487966486 +0000 UTC m=+4615.578028824" Oct 30 19:15:18.526439 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:18.526385 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b963c8b4-j8jgz" podStartSLOduration=1.526367926 podStartE2EDuration="1.526367926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:18.524030217 +0000 UTC m=+4615.614092557" watchObservedRunningTime="2026-10-30 19:15:18.526367926 +0000 UTC m=+4615.616430266" Oct 30 19:15:19.065223 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:19.065186 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07aa8798-3d58-4d0d-b33e-715d8871c259" path="/var/lib/kubelet/pods/07aa8798-3d58-4d0d-b33e-715d8871c259/volumes" Oct 30 19:15:19.458016 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:19.457980 4442 generic.go:356] "Generic (PLEG): container finished" podID="6dab3a37-4bad-464c-b98f-5f6cc0592e9f" containerID="c1d6fee18f8be26ccfa6de4fe616bd30c753177597d6f69570c41b151e8ace2b" exitCode=0 Oct 30 19:15:19.458542 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:19.458061 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1fba6e3-ld59t" event={"ID":"6dab3a37-4bad-464c-b98f-5f6cc0592e9f","Type":"ContainerDied","Data":"c1d6fee18f8be26ccfa6de4fe616bd30c753177597d6f69570c41b151e8ace2b"} Oct 30 19:15:19.459997 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:19.459970 4442 generic.go:356] "Generic (PLEG): container finished" podID="05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e" containerID="bfeff1ac9c2da9957069c6e5d5af9d2ee181ca80c7672132f00d0c792fab9f03" exitCode=0 Oct 30 19:15:19.460118 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:19.460055 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b963c8b4-j8jgz" event={"ID":"05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e","Type":"ContainerDied","Data":"bfeff1ac9c2da9957069c6e5d5af9d2ee181ca80c7672132f00d0c792fab9f03"} Oct 30 19:15:20.378797 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:20.378751 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-45dc34-79f9556b56-st7qr" podUID="269e36c9-21e8-412e-b49e-e446b7d57def" containerName="payload-processing" containerID="cri-o://a5556d595f6870a40a95db09538a26423c8457392bad43c25bd378fa6f31d6fc" gracePeriod=25 Oct 30 19:15:20.408205 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:20.408173 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-45dc34-d7f945cff-rqxz2" podUID="96cf77ba-eb58-4e52-bab4-2a4e362dc95c" containerName="payload-pre-processing" containerID="cri-o://b7c2a09ef070bb42908aea672d1b7ce7404283fe14a3f92baf2f8456ea0580ac" gracePeriod=25 Oct 30 19:15:20.637674 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:20.637597 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1fba6e3-ld59t" Oct 30 19:15:20.688608 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:20.688587 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b963c8b4-j8jgz" Oct 30 19:15:20.689991 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:20.689967 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dab3a37-4bad-464c-b98f-5f6cc0592e9f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dab3a37-4bad-464c-b98f-5f6cc0592e9f-maas-api-xks-ca\") pod \"6dab3a37-4bad-464c-b98f-5f6cc0592e9f\" (UID: \"6dab3a37-4bad-464c-b98f-5f6cc0592e9f\") " Oct 30 19:15:20.690207 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:20.690188 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dab3a37-4bad-464c-b98f-5f6cc0592e9f-kube-api-access-9mjs9\" (UniqueName: \"kubernetes.io/projected/6dab3a37-4bad-464c-b98f-5f6cc0592e9f-kube-api-access-9mjs9\") pod \"6dab3a37-4bad-464c-b98f-5f6cc0592e9f\" (UID: \"6dab3a37-4bad-464c-b98f-5f6cc0592e9f\") " Oct 30 19:15:20.690271 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:20.690232 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dab3a37-4bad-464c-b98f-5f6cc0592e9f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dab3a37-4bad-464c-b98f-5f6cc0592e9f-maas-api-openshift-ca\") pod \"6dab3a37-4bad-464c-b98f-5f6cc0592e9f\" (UID: \"6dab3a37-4bad-464c-b98f-5f6cc0592e9f\") " Oct 30 19:15:20.690875 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:20.690835 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dab3a37-4bad-464c-b98f-5f6cc0592e9f-maas-api-openshift-ca" pod "6dab3a37-4bad-464c-b98f-5f6cc0592e9f" (UID: "6dab3a37-4bad-464c-b98f-5f6cc0592e9f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:20.693220 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:20.693191 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dab3a37-4bad-464c-b98f-5f6cc0592e9f-kube-api-access-9mjs9" pod "6dab3a37-4bad-464c-b98f-5f6cc0592e9f" (UID: "6dab3a37-4bad-464c-b98f-5f6cc0592e9f"). InnerVolumeSpecName "kube-api-access-9mjs9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:20.694971 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:20.694948 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dab3a37-4bad-464c-b98f-5f6cc0592e9f-maas-api-xks-ca" pod "6dab3a37-4bad-464c-b98f-5f6cc0592e9f" (UID: "6dab3a37-4bad-464c-b98f-5f6cc0592e9f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:15:20.699030 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:20.699011 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-45dc34-79f9556b56-st7qr" Oct 30 19:15:20.791686 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:20.791644 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/269e36c9-21e8-412e-b49e-e446b7d57def-kube-api-access-vnt22\" (UniqueName: \"kubernetes.io/projected/269e36c9-21e8-412e-b49e-e446b7d57def-kube-api-access-vnt22\") pod \"269e36c9-21e8-412e-b49e-e446b7d57def\" (UID: \"269e36c9-21e8-412e-b49e-e446b7d57def\") " Oct 30 19:15:20.791865 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:20.791725 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e-maas-api-xks-ca\") pod \"05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e\" (UID: \"05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e\") " Oct 30 19:15:20.791865 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:20.791783 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e-kube-api-access-kkbtp\" (UniqueName: \"kubernetes.io/projected/05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e-kube-api-access-kkbtp\") pod \"05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e\" (UID: \"05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e\") " Oct 30 19:15:20.791865 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:20.791831 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e-maas-api-openshift-ca\") pod \"05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e\" (UID: \"05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e\") " Oct 30 19:15:20.792084 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:20.792015 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/269e36c9-21e8-412e-b49e-e446b7d57def-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/269e36c9-21e8-412e-b49e-e446b7d57def-plugins-config-volume\") pod \"269e36c9-21e8-412e-b49e-e446b7d57def\" (UID: \"269e36c9-21e8-412e-b49e-e446b7d57def\") " Oct 30 19:15:20.792316 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:20.792286 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e-maas-api-openshift-ca" pod "05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e" (UID: "05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:20.792582 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:20.792443 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mjs9\" (UniqueName: \"kubernetes.io/projected/6dab3a37-4bad-464c-b98f-5f6cc0592e9f-kube-api-access-9mjs9\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:20.792582 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:20.792528 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dab3a37-4bad-464c-b98f-5f6cc0592e9f-maas-api-openshift-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:20.792582 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:20.792544 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dab3a37-4bad-464c-b98f-5f6cc0592e9f-maas-api-xks-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:20.792582 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:20.792558 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e-maas-api-openshift-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:20.792806 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:20.792706 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/269e36c9-21e8-412e-b49e-e446b7d57def-plugins-config-volume" pod "269e36c9-21e8-412e-b49e-e446b7d57def" (UID: "269e36c9-21e8-412e-b49e-e446b7d57def"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:20.794692 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:20.794671 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e-maas-api-xks-ca" pod "05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e" (UID: "05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:15:20.794792 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:20.794696 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/269e36c9-21e8-412e-b49e-e446b7d57def-kube-api-access-vnt22" pod "269e36c9-21e8-412e-b49e-e446b7d57def" (UID: "269e36c9-21e8-412e-b49e-e446b7d57def"). InnerVolumeSpecName "kube-api-access-vnt22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:20.794888 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:20.794792 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e-kube-api-access-kkbtp" pod "05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e" (UID: "05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e"). InnerVolumeSpecName "kube-api-access-kkbtp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:20.893326 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:20.893233 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnt22\" (UniqueName: \"kubernetes.io/projected/269e36c9-21e8-412e-b49e-e446b7d57def-kube-api-access-vnt22\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:20.893326 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:20.893266 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e-maas-api-xks-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:20.893326 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:20.893279 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkbtp\" (UniqueName: \"kubernetes.io/projected/05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e-kube-api-access-kkbtp\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:20.893326 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:20.893291 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/269e36c9-21e8-412e-b49e-e446b7d57def-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:21.422243 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:21.422196 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fe1374-5c8465bf67-bl75c" podUID="6170e608-fbfb-4cfe-acf6-2c09756134b5" containerName="payload-pre-processing" containerID="cri-o://04b2dee4587e21f965c8016a926185e606341aea019d4f1327971fcf1d3f7fbb" gracePeriod=25 Oct 30 19:15:21.479649 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:21.479608 4442 generic.go:356] "Generic (PLEG): container finished" podID="269e36c9-21e8-412e-b49e-e446b7d57def" containerID="a5556d595f6870a40a95db09538a26423c8457392bad43c25bd378fa6f31d6fc" exitCode=0 Oct 30 19:15:21.479854 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:21.479677 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-45dc34-79f9556b56-st7qr" Oct 30 19:15:21.479854 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:21.479804 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-45dc34-79f9556b56-st7qr" event={"ID":"269e36c9-21e8-412e-b49e-e446b7d57def","Type":"ContainerDied","Data":"a5556d595f6870a40a95db09538a26423c8457392bad43c25bd378fa6f31d6fc"} Oct 30 19:15:21.479994 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:21.479856 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-45dc34-79f9556b56-st7qr" event={"ID":"269e36c9-21e8-412e-b49e-e446b7d57def","Type":"ContainerDied","Data":"24bed1eb47bb4996d82c11a955960da28c84491b267b843ffb486148ad258eeb"} Oct 30 19:15:21.479994 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:21.479885 4442 scope.go:122] "RemoveContainer" containerID="a5556d595f6870a40a95db09538a26423c8457392bad43c25bd378fa6f31d6fc" Oct 30 19:15:21.483399 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:21.483100 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1fba6e3-ld59t" event={"ID":"6dab3a37-4bad-464c-b98f-5f6cc0592e9f","Type":"ContainerDied","Data":"23bcfad1a396c46988882f9e1c5edf53c62e42c44120602b24ce3faa330001e6"} Oct 30 19:15:21.483399 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:21.483168 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1fba6e3-ld59t" Oct 30 19:15:21.483399 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:21.483185 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23bcfad1a396c46988882f9e1c5edf53c62e42c44120602b24ce3faa330001e6" Oct 30 19:15:21.485960 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:21.485942 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b963c8b4-j8jgz" event={"ID":"05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e","Type":"ContainerDied","Data":"34000c86908b13fd54fef2fcbb86ddb1fc1ce19b4464aea3f92d03a3d93b6b48"} Oct 30 19:15:21.485960 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:21.485957 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b963c8b4-j8jgz" Oct 30 19:15:21.486114 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:21.485973 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34000c86908b13fd54fef2fcbb86ddb1fc1ce19b4464aea3f92d03a3d93b6b48" Oct 30 19:15:21.492689 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:21.492666 4442 scope.go:122] "RemoveContainer" containerID="a5556d595f6870a40a95db09538a26423c8457392bad43c25bd378fa6f31d6fc" Oct 30 19:15:21.492964 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:15:21.492945 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5556d595f6870a40a95db09538a26423c8457392bad43c25bd378fa6f31d6fc\": container with ID starting with a5556d595f6870a40a95db09538a26423c8457392bad43c25bd378fa6f31d6fc not found: ID does not exist" containerID="a5556d595f6870a40a95db09538a26423c8457392bad43c25bd378fa6f31d6fc" Oct 30 19:15:21.493038 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:21.492974 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5556d595f6870a40a95db09538a26423c8457392bad43c25bd378fa6f31d6fc"} err="failed to get container status \"a5556d595f6870a40a95db09538a26423c8457392bad43c25bd378fa6f31d6fc\": rpc error: code = NotFound desc = could not find container \"a5556d595f6870a40a95db09538a26423c8457392bad43c25bd378fa6f31d6fc\": container with ID starting with a5556d595f6870a40a95db09538a26423c8457392bad43c25bd378fa6f31d6fc not found: ID does not exist" Oct 30 19:15:21.569761 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:21.569727 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-45dc34-79f9556b56-st7qr"] Oct 30 19:15:21.572339 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:21.572312 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-45dc34-79f9556b56-st7qr"] Oct 30 19:15:21.929945 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:21.929901 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-921450-8d6546c9b-5pxxc" podUID="17bb49e3-80a9-4216-bc19-20b1679782a9" containerName="payload-processing" containerID="cri-o://6162680b62ce5d82dcccecc4dfd922c110a617f7fedee8a8d3a750c98f15febb" gracePeriod=25 Oct 30 19:15:21.962484 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:21.962422 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-921450-7f64fb4c79-xs76b" podUID="02cb3b89-6926-459a-a7ed-de16a0d30d6f" containerName="payload-pre-processing" containerID="cri-o://944ac059d0d326575fa0bd96694de71513b880a1af9e5757fc04490b5715dcbb" gracePeriod=25 Oct 30 19:15:23.072943 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:23.072910 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="269e36c9-21e8-412e-b49e-e446b7d57def" path="/var/lib/kubelet/pods/269e36c9-21e8-412e-b49e-e446b7d57def/volumes" Oct 30 19:15:23.092888 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:23.092851 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-45dc34-d7f945cff-rqxz2" podUID="96cf77ba-eb58-4e52-bab4-2a4e362dc95c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:15:25.966937 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:25.966893 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4" podUID="cf7d5f54-e091-4032-ab11-af2687e453c8" containerName="main" containerID="cri-o://2f553df37dc99256315d25087ceb4729bbe92e65039be46c311fcffbe3aeccd9" gracePeriod=45 Oct 30 19:15:26.219896 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:26.219829 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4" Oct 30 19:15:26.350598 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:26.350562 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf7d5f54-e091-4032-ab11-af2687e453c8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf7d5f54-e091-4032-ab11-af2687e453c8-tmp-dir\") pod \"cf7d5f54-e091-4032-ab11-af2687e453c8\" (UID: \"cf7d5f54-e091-4032-ab11-af2687e453c8\") " Oct 30 19:15:26.350871 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:26.350845 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf7d5f54-e091-4032-ab11-af2687e453c8-tmp-dir" pod "cf7d5f54-e091-4032-ab11-af2687e453c8" (UID: "cf7d5f54-e091-4032-ab11-af2687e453c8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:26.350974 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:26.350916 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf7d5f54-e091-4032-ab11-af2687e453c8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf7d5f54-e091-4032-ab11-af2687e453c8-model-cache\") pod \"cf7d5f54-e091-4032-ab11-af2687e453c8\" (UID: \"cf7d5f54-e091-4032-ab11-af2687e453c8\") " Oct 30 19:15:26.351043 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:26.350976 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf7d5f54-e091-4032-ab11-af2687e453c8-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf7d5f54-e091-4032-ab11-af2687e453c8-tls-certs\") pod \"cf7d5f54-e091-4032-ab11-af2687e453c8\" (UID: \"cf7d5f54-e091-4032-ab11-af2687e453c8\") " Oct 30 19:15:26.351043 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:26.351027 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf7d5f54-e091-4032-ab11-af2687e453c8-home\" (UniqueName: \"kubernetes.io/empty-dir/cf7d5f54-e091-4032-ab11-af2687e453c8-home\") pod \"cf7d5f54-e091-4032-ab11-af2687e453c8\" (UID: \"cf7d5f54-e091-4032-ab11-af2687e453c8\") " Oct 30 19:15:26.351178 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:26.351106 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf7d5f54-e091-4032-ab11-af2687e453c8-kube-api-access-d964n\" (UniqueName: \"kubernetes.io/projected/cf7d5f54-e091-4032-ab11-af2687e453c8-kube-api-access-d964n\") pod \"cf7d5f54-e091-4032-ab11-af2687e453c8\" (UID: \"cf7d5f54-e091-4032-ab11-af2687e453c8\") " Oct 30 19:15:26.351245 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:26.351164 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf7d5f54-e091-4032-ab11-af2687e453c8-model-cache" pod "cf7d5f54-e091-4032-ab11-af2687e453c8" (UID: "cf7d5f54-e091-4032-ab11-af2687e453c8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:26.351245 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:26.351190 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf7d5f54-e091-4032-ab11-af2687e453c8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf7d5f54-e091-4032-ab11-af2687e453c8-dshm\") pod \"cf7d5f54-e091-4032-ab11-af2687e453c8\" (UID: \"cf7d5f54-e091-4032-ab11-af2687e453c8\") " Oct 30 19:15:26.351387 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:26.351367 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf7d5f54-e091-4032-ab11-af2687e453c8-home" pod "cf7d5f54-e091-4032-ab11-af2687e453c8" (UID: "cf7d5f54-e091-4032-ab11-af2687e453c8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:26.351979 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:26.351600 4442 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf7d5f54-e091-4032-ab11-af2687e453c8-home\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:26.351979 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:26.351621 4442 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf7d5f54-e091-4032-ab11-af2687e453c8-tmp-dir\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:26.351979 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:26.351635 4442 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf7d5f54-e091-4032-ab11-af2687e453c8-model-cache\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:26.354686 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:26.354644 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf7d5f54-e091-4032-ab11-af2687e453c8-dshm" pod "cf7d5f54-e091-4032-ab11-af2687e453c8" (UID: "cf7d5f54-e091-4032-ab11-af2687e453c8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:26.360633 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:26.360602 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf7d5f54-e091-4032-ab11-af2687e453c8-kube-api-access-d964n" pod "cf7d5f54-e091-4032-ab11-af2687e453c8" (UID: "cf7d5f54-e091-4032-ab11-af2687e453c8"). InnerVolumeSpecName "kube-api-access-d964n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:26.360829 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:26.360804 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf7d5f54-e091-4032-ab11-af2687e453c8-tls-certs" pod "cf7d5f54-e091-4032-ab11-af2687e453c8" (UID: "cf7d5f54-e091-4032-ab11-af2687e453c8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:15:26.452520 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:26.452482 4442 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf7d5f54-e091-4032-ab11-af2687e453c8-tls-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:26.452520 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:26.452520 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d964n\" (UniqueName: \"kubernetes.io/projected/cf7d5f54-e091-4032-ab11-af2687e453c8-kube-api-access-d964n\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:26.452735 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:26.452534 4442 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf7d5f54-e091-4032-ab11-af2687e453c8-dshm\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:26.522726 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:26.522629 4442 generic.go:356] "Generic (PLEG): container finished" podID="cf7d5f54-e091-4032-ab11-af2687e453c8" containerID="2f553df37dc99256315d25087ceb4729bbe92e65039be46c311fcffbe3aeccd9" exitCode=0 Oct 30 19:15:26.522726 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:26.522678 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4" Oct 30 19:15:26.522942 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:26.522732 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4" event={"ID":"cf7d5f54-e091-4032-ab11-af2687e453c8","Type":"ContainerDied","Data":"2f553df37dc99256315d25087ceb4729bbe92e65039be46c311fcffbe3aeccd9"} Oct 30 19:15:26.522942 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:26.522778 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4" event={"ID":"cf7d5f54-e091-4032-ab11-af2687e453c8","Type":"ContainerDied","Data":"63b8c4f7e77401768a7fb81b3a73f686b2dc3757b3e4d8aac2736897db7d8714"} Oct 30 19:15:26.522942 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:26.522797 4442 scope.go:122] "RemoveContainer" containerID="2f553df37dc99256315d25087ceb4729bbe92e65039be46c311fcffbe3aeccd9" Oct 30 19:15:26.533670 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:26.533648 4442 scope.go:122] "RemoveContainer" containerID="2f553df37dc99256315d25087ceb4729bbe92e65039be46c311fcffbe3aeccd9" Oct 30 19:15:26.533957 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:15:26.533929 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f553df37dc99256315d25087ceb4729bbe92e65039be46c311fcffbe3aeccd9\": container with ID starting with 2f553df37dc99256315d25087ceb4729bbe92e65039be46c311fcffbe3aeccd9 not found: ID does not exist" containerID="2f553df37dc99256315d25087ceb4729bbe92e65039be46c311fcffbe3aeccd9" Oct 30 19:15:26.534019 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:26.533968 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f553df37dc99256315d25087ceb4729bbe92e65039be46c311fcffbe3aeccd9"} err="failed to get container status \"2f553df37dc99256315d25087ceb4729bbe92e65039be46c311fcffbe3aeccd9\": rpc error: code = NotFound desc = could not find container \"2f553df37dc99256315d25087ceb4729bbe92e65039be46c311fcffbe3aeccd9\": container with ID starting with 2f553df37dc99256315d25087ceb4729bbe92e65039be46c311fcffbe3aeccd9 not found: ID does not exist" Oct 30 19:15:26.576004 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:26.575970 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4"] Oct 30 19:15:26.578446 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:26.578427 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4"] Oct 30 19:15:27.063393 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:27.063361 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf7d5f54-e091-4032-ab11-af2687e453c8" path="/var/lib/kubelet/pods/cf7d5f54-e091-4032-ab11-af2687e453c8/volumes" Oct 30 19:15:27.171533 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:27.171484 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/distinct-2-w0-921450-kserve-784954bd5c-2ttw4" podUID="cf7d5f54-e091-4032-ab11-af2687e453c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 19:15:29.276466 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:29.276432 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fe1374-5c8465bf67-bl75c" podUID="6170e608-fbfb-4cfe-acf6-2c09756134b5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:15:29.405995 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:29.405963 4442 scope.go:122] "RemoveContainer" containerID="25c81b4c4da19c1e4e250313672fd83a2493e200ea23b6a1863a4e40fcb74970" Oct 30 19:15:29.415710 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:29.415690 4442 scope.go:122] "RemoveContainer" containerID="733abb0344a8b76e9a54e07a79abe990dd59ad56da02401951b833e172bbe4d4" Oct 30 19:15:30.439765 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:30.439732 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-45dc34-6cc4b7467c-8fgqt"] Oct 30 19:15:30.440259 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:30.439938 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-45dc34-6cc4b7467c-8fgqt" podUID="13d41d70-3283-4135-9f4e-3ff0c53cec22" containerName="maas-api" containerID="cri-o://23ccb882d0e307887df5c75c094b0293b215e86956e50fb35da1a3ac28b78d40" gracePeriod=30 Oct 30 19:15:30.710992 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:30.710559 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-45dc34-6cc4b7467c-8fgqt" Oct 30 19:15:30.803200 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:30.803162 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13d41d70-3283-4135-9f4e-3ff0c53cec22-metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d41d70-3283-4135-9f4e-3ff0c53cec22-metrics-certs\") pod \"13d41d70-3283-4135-9f4e-3ff0c53cec22\" (UID: \"13d41d70-3283-4135-9f4e-3ff0c53cec22\") " Oct 30 19:15:30.803412 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:30.803223 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13d41d70-3283-4135-9f4e-3ff0c53cec22-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13d41d70-3283-4135-9f4e-3ff0c53cec22-maas-api-tls\") pod \"13d41d70-3283-4135-9f4e-3ff0c53cec22\" (UID: \"13d41d70-3283-4135-9f4e-3ff0c53cec22\") " Oct 30 19:15:30.803412 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:30.803274 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13d41d70-3283-4135-9f4e-3ff0c53cec22-kube-api-access-kgtww\" (UniqueName: \"kubernetes.io/projected/13d41d70-3283-4135-9f4e-3ff0c53cec22-kube-api-access-kgtww\") pod \"13d41d70-3283-4135-9f4e-3ff0c53cec22\" (UID: \"13d41d70-3283-4135-9f4e-3ff0c53cec22\") " Oct 30 19:15:30.803412 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:30.803301 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13d41d70-3283-4135-9f4e-3ff0c53cec22-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d41d70-3283-4135-9f4e-3ff0c53cec22-openshift-service-ca\") pod \"13d41d70-3283-4135-9f4e-3ff0c53cec22\" (UID: \"13d41d70-3283-4135-9f4e-3ff0c53cec22\") " Oct 30 19:15:30.803805 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:30.803760 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13d41d70-3283-4135-9f4e-3ff0c53cec22-openshift-service-ca" pod "13d41d70-3283-4135-9f4e-3ff0c53cec22" (UID: "13d41d70-3283-4135-9f4e-3ff0c53cec22"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:30.805557 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:30.805528 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13d41d70-3283-4135-9f4e-3ff0c53cec22-metrics-certs" pod "13d41d70-3283-4135-9f4e-3ff0c53cec22" (UID: "13d41d70-3283-4135-9f4e-3ff0c53cec22"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:15:30.806092 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:30.806068 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13d41d70-3283-4135-9f4e-3ff0c53cec22-maas-api-tls" pod "13d41d70-3283-4135-9f4e-3ff0c53cec22" (UID: "13d41d70-3283-4135-9f4e-3ff0c53cec22"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:15:30.806182 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:30.806102 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13d41d70-3283-4135-9f4e-3ff0c53cec22-kube-api-access-kgtww" pod "13d41d70-3283-4135-9f4e-3ff0c53cec22" (UID: "13d41d70-3283-4135-9f4e-3ff0c53cec22"). InnerVolumeSpecName "kube-api-access-kgtww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:30.905313 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:30.905273 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d41d70-3283-4135-9f4e-3ff0c53cec22-metrics-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:30.905522 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:30.905312 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13d41d70-3283-4135-9f4e-3ff0c53cec22-maas-api-tls\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:30.905522 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:30.905373 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgtww\" (UniqueName: \"kubernetes.io/projected/13d41d70-3283-4135-9f4e-3ff0c53cec22-kube-api-access-kgtww\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:30.905522 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:30.905386 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d41d70-3283-4135-9f4e-3ff0c53cec22-openshift-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:31.064400 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:31.064306 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-921450-7f64fb4c79-xs76b" podUID="02cb3b89-6926-459a-a7ed-de16a0d30d6f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:15:31.064626 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:31.064586 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-921450-8d6546c9b-5pxxc" podUID="17bb49e3-80a9-4216-bc19-20b1679782a9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:15:31.558959 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:31.558921 4442 generic.go:356] "Generic (PLEG): container finished" podID="13d41d70-3283-4135-9f4e-3ff0c53cec22" containerID="23ccb882d0e307887df5c75c094b0293b215e86956e50fb35da1a3ac28b78d40" exitCode=0 Oct 30 19:15:31.559433 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:31.559001 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-45dc34-6cc4b7467c-8fgqt" Oct 30 19:15:31.559433 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:31.559016 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-45dc34-6cc4b7467c-8fgqt" event={"ID":"13d41d70-3283-4135-9f4e-3ff0c53cec22","Type":"ContainerDied","Data":"23ccb882d0e307887df5c75c094b0293b215e86956e50fb35da1a3ac28b78d40"} Oct 30 19:15:31.559433 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:31.559069 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-45dc34-6cc4b7467c-8fgqt" event={"ID":"13d41d70-3283-4135-9f4e-3ff0c53cec22","Type":"ContainerDied","Data":"96cc79e782238181c753d0e5b7498121979219d235c9f98eb79b81c435cbc1ff"} Oct 30 19:15:31.559433 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:31.559091 4442 scope.go:122] "RemoveContainer" containerID="23ccb882d0e307887df5c75c094b0293b215e86956e50fb35da1a3ac28b78d40" Oct 30 19:15:31.569971 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:31.569945 4442 scope.go:122] "RemoveContainer" containerID="23ccb882d0e307887df5c75c094b0293b215e86956e50fb35da1a3ac28b78d40" Oct 30 19:15:31.570265 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:15:31.570241 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23ccb882d0e307887df5c75c094b0293b215e86956e50fb35da1a3ac28b78d40\": container with ID starting with 23ccb882d0e307887df5c75c094b0293b215e86956e50fb35da1a3ac28b78d40 not found: ID does not exist" containerID="23ccb882d0e307887df5c75c094b0293b215e86956e50fb35da1a3ac28b78d40" Oct 30 19:15:31.570362 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:31.570272 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23ccb882d0e307887df5c75c094b0293b215e86956e50fb35da1a3ac28b78d40"} err="failed to get container status \"23ccb882d0e307887df5c75c094b0293b215e86956e50fb35da1a3ac28b78d40\": rpc error: code = NotFound desc = could not find container \"23ccb882d0e307887df5c75c094b0293b215e86956e50fb35da1a3ac28b78d40\": container with ID starting with 23ccb882d0e307887df5c75c094b0293b215e86956e50fb35da1a3ac28b78d40 not found: ID does not exist" Oct 30 19:15:31.613072 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:31.613044 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-45dc34-6cc4b7467c-8fgqt"] Oct 30 19:15:31.624748 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:31.624721 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-45dc34-6cc4b7467c-8fgqt"] Oct 30 19:15:33.064439 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:33.064408 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13d41d70-3283-4135-9f4e-3ff0c53cec22" path="/var/lib/kubelet/pods/13d41d70-3283-4135-9f4e-3ff0c53cec22/volumes" Oct 30 19:15:33.092896 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:33.092856 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-45dc34-d7f945cff-rqxz2" podUID="96cf77ba-eb58-4e52-bab4-2a4e362dc95c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:15:34.099737 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:34.099702 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-921450-7cc76dbcd5-vwjkc"] Oct 30 19:15:34.100186 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:34.099933 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-921450-7cc76dbcd5-vwjkc" podUID="6bc83ed4-302b-466c-9599-b022b3f0afb1" containerName="maas-api" containerID="cri-o://e439cf17531bbf157d05ce8d4d2a1e30ea03799edb4583965f6cf29acea39b5c" gracePeriod=30 Oct 30 19:15:34.387646 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:34.387621 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-921450-7cc76dbcd5-vwjkc" Oct 30 19:15:34.556204 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:34.548019 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bc83ed4-302b-466c-9599-b022b3f0afb1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bc83ed4-302b-466c-9599-b022b3f0afb1-maas-api-tls\") pod \"6bc83ed4-302b-466c-9599-b022b3f0afb1\" (UID: \"6bc83ed4-302b-466c-9599-b022b3f0afb1\") " Oct 30 19:15:34.556204 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:34.548188 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bc83ed4-302b-466c-9599-b022b3f0afb1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bc83ed4-302b-466c-9599-b022b3f0afb1-metrics-certs\") pod \"6bc83ed4-302b-466c-9599-b022b3f0afb1\" (UID: \"6bc83ed4-302b-466c-9599-b022b3f0afb1\") " Oct 30 19:15:34.556204 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:34.548279 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bc83ed4-302b-466c-9599-b022b3f0afb1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bc83ed4-302b-466c-9599-b022b3f0afb1-openshift-service-ca\") pod \"6bc83ed4-302b-466c-9599-b022b3f0afb1\" (UID: \"6bc83ed4-302b-466c-9599-b022b3f0afb1\") " Oct 30 19:15:34.556204 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:34.548324 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bc83ed4-302b-466c-9599-b022b3f0afb1-kube-api-access-6j2k2\" (UniqueName: \"kubernetes.io/projected/6bc83ed4-302b-466c-9599-b022b3f0afb1-kube-api-access-6j2k2\") pod \"6bc83ed4-302b-466c-9599-b022b3f0afb1\" (UID: \"6bc83ed4-302b-466c-9599-b022b3f0afb1\") " Oct 30 19:15:34.556204 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:34.550835 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bc83ed4-302b-466c-9599-b022b3f0afb1-openshift-service-ca" pod "6bc83ed4-302b-466c-9599-b022b3f0afb1" (UID: "6bc83ed4-302b-466c-9599-b022b3f0afb1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:34.556204 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:34.555742 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bc83ed4-302b-466c-9599-b022b3f0afb1-kube-api-access-6j2k2" pod "6bc83ed4-302b-466c-9599-b022b3f0afb1" (UID: "6bc83ed4-302b-466c-9599-b022b3f0afb1"). InnerVolumeSpecName "kube-api-access-6j2k2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:34.556204 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:34.555751 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bc83ed4-302b-466c-9599-b022b3f0afb1-metrics-certs" pod "6bc83ed4-302b-466c-9599-b022b3f0afb1" (UID: "6bc83ed4-302b-466c-9599-b022b3f0afb1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:15:34.560592 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:34.560559 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bc83ed4-302b-466c-9599-b022b3f0afb1-maas-api-tls" pod "6bc83ed4-302b-466c-9599-b022b3f0afb1" (UID: "6bc83ed4-302b-466c-9599-b022b3f0afb1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:15:34.589369 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:34.589335 4442 generic.go:356] "Generic (PLEG): container finished" podID="6bc83ed4-302b-466c-9599-b022b3f0afb1" containerID="e439cf17531bbf157d05ce8d4d2a1e30ea03799edb4583965f6cf29acea39b5c" exitCode=0 Oct 30 19:15:34.589565 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:34.589383 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-921450-7cc76dbcd5-vwjkc" Oct 30 19:15:34.589565 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:34.589419 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-921450-7cc76dbcd5-vwjkc" event={"ID":"6bc83ed4-302b-466c-9599-b022b3f0afb1","Type":"ContainerDied","Data":"e439cf17531bbf157d05ce8d4d2a1e30ea03799edb4583965f6cf29acea39b5c"} Oct 30 19:15:34.589565 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:34.589472 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-921450-7cc76dbcd5-vwjkc" event={"ID":"6bc83ed4-302b-466c-9599-b022b3f0afb1","Type":"ContainerDied","Data":"fe05c01fd517ebdaaa02280655346eb447646cb1cb99d73a213eee0bca75a3f4"} Oct 30 19:15:34.589565 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:34.589495 4442 scope.go:122] "RemoveContainer" containerID="e439cf17531bbf157d05ce8d4d2a1e30ea03799edb4583965f6cf29acea39b5c" Oct 30 19:15:34.603966 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:34.603943 4442 scope.go:122] "RemoveContainer" containerID="e439cf17531bbf157d05ce8d4d2a1e30ea03799edb4583965f6cf29acea39b5c" Oct 30 19:15:34.604271 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:15:34.604248 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e439cf17531bbf157d05ce8d4d2a1e30ea03799edb4583965f6cf29acea39b5c\": container with ID starting with e439cf17531bbf157d05ce8d4d2a1e30ea03799edb4583965f6cf29acea39b5c not found: ID does not exist" containerID="e439cf17531bbf157d05ce8d4d2a1e30ea03799edb4583965f6cf29acea39b5c" Oct 30 19:15:34.604355 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:34.604280 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e439cf17531bbf157d05ce8d4d2a1e30ea03799edb4583965f6cf29acea39b5c"} err="failed to get container status \"e439cf17531bbf157d05ce8d4d2a1e30ea03799edb4583965f6cf29acea39b5c\": rpc error: code = NotFound desc = could not find container \"e439cf17531bbf157d05ce8d4d2a1e30ea03799edb4583965f6cf29acea39b5c\": container with ID starting with e439cf17531bbf157d05ce8d4d2a1e30ea03799edb4583965f6cf29acea39b5c not found: ID does not exist" Oct 30 19:15:34.642793 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:34.642703 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-921450-7cc76dbcd5-vwjkc"] Oct 30 19:15:34.645191 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:34.645163 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-921450-7cc76dbcd5-vwjkc"] Oct 30 19:15:34.649277 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:34.649250 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bc83ed4-302b-466c-9599-b022b3f0afb1-metrics-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:34.649277 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:34.649273 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bc83ed4-302b-466c-9599-b022b3f0afb1-openshift-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:34.649416 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:34.649303 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j2k2\" (UniqueName: \"kubernetes.io/projected/6bc83ed4-302b-466c-9599-b022b3f0afb1-kube-api-access-6j2k2\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:34.649416 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:34.649313 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bc83ed4-302b-466c-9599-b022b3f0afb1-maas-api-tls\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:34.876369 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:34.876337 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-921450-mtrn6"] Oct 30 19:15:34.881891 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:34.881865 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-921450-mtrn6"] Oct 30 19:15:35.064011 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:35.063977 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bc83ed4-302b-466c-9599-b022b3f0afb1" path="/var/lib/kubelet/pods/6bc83ed4-302b-466c-9599-b022b3f0afb1/volumes" Oct 30 19:15:35.064585 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:35.064566 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4c7339b-0d56-4483-8323-2b9d0059f21d" path="/var/lib/kubelet/pods/e4c7339b-0d56-4483-8323-2b9d0059f21d/volumes" Oct 30 19:15:39.276702 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:39.276664 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fe1374-5c8465bf67-bl75c" podUID="6170e608-fbfb-4cfe-acf6-2c09756134b5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:15:40.074656 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:40.074623 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb35b1f4-nghc5"] Oct 30 19:15:40.085965 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:40.085937 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb35b1f4-nghc5"] Oct 30 19:15:40.632951 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:40.632916 4442 generic.go:356] "Generic (PLEG): container finished" podID="96cf77ba-eb58-4e52-bab4-2a4e362dc95c" containerID="b7c2a09ef070bb42908aea672d1b7ce7404283fe14a3f92baf2f8456ea0580ac" exitCode=0 Oct 30 19:15:40.633379 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:40.632957 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-45dc34-d7f945cff-rqxz2" event={"ID":"96cf77ba-eb58-4e52-bab4-2a4e362dc95c","Type":"ContainerDied","Data":"b7c2a09ef070bb42908aea672d1b7ce7404283fe14a3f92baf2f8456ea0580ac"} Oct 30 19:15:40.667736 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:40.667675 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-45dc34-d7f945cff-rqxz2" Oct 30 19:15:40.713873 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:40.713842 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96cf77ba-eb58-4e52-bab4-2a4e362dc95c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96cf77ba-eb58-4e52-bab4-2a4e362dc95c-plugins-config-volume\") pod \"96cf77ba-eb58-4e52-bab4-2a4e362dc95c\" (UID: \"96cf77ba-eb58-4e52-bab4-2a4e362dc95c\") " Oct 30 19:15:40.714069 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:40.713913 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96cf77ba-eb58-4e52-bab4-2a4e362dc95c-kube-api-access-q8zcz\" (UniqueName: \"kubernetes.io/projected/96cf77ba-eb58-4e52-bab4-2a4e362dc95c-kube-api-access-q8zcz\") pod \"96cf77ba-eb58-4e52-bab4-2a4e362dc95c\" (UID: \"96cf77ba-eb58-4e52-bab4-2a4e362dc95c\") " Oct 30 19:15:40.714291 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:40.714270 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96cf77ba-eb58-4e52-bab4-2a4e362dc95c-plugins-config-volume" pod "96cf77ba-eb58-4e52-bab4-2a4e362dc95c" (UID: "96cf77ba-eb58-4e52-bab4-2a4e362dc95c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:40.716399 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:40.716370 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96cf77ba-eb58-4e52-bab4-2a4e362dc95c-kube-api-access-q8zcz" pod "96cf77ba-eb58-4e52-bab4-2a4e362dc95c" (UID: "96cf77ba-eb58-4e52-bab4-2a4e362dc95c"). InnerVolumeSpecName "kube-api-access-q8zcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:40.814784 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:40.814746 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96cf77ba-eb58-4e52-bab4-2a4e362dc95c-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:40.814784 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:40.814780 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8zcz\" (UniqueName: \"kubernetes.io/projected/96cf77ba-eb58-4e52-bab4-2a4e362dc95c-kube-api-access-q8zcz\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:41.063066 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:41.062971 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34f236f3-7dde-40d7-a8bc-41c56ede366c" path="/var/lib/kubelet/pods/34f236f3-7dde-40d7-a8bc-41c56ede366c/volumes" Oct 30 19:15:41.064048 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:41.064018 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-921450-8d6546c9b-5pxxc" podUID="17bb49e3-80a9-4216-bc19-20b1679782a9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:15:41.064048 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:41.064029 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-921450-7f64fb4c79-xs76b" podUID="02cb3b89-6926-459a-a7ed-de16a0d30d6f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:15:41.646496 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:41.646466 4442 generic.go:356] "Generic (PLEG): container finished" podID="6170e608-fbfb-4cfe-acf6-2c09756134b5" containerID="04b2dee4587e21f965c8016a926185e606341aea019d4f1327971fcf1d3f7fbb" exitCode=0 Oct 30 19:15:41.647036 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:41.646556 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fe1374-5c8465bf67-bl75c" event={"ID":"6170e608-fbfb-4cfe-acf6-2c09756134b5","Type":"ContainerDied","Data":"04b2dee4587e21f965c8016a926185e606341aea019d4f1327971fcf1d3f7fbb"} Oct 30 19:15:41.653453 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:41.653418 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-45dc34-d7f945cff-rqxz2" event={"ID":"96cf77ba-eb58-4e52-bab4-2a4e362dc95c","Type":"ContainerDied","Data":"c6c7dcbf52efb355859309265f24afd2132105bfac3857ed89c9ce7728f019a2"} Oct 30 19:15:41.653623 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:41.653466 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-45dc34-d7f945cff-rqxz2" Oct 30 19:15:41.653623 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:41.653476 4442 scope.go:122] "RemoveContainer" containerID="b7c2a09ef070bb42908aea672d1b7ce7404283fe14a3f92baf2f8456ea0580ac" Oct 30 19:15:41.685887 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:41.685820 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fe1374-5c8465bf67-bl75c" Oct 30 19:15:41.709426 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:41.709391 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-45dc34-d7f945cff-rqxz2"] Oct 30 19:15:41.712910 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:41.712885 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-45dc34-d7f945cff-rqxz2"] Oct 30 19:15:41.722869 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:41.722845 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6170e608-fbfb-4cfe-acf6-2c09756134b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6170e608-fbfb-4cfe-acf6-2c09756134b5-plugins-config-volume\") pod \"6170e608-fbfb-4cfe-acf6-2c09756134b5\" (UID: \"6170e608-fbfb-4cfe-acf6-2c09756134b5\") " Oct 30 19:15:41.723039 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:41.722945 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6170e608-fbfb-4cfe-acf6-2c09756134b5-kube-api-access-xgl6k\" (UniqueName: \"kubernetes.io/projected/6170e608-fbfb-4cfe-acf6-2c09756134b5-kube-api-access-xgl6k\") pod \"6170e608-fbfb-4cfe-acf6-2c09756134b5\" (UID: \"6170e608-fbfb-4cfe-acf6-2c09756134b5\") " Oct 30 19:15:41.723245 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:41.723222 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6170e608-fbfb-4cfe-acf6-2c09756134b5-plugins-config-volume" pod "6170e608-fbfb-4cfe-acf6-2c09756134b5" (UID: "6170e608-fbfb-4cfe-acf6-2c09756134b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:41.725126 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:41.725104 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6170e608-fbfb-4cfe-acf6-2c09756134b5-kube-api-access-xgl6k" pod "6170e608-fbfb-4cfe-acf6-2c09756134b5" (UID: "6170e608-fbfb-4cfe-acf6-2c09756134b5"). InnerVolumeSpecName "kube-api-access-xgl6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:41.824649 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:41.824618 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6170e608-fbfb-4cfe-acf6-2c09756134b5-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:41.824649 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:41.824644 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgl6k\" (UniqueName: \"kubernetes.io/projected/6170e608-fbfb-4cfe-acf6-2c09756134b5-kube-api-access-xgl6k\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:42.203950 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:42.203915 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-921450-8d6546c9b-5pxxc" Oct 30 19:15:42.228744 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:42.228719 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17bb49e3-80a9-4216-bc19-20b1679782a9-kube-api-access-t5pg8\" (UniqueName: \"kubernetes.io/projected/17bb49e3-80a9-4216-bc19-20b1679782a9-kube-api-access-t5pg8\") pod \"17bb49e3-80a9-4216-bc19-20b1679782a9\" (UID: \"17bb49e3-80a9-4216-bc19-20b1679782a9\") " Oct 30 19:15:42.228880 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:42.228807 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17bb49e3-80a9-4216-bc19-20b1679782a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17bb49e3-80a9-4216-bc19-20b1679782a9-plugins-config-volume\") pod \"17bb49e3-80a9-4216-bc19-20b1679782a9\" (UID: \"17bb49e3-80a9-4216-bc19-20b1679782a9\") " Oct 30 19:15:42.229201 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:42.229178 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17bb49e3-80a9-4216-bc19-20b1679782a9-plugins-config-volume" pod "17bb49e3-80a9-4216-bc19-20b1679782a9" (UID: "17bb49e3-80a9-4216-bc19-20b1679782a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:42.231303 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:42.231282 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17bb49e3-80a9-4216-bc19-20b1679782a9-kube-api-access-t5pg8" pod "17bb49e3-80a9-4216-bc19-20b1679782a9" (UID: "17bb49e3-80a9-4216-bc19-20b1679782a9"). InnerVolumeSpecName "kube-api-access-t5pg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:42.234613 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:42.234594 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-921450-7f64fb4c79-xs76b" Oct 30 19:15:42.329930 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:42.329901 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02cb3b89-6926-459a-a7ed-de16a0d30d6f-kube-api-access-wjrgq\" (UniqueName: \"kubernetes.io/projected/02cb3b89-6926-459a-a7ed-de16a0d30d6f-kube-api-access-wjrgq\") pod \"02cb3b89-6926-459a-a7ed-de16a0d30d6f\" (UID: \"02cb3b89-6926-459a-a7ed-de16a0d30d6f\") " Oct 30 19:15:42.330165 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:42.329999 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02cb3b89-6926-459a-a7ed-de16a0d30d6f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02cb3b89-6926-459a-a7ed-de16a0d30d6f-plugins-config-volume\") pod \"02cb3b89-6926-459a-a7ed-de16a0d30d6f\" (UID: \"02cb3b89-6926-459a-a7ed-de16a0d30d6f\") " Oct 30 19:15:42.330259 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:42.330183 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5pg8\" (UniqueName: \"kubernetes.io/projected/17bb49e3-80a9-4216-bc19-20b1679782a9-kube-api-access-t5pg8\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:42.330259 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:42.330194 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17bb49e3-80a9-4216-bc19-20b1679782a9-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:42.330381 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:42.330361 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02cb3b89-6926-459a-a7ed-de16a0d30d6f-plugins-config-volume" pod "02cb3b89-6926-459a-a7ed-de16a0d30d6f" (UID: "02cb3b89-6926-459a-a7ed-de16a0d30d6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:42.332367 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:42.332334 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02cb3b89-6926-459a-a7ed-de16a0d30d6f-kube-api-access-wjrgq" pod "02cb3b89-6926-459a-a7ed-de16a0d30d6f" (UID: "02cb3b89-6926-459a-a7ed-de16a0d30d6f"). InnerVolumeSpecName "kube-api-access-wjrgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:42.430868 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:42.430838 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjrgq\" (UniqueName: \"kubernetes.io/projected/02cb3b89-6926-459a-a7ed-de16a0d30d6f-kube-api-access-wjrgq\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:42.430868 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:42.430866 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02cb3b89-6926-459a-a7ed-de16a0d30d6f-plugins-config-volume\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:42.661620 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:42.661586 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fe1374-5c8465bf67-bl75c" Oct 30 19:15:42.662106 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:42.661591 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fe1374-5c8465bf67-bl75c" event={"ID":"6170e608-fbfb-4cfe-acf6-2c09756134b5","Type":"ContainerDied","Data":"e2f6928c84aead200706f29214237d94b7cabd5beafe8d2fc019084362a5df63"} Oct 30 19:15:42.662106 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:42.661726 4442 scope.go:122] "RemoveContainer" containerID="04b2dee4587e21f965c8016a926185e606341aea019d4f1327971fcf1d3f7fbb" Oct 30 19:15:42.663556 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:42.663520 4442 generic.go:356] "Generic (PLEG): container finished" podID="17bb49e3-80a9-4216-bc19-20b1679782a9" containerID="6162680b62ce5d82dcccecc4dfd922c110a617f7fedee8a8d3a750c98f15febb" exitCode=0 Oct 30 19:15:42.663669 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:42.663561 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-921450-8d6546c9b-5pxxc" event={"ID":"17bb49e3-80a9-4216-bc19-20b1679782a9","Type":"ContainerDied","Data":"6162680b62ce5d82dcccecc4dfd922c110a617f7fedee8a8d3a750c98f15febb"} Oct 30 19:15:42.663669 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:42.663580 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-921450-8d6546c9b-5pxxc" Oct 30 19:15:42.663669 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:42.663620 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-921450-8d6546c9b-5pxxc" event={"ID":"17bb49e3-80a9-4216-bc19-20b1679782a9","Type":"ContainerDied","Data":"0727b8630a7383cac66c687913d3be6802b9c5904e1298d9df601624b4504fde"} Oct 30 19:15:42.665324 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:42.665299 4442 generic.go:356] "Generic (PLEG): container finished" podID="02cb3b89-6926-459a-a7ed-de16a0d30d6f" containerID="944ac059d0d326575fa0bd96694de71513b880a1af9e5757fc04490b5715dcbb" exitCode=0 Oct 30 19:15:42.665451 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:42.665341 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-921450-7f64fb4c79-xs76b" Oct 30 19:15:42.665451 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:42.665428 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-921450-7f64fb4c79-xs76b" event={"ID":"02cb3b89-6926-459a-a7ed-de16a0d30d6f","Type":"ContainerDied","Data":"944ac059d0d326575fa0bd96694de71513b880a1af9e5757fc04490b5715dcbb"} Oct 30 19:15:42.665597 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:42.665461 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-921450-7f64fb4c79-xs76b" event={"ID":"02cb3b89-6926-459a-a7ed-de16a0d30d6f","Type":"ContainerDied","Data":"732b1ed0477e53371eeea9c251899c55c10e72832362b69bdb9da0616e84afca"} Oct 30 19:15:42.675214 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:42.675076 4442 scope.go:122] "RemoveContainer" containerID="6162680b62ce5d82dcccecc4dfd922c110a617f7fedee8a8d3a750c98f15febb" Oct 30 19:15:42.686553 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:42.686532 4442 scope.go:122] "RemoveContainer" containerID="6162680b62ce5d82dcccecc4dfd922c110a617f7fedee8a8d3a750c98f15febb" Oct 30 19:15:42.686885 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:15:42.686866 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6162680b62ce5d82dcccecc4dfd922c110a617f7fedee8a8d3a750c98f15febb\": container with ID starting with 6162680b62ce5d82dcccecc4dfd922c110a617f7fedee8a8d3a750c98f15febb not found: ID does not exist" containerID="6162680b62ce5d82dcccecc4dfd922c110a617f7fedee8a8d3a750c98f15febb" Oct 30 19:15:42.686963 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:42.686892 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6162680b62ce5d82dcccecc4dfd922c110a617f7fedee8a8d3a750c98f15febb"} err="failed to get container status \"6162680b62ce5d82dcccecc4dfd922c110a617f7fedee8a8d3a750c98f15febb\": rpc error: code = NotFound desc = could not find container \"6162680b62ce5d82dcccecc4dfd922c110a617f7fedee8a8d3a750c98f15febb\": container with ID starting with 6162680b62ce5d82dcccecc4dfd922c110a617f7fedee8a8d3a750c98f15febb not found: ID does not exist" Oct 30 19:15:42.686963 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:42.686908 4442 scope.go:122] "RemoveContainer" containerID="944ac059d0d326575fa0bd96694de71513b880a1af9e5757fc04490b5715dcbb" Oct 30 19:15:42.697540 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:42.697519 4442 scope.go:122] "RemoveContainer" containerID="944ac059d0d326575fa0bd96694de71513b880a1af9e5757fc04490b5715dcbb" Oct 30 19:15:42.697842 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:15:42.697819 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"944ac059d0d326575fa0bd96694de71513b880a1af9e5757fc04490b5715dcbb\": container with ID starting with 944ac059d0d326575fa0bd96694de71513b880a1af9e5757fc04490b5715dcbb not found: ID does not exist" containerID="944ac059d0d326575fa0bd96694de71513b880a1af9e5757fc04490b5715dcbb" Oct 30 19:15:42.697920 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:42.697849 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"944ac059d0d326575fa0bd96694de71513b880a1af9e5757fc04490b5715dcbb"} err="failed to get container status \"944ac059d0d326575fa0bd96694de71513b880a1af9e5757fc04490b5715dcbb\": rpc error: code = NotFound desc = could not find container \"944ac059d0d326575fa0bd96694de71513b880a1af9e5757fc04490b5715dcbb\": container with ID starting with 944ac059d0d326575fa0bd96694de71513b880a1af9e5757fc04490b5715dcbb not found: ID does not exist" Oct 30 19:15:42.760701 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:42.760662 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-921450-7f64fb4c79-xs76b"] Oct 30 19:15:42.772167 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:42.772106 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-921450-7f64fb4c79-xs76b"] Oct 30 19:15:42.812217 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:42.812180 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-921450-8d6546c9b-5pxxc"] Oct 30 19:15:42.822420 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:42.822394 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-921450-8d6546c9b-5pxxc"] Oct 30 19:15:42.858055 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:42.858024 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-fe1374-5c8465bf67-bl75c"] Oct 30 19:15:42.863450 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:42.863422 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-fe1374-5c8465bf67-bl75c"] Oct 30 19:15:43.064071 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:43.063985 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02cb3b89-6926-459a-a7ed-de16a0d30d6f" path="/var/lib/kubelet/pods/02cb3b89-6926-459a-a7ed-de16a0d30d6f/volumes" Oct 30 19:15:43.064444 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:43.064429 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17bb49e3-80a9-4216-bc19-20b1679782a9" path="/var/lib/kubelet/pods/17bb49e3-80a9-4216-bc19-20b1679782a9/volumes" Oct 30 19:15:43.064795 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:43.064780 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6170e608-fbfb-4cfe-acf6-2c09756134b5" path="/var/lib/kubelet/pods/6170e608-fbfb-4cfe-acf6-2c09756134b5/volumes" Oct 30 19:15:43.065196 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:43.065183 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96cf77ba-eb58-4e52-bab4-2a4e362dc95c" path="/var/lib/kubelet/pods/96cf77ba-eb58-4e52-bab4-2a4e362dc95c/volumes" Oct 30 19:15:43.397927 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:43.397894 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm"] Oct 30 19:15:43.398191 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:43.398163 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" podUID="bafcd4d0-8a69-460a-9e0c-21b69dc4327a" containerName="istio-proxy" containerID="cri-o://b12af34751cdc6a6785aaff9bc012ddb7a00e3b2932340dd8b1c5d63fdfd6361" gracePeriod=30 Oct 30 19:15:43.746515 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:43.746429 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:15:43.746515 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:43.746491 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" podUID="bafcd4d0-8a69-460a-9e0c-21b69dc4327a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:15:45.296146 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:45.296106 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-k8hhs_ea74abc8-3819-4e41-9705-9c95421816ce/manager/0.log" Oct 30 19:15:45.498902 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:45.498868 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-v4fq8_7695bc48-f4a1-4380-b25f-a32c35f6a500/manager/0.log" Oct 30 19:15:46.476788 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:46.476750 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-h2877_ec2b6bb1-c8d6-43ee-8234-0556646382a7/manager/1.log" Oct 30 19:15:46.885329 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:46.885297 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-dv6zs_24752fc1-eb92-452c-8f20-91cd12fc0ab0/manager/1.log" Oct 30 19:15:47.535852 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:47.535820 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-h2877_ec2b6bb1-c8d6-43ee-8234-0556646382a7/manager/0.log" Oct 30 19:15:47.559422 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:15:47.559381 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 19:15:47.768361 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:47.768323 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-dv6zs_24752fc1-eb92-452c-8f20-91cd12fc0ab0/manager/0.log" Oct 30 19:15:48.663001 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:48.662937 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" Oct 30 19:15:48.708362 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:48.708330 4442 generic.go:356] "Generic (PLEG): container finished" podID="bafcd4d0-8a69-460a-9e0c-21b69dc4327a" containerID="b12af34751cdc6a6785aaff9bc012ddb7a00e3b2932340dd8b1c5d63fdfd6361" exitCode=0 Oct 30 19:15:48.708550 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:48.708381 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" Oct 30 19:15:48.708550 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:48.708419 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" event={"ID":"bafcd4d0-8a69-460a-9e0c-21b69dc4327a","Type":"ContainerDied","Data":"b12af34751cdc6a6785aaff9bc012ddb7a00e3b2932340dd8b1c5d63fdfd6361"} Oct 30 19:15:48.708550 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:48.708522 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm" event={"ID":"bafcd4d0-8a69-460a-9e0c-21b69dc4327a","Type":"ContainerDied","Data":"7c528e24d3ef1397c42640450601cc9d79ac94764c01fe51dfbc8f2d1d066c92"} Oct 30 19:15:48.708743 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:48.708553 4442 scope.go:122] "RemoveContainer" containerID="b12af34751cdc6a6785aaff9bc012ddb7a00e3b2932340dd8b1c5d63fdfd6361" Oct 30 19:15:48.718239 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:48.718221 4442 scope.go:122] "RemoveContainer" containerID="b12af34751cdc6a6785aaff9bc012ddb7a00e3b2932340dd8b1c5d63fdfd6361" Oct 30 19:15:48.721848 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:15:48.721822 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b12af34751cdc6a6785aaff9bc012ddb7a00e3b2932340dd8b1c5d63fdfd6361\": container with ID starting with b12af34751cdc6a6785aaff9bc012ddb7a00e3b2932340dd8b1c5d63fdfd6361 not found: ID does not exist" containerID="b12af34751cdc6a6785aaff9bc012ddb7a00e3b2932340dd8b1c5d63fdfd6361" Oct 30 19:15:48.721971 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:48.721853 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b12af34751cdc6a6785aaff9bc012ddb7a00e3b2932340dd8b1c5d63fdfd6361"} err="failed to get container status \"b12af34751cdc6a6785aaff9bc012ddb7a00e3b2932340dd8b1c5d63fdfd6361\": rpc error: code = NotFound desc = could not find container \"b12af34751cdc6a6785aaff9bc012ddb7a00e3b2932340dd8b1c5d63fdfd6361\": container with ID starting with b12af34751cdc6a6785aaff9bc012ddb7a00e3b2932340dd8b1c5d63fdfd6361 not found: ID does not exist" Oct 30 19:15:48.803926 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:48.803889 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-kube-api-access-4plcf\" (UniqueName: \"kubernetes.io/projected/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-kube-api-access-4plcf\") pod \"bafcd4d0-8a69-460a-9e0c-21b69dc4327a\" (UID: \"bafcd4d0-8a69-460a-9e0c-21b69dc4327a\") " Oct 30 19:15:48.804086 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:48.803997 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-workload-certs\") pod \"bafcd4d0-8a69-460a-9e0c-21b69dc4327a\" (UID: \"bafcd4d0-8a69-460a-9e0c-21b69dc4327a\") " Oct 30 19:15:48.804086 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:48.804040 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-istio-data\") pod \"bafcd4d0-8a69-460a-9e0c-21b69dc4327a\" (UID: \"bafcd4d0-8a69-460a-9e0c-21b69dc4327a\") " Oct 30 19:15:48.804086 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:48.804079 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-istio-envoy\") pod \"bafcd4d0-8a69-460a-9e0c-21b69dc4327a\" (UID: \"bafcd4d0-8a69-460a-9e0c-21b69dc4327a\") " Oct 30 19:15:48.804331 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:48.804108 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-istiod-ca-cert\") pod \"bafcd4d0-8a69-460a-9e0c-21b69dc4327a\" (UID: \"bafcd4d0-8a69-460a-9e0c-21b69dc4327a\") " Oct 30 19:15:48.804331 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:48.804172 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-istio-podinfo\") pod \"bafcd4d0-8a69-460a-9e0c-21b69dc4327a\" (UID: \"bafcd4d0-8a69-460a-9e0c-21b69dc4327a\") " Oct 30 19:15:48.804446 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:48.804334 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-workload-socket\") pod \"bafcd4d0-8a69-460a-9e0c-21b69dc4327a\" (UID: \"bafcd4d0-8a69-460a-9e0c-21b69dc4327a\") " Oct 30 19:15:48.804446 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:48.804337 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-istio-data" pod "bafcd4d0-8a69-460a-9e0c-21b69dc4327a" (UID: "bafcd4d0-8a69-460a-9e0c-21b69dc4327a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:48.804446 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:48.804403 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-credential-socket\") pod \"bafcd4d0-8a69-460a-9e0c-21b69dc4327a\" (UID: \"bafcd4d0-8a69-460a-9e0c-21b69dc4327a\") " Oct 30 19:15:48.804446 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:48.804378 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-workload-certs" pod "bafcd4d0-8a69-460a-9e0c-21b69dc4327a" (UID: "bafcd4d0-8a69-460a-9e0c-21b69dc4327a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:48.804674 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:48.804456 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-istio-token\" (UniqueName: \"kubernetes.io/projected/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-istio-token\") pod \"bafcd4d0-8a69-460a-9e0c-21b69dc4327a\" (UID: \"bafcd4d0-8a69-460a-9e0c-21b69dc4327a\") " Oct 30 19:15:48.804674 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:48.804552 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-workload-socket" pod "bafcd4d0-8a69-460a-9e0c-21b69dc4327a" (UID: "bafcd4d0-8a69-460a-9e0c-21b69dc4327a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:48.804771 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:48.804678 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-credential-socket" pod "bafcd4d0-8a69-460a-9e0c-21b69dc4327a" (UID: "bafcd4d0-8a69-460a-9e0c-21b69dc4327a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:48.804818 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:48.804764 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:48.804818 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:48.804786 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:48.804818 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:48.804799 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:48.804818 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:48.804814 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:48.805027 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:48.804912 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-istiod-ca-cert" pod "bafcd4d0-8a69-460a-9e0c-21b69dc4327a" (UID: "bafcd4d0-8a69-460a-9e0c-21b69dc4327a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:48.806471 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:48.806448 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-istio-podinfo" pod "bafcd4d0-8a69-460a-9e0c-21b69dc4327a" (UID: "bafcd4d0-8a69-460a-9e0c-21b69dc4327a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:15:48.806676 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:48.806656 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-istio-token" pod "bafcd4d0-8a69-460a-9e0c-21b69dc4327a" (UID: "bafcd4d0-8a69-460a-9e0c-21b69dc4327a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:48.806898 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:48.806881 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-kube-api-access-4plcf" pod "bafcd4d0-8a69-460a-9e0c-21b69dc4327a" (UID: "bafcd4d0-8a69-460a-9e0c-21b69dc4327a"). InnerVolumeSpecName "kube-api-access-4plcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:48.806960 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:48.806888 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-istio-envoy" pod "bafcd4d0-8a69-460a-9e0c-21b69dc4327a" (UID: "bafcd4d0-8a69-460a-9e0c-21b69dc4327a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:48.905474 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:48.905438 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:48.905474 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:48.905467 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:48.905474 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:48.905477 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:48.905720 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:48.905485 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:48.905720 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:48.905496 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4plcf\" (UniqueName: \"kubernetes.io/projected/bafcd4d0-8a69-460a-9e0c-21b69dc4327a-kube-api-access-4plcf\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\"" Oct 30 19:15:49.053603 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:49.053570 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm"] Oct 30 19:15:49.066535 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:49.066498 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-nvjxm"] Oct 30 19:15:51.065158 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:51.065105 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bafcd4d0-8a69-460a-9e0c-21b69dc4327a" path="/var/lib/kubelet/pods/bafcd4d0-8a69-460a-9e0c-21b69dc4327a/volumes" Oct 30 19:15:59.847360 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:15:59.847323 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-nxg2g_dd0fb59a-f6bb-43fd-95e4-6f740ace1c6e/manager/1.log" Oct 30 19:16:10.551276 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:16:10.551245 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-67cc46c64-66wvf_024b2852-2e03-4f0b-a7dd-858dc9ed69ed/manager/1.log" Oct 30 19:16:17.558776 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:16:17.558739 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 19:16:21.687302 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:16:21.687271 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-nxg2g_dd0fb59a-f6bb-43fd-95e4-6f740ace1c6e/manager/0.log" Oct 30 19:16:22.248812 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:16:22.248782 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-67cc46c64-66wvf_024b2852-2e03-4f0b-a7dd-858dc9ed69ed/manager/0.log" Oct 30 19:16:29.512278 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:16:29.512251 4442 scope.go:122] "RemoveContainer" containerID="1842bd253d4c09e7632a481163c766517ce78910d9cec408ebaba8b7370a584d" Oct 30 19:16:29.521928 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:16:29.521909 4442 scope.go:122] "RemoveContainer" containerID="d84d5af7ae4a789f6973f93eeb805460fa02e6f2f904d3b026d2df8a51a02e96" Oct 30 19:16:29.530597 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:16:29.530580 4442 scope.go:122] "RemoveContainer" containerID="fd6a2d78739f6a2627f8834dedd93ae3f253fa9084a2cb243dbbefde2d316184" Oct 30 19:16:29.538535 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:16:29.538517 4442 scope.go:122] "RemoveContainer" containerID="8cb609731d517ef62f6d0aa166f2da14c5b91b6597eb4f0ef5ab2ddfbca166d3" Oct 30 19:16:44.138305 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:16:44.138262 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-nxg2g_dd0fb59a-f6bb-43fd-95e4-6f740ace1c6e/manager/1.log" Oct 30 19:16:45.006306 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:16:45.006261 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-67cc46c64-66wvf_024b2852-2e03-4f0b-a7dd-858dc9ed69ed/manager/1.log" Oct 30 19:16:46.139645 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:16:46.139615 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-nxg2g_dd0fb59a-f6bb-43fd-95e4-6f740ace1c6e/manager/0.log" Oct 30 19:16:46.644993 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:16:46.644964 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-67cc46c64-66wvf_024b2852-2e03-4f0b-a7dd-858dc9ed69ed/manager/0.log" Oct 30 19:16:47.559986 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:16:47.559950 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 19:16:48.615031 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:16:48.615001 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-8a33da54-qffvt_d2f660b9-2642-4b8a-95ff-c090e33ae02e/revoke-keys/0.log" Oct 30 19:16:49.388485 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:16:49.388456 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b963c8b4-j8jgz_05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e/revoke-keys/0.log" Oct 30 19:16:49.999761 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:16:49.999736 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e1fba6e3-ld59t_6dab3a37-4bad-464c-b98f-5f6cc0592e9f/revoke-keys/0.log" Oct 30 19:16:50.188780 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:16:50.188752 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-d5s69_c8c00dd2-1002-436b-810d-7555fc4286df/postgres/0.log" Oct 30 19:16:51.911593 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:16:51.911562 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-k8hhs_ea74abc8-3819-4e41-9705-9c95421816ce/manager/0.log" Oct 30 19:16:52.122107 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:16:52.122081 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-v4fq8_7695bc48-f4a1-4380-b25f-a32c35f6a500/manager/0.log" Oct 30 19:16:53.014339 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:16:53.014303 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-h2877_ec2b6bb1-c8d6-43ee-8234-0556646382a7/manager/1.log" Oct 30 19:16:53.444562 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:16:53.444529 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-dv6zs_24752fc1-eb92-452c-8f20-91cd12fc0ab0/manager/1.log" Oct 30 19:16:54.210165 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:16:54.210118 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-h2877_ec2b6bb1-c8d6-43ee-8234-0556646382a7/manager/0.log" Oct 30 19:16:54.450189 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:16:54.450159 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-dv6zs_24752fc1-eb92-452c-8f20-91cd12fc0ab0/manager/0.log" Oct 30 19:16:54.999579 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:16:54.999544 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6545b48489hbqng_c165fa36-f6e3-481f-a56e-05f046c197cd/istio-proxy/0.log" Oct 30 19:16:55.207738 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:16:55.207710 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-j44sc_8e1a75a1-61d7-4285-bf95-ad9b23b57ad2/discovery/0.log" Oct 30 19:16:56.388617 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:16:56.388569 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c_aa4194a4-34ec-4500-a28f-e7a55df72764/istio-proxy/0.log" Oct 30 19:16:57.929878 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:16:57.929852 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-dctgm_ac100f87-aa55-49c3-8760-85523d3927db/main/0.log" Oct 30 19:16:58.099404 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:16:58.099372 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-7s88v_764a8232-9c6d-48aa-b055-8a15bb2e352d/main/0.log" Oct 30 19:16:58.690089 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:16:58.690049 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-qq7fh_31abe66d-4cdc-4258-afbc-39a2d5d05cbf/main/0.log" Oct 30 19:17:17.559385 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:17:17.559343 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 19:17:19.411455 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:19.411415 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-z8svd/must-gather-lxtzn"] Oct 30 19:17:19.413768 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:19.411897 4442 cpu_manager.go:414] "containerMap: removing container" podUID="cf7d5f54-e091-4032-ab11-af2687e453c8" containerName="main" Oct 30 19:17:19.413768 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:19.411915 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf7d5f54-e091-4032-ab11-af2687e453c8" containerName="main" Oct 30 19:17:19.413768 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:19.411935 4442 cpu_manager.go:414] "containerMap: removing container" podUID="05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e" containerName="revoke-keys" Oct 30 19:17:19.413768 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:19.411944 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e" containerName="revoke-keys" Oct 30 19:17:19.413768 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:19.411959 4442 cpu_manager.go:414] "containerMap: removing container" podUID="6dab3a37-4bad-464c-b98f-5f6cc0592e9f" containerName="revoke-keys" Oct 30 19:17:19.413768 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:19.411968 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dab3a37-4bad-464c-b98f-5f6cc0592e9f" containerName="revoke-keys" Oct 30 19:17:19.413768 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:19.411982 4442 cpu_manager.go:414] "containerMap: removing container" podUID="6170e608-fbfb-4cfe-acf6-2c09756134b5" containerName="payload-pre-processing" Oct 30 19:17:19.413768 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:19.411989 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6170e608-fbfb-4cfe-acf6-2c09756134b5" containerName="payload-pre-processing" Oct 30 19:17:19.413768 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:19.412006 4442 cpu_manager.go:414] "containerMap: removing container" podUID="269e36c9-21e8-412e-b49e-e446b7d57def" containerName="payload-processing" Oct 30 19:17:19.413768 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:19.412014 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="269e36c9-21e8-412e-b49e-e446b7d57def" containerName="payload-processing" Oct 30 19:17:19.413768 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:19.412028 4442 cpu_manager.go:414] "containerMap: removing container" podUID="13d41d70-3283-4135-9f4e-3ff0c53cec22" containerName="maas-api" Oct 30 19:17:19.413768 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:19.412034 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13d41d70-3283-4135-9f4e-3ff0c53cec22" containerName="maas-api" Oct 30 19:17:19.413768 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:19.412053 4442 cpu_manager.go:414] "containerMap: removing container" podUID="96cf77ba-eb58-4e52-bab4-2a4e362dc95c" containerName="payload-pre-processing" Oct 30 19:17:19.413768 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:19.412062 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96cf77ba-eb58-4e52-bab4-2a4e362dc95c" containerName="payload-pre-processing" Oct 30 19:17:19.413768 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:19.412077 4442 cpu_manager.go:414] "containerMap: removing container" podUID="6bc83ed4-302b-466c-9599-b022b3f0afb1" containerName="maas-api" Oct 30 19:17:19.413768 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:19.412083 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bc83ed4-302b-466c-9599-b022b3f0afb1" containerName="maas-api" Oct 30 19:17:19.413768 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:19.412093 4442 cpu_manager.go:414] "containerMap: removing container" podUID="bafcd4d0-8a69-460a-9e0c-21b69dc4327a" containerName="istio-proxy" Oct 30 19:17:19.413768 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:19.412100 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bafcd4d0-8a69-460a-9e0c-21b69dc4327a" containerName="istio-proxy" Oct 30 19:17:19.413768 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:19.412111 4442 cpu_manager.go:414] "containerMap: removing container" podUID="02cb3b89-6926-459a-a7ed-de16a0d30d6f" containerName="payload-pre-processing" Oct 30 19:17:19.413768 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:19.412118 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02cb3b89-6926-459a-a7ed-de16a0d30d6f" containerName="payload-pre-processing" Oct 30 19:17:19.413768 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:19.412150 4442 cpu_manager.go:414] "containerMap: removing container" podUID="17bb49e3-80a9-4216-bc19-20b1679782a9" containerName="payload-processing" Oct 30 19:17:19.413768 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:19.412157 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17bb49e3-80a9-4216-bc19-20b1679782a9" containerName="payload-processing" Oct 30 19:17:19.413768 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:19.412261 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="96cf77ba-eb58-4e52-bab4-2a4e362dc95c" containerName="payload-pre-processing" Oct 30 19:17:19.413768 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:19.412275 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bc83ed4-302b-466c-9599-b022b3f0afb1" containerName="maas-api" Oct 30 19:17:19.413768 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:19.412286 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="bafcd4d0-8a69-460a-9e0c-21b69dc4327a" containerName="istio-proxy" Oct 30 19:17:19.413768 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:19.412297 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="02cb3b89-6926-459a-a7ed-de16a0d30d6f" containerName="payload-pre-processing" Oct 30 19:17:19.413768 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:19.412307 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="17bb49e3-80a9-4216-bc19-20b1679782a9" containerName="payload-processing" Oct 30 19:17:19.413768 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:19.412318 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf7d5f54-e091-4032-ab11-af2687e453c8" containerName="main" Oct 30 19:17:19.413768 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:19.412329 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="05a3ac2b-77cc-41ce-bc8e-8e88460d5c3e" containerName="revoke-keys" Oct 30 19:17:19.413768 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:19.412339 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dab3a37-4bad-464c-b98f-5f6cc0592e9f" containerName="revoke-keys" Oct 30 19:17:19.413768 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:19.412348 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="6170e608-fbfb-4cfe-acf6-2c09756134b5" containerName="payload-pre-processing" Oct 30 19:17:19.413768 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:19.412356 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="269e36c9-21e8-412e-b49e-e446b7d57def" containerName="payload-processing" Oct 30 19:17:19.413768 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:19.412365 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="13d41d70-3283-4135-9f4e-3ff0c53cec22" containerName="maas-api" Oct 30 19:17:19.415549 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:19.415534 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z8svd/must-gather-lxtzn" Oct 30 19:17:19.434686 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:19.434664 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z8svd\"/\"kube-root-ca.crt\"" Oct 30 19:17:19.434827 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:19.434687 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z8svd\"/\"openshift-service-ca.crt\"" Oct 30 19:17:19.445977 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:19.445950 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-z8svd\"/\"default-dockercfg-24v42\"" Oct 30 19:17:19.453883 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:19.453856 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z8svd/must-gather-lxtzn"] Oct 30 19:17:19.485256 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:19.485223 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/34a34766-b52a-43a0-8892-92435d5f1e10-must-gather-output\") pod \"must-gather-lxtzn\" (UID: \"34a34766-b52a-43a0-8892-92435d5f1e10\") " pod="openshift-must-gather-z8svd/must-gather-lxtzn" Oct 30 19:17:19.485441 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:19.485285 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j246d\" (UniqueName: \"kubernetes.io/projected/34a34766-b52a-43a0-8892-92435d5f1e10-kube-api-access-j246d\") pod \"must-gather-lxtzn\" (UID: \"34a34766-b52a-43a0-8892-92435d5f1e10\") " pod="openshift-must-gather-z8svd/must-gather-lxtzn" Oct 30 19:17:19.585932 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:19.585896 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/34a34766-b52a-43a0-8892-92435d5f1e10-must-gather-output\") pod \"must-gather-lxtzn\" (UID: \"34a34766-b52a-43a0-8892-92435d5f1e10\") " pod="openshift-must-gather-z8svd/must-gather-lxtzn" Oct 30 19:17:19.586127 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:19.585946 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j246d\" (UniqueName: \"kubernetes.io/projected/34a34766-b52a-43a0-8892-92435d5f1e10-kube-api-access-j246d\") pod \"must-gather-lxtzn\" (UID: \"34a34766-b52a-43a0-8892-92435d5f1e10\") " pod="openshift-must-gather-z8svd/must-gather-lxtzn" Oct 30 19:17:19.586285 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:19.586267 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/34a34766-b52a-43a0-8892-92435d5f1e10-must-gather-output\") pod \"must-gather-lxtzn\" (UID: \"34a34766-b52a-43a0-8892-92435d5f1e10\") " pod="openshift-must-gather-z8svd/must-gather-lxtzn" Oct 30 19:17:19.608312 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:19.608278 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j246d\" (UniqueName: \"kubernetes.io/projected/34a34766-b52a-43a0-8892-92435d5f1e10-kube-api-access-j246d\") pod \"must-gather-lxtzn\" (UID: \"34a34766-b52a-43a0-8892-92435d5f1e10\") " pod="openshift-must-gather-z8svd/must-gather-lxtzn" Oct 30 19:17:19.747013 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:19.746925 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z8svd/must-gather-lxtzn" Oct 30 19:17:20.054642 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:20.054609 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z8svd/must-gather-lxtzn"] Oct 30 19:17:20.055488 ip-10-0-138-176 kubenswrapper[4442]: W1030 19:17:20.055461 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34a34766_b52a_43a0_8892_92435d5f1e10.slice/crio-a1ced61911539e1ddf0e5d6628d4aad0df06044e6ea3cac07726c05d6dc71032 WatchSource:0}: Error finding container a1ced61911539e1ddf0e5d6628d4aad0df06044e6ea3cac07726c05d6dc71032: Status 404 returned error can't find the container with id a1ced61911539e1ddf0e5d6628d4aad0df06044e6ea3cac07726c05d6dc71032 Oct 30 19:17:20.057479 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:20.057462 4442 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:17:20.245192 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:20.245151 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z8svd/must-gather-lxtzn" event={"ID":"34a34766-b52a-43a0-8892-92435d5f1e10","Type":"ContainerStarted","Data":"a1ced61911539e1ddf0e5d6628d4aad0df06044e6ea3cac07726c05d6dc71032"} Oct 30 19:17:25.283353 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:25.283277 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z8svd/must-gather-lxtzn" event={"ID":"34a34766-b52a-43a0-8892-92435d5f1e10","Type":"ContainerStarted","Data":"44d20130ba7c9cf4124d797cf724ef69327dd84e72df2c3bab22de4fac3bb167"} Oct 30 19:17:25.283353 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:25.283326 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z8svd/must-gather-lxtzn" event={"ID":"34a34766-b52a-43a0-8892-92435d5f1e10","Type":"ContainerStarted","Data":"9a4e8e4c9734a57ef4175638c512be0652211241f4426290eb572c17c6f767cf"} Oct 30 19:17:25.350338 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:25.350286 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-z8svd/must-gather-lxtzn" podStartSLOduration=1.379896753 podStartE2EDuration="6.350270887s" totalImagesPullingTime="4.970374134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:25.346680698 +0000 UTC m=+4742.436743037" watchObservedRunningTime="2026-10-30 19:17:25.350270887 +0000 UTC m=+4742.440333226" Oct 30 19:17:25.431716 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:25.431684 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-srgjm_a24e24cc-a321-4a62-b023-7b015f155d55/global-pull-secret-syncer/0.log" Oct 30 19:17:26.194697 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:26.194660 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-c84zp_59229234-921c-4177-8b85-3e982194f625/konnectivity-agent/0.log" Oct 30 19:17:26.826494 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:26.826461 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-176.ec2.internal_9e81bd78300430e51b01d30caa1123e7/haproxy/0.log" Oct 30 19:17:29.058643 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:29.051870 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a33da54-qffvt"] Oct 30 19:17:29.066913 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:29.066887 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a33da54-qffvt"] Oct 30 19:17:29.600806 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:29.600776 4442 scope.go:122] "RemoveContainer" containerID="e22862c383f74a970cf2f2c76ac36565690f698e7b14d09da055b830b97e1f69" Oct 30 19:17:31.062322 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:31.062292 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2f660b9-2642-4b8a-95ff-c090e33ae02e" path="/var/lib/kubelet/pods/d2f660b9-2642-4b8a-95ff-c090e33ae02e/volumes" Oct 30 19:17:31.811658 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:31.811632 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-nxg2g_dd0fb59a-f6bb-43fd-95e4-6f740ace1c6e/manager/0.log" Oct 30 19:17:31.830749 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:31.830719 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-nxg2g_dd0fb59a-f6bb-43fd-95e4-6f740ace1c6e/manager/1.log" Oct 30 19:17:32.973274 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:32.973234 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-67cc46c64-66wvf_024b2852-2e03-4f0b-a7dd-858dc9ed69ed/manager/0.log" Oct 30 19:17:33.123070 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:33.123042 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-67cc46c64-66wvf_024b2852-2e03-4f0b-a7dd-858dc9ed69ed/manager/1.log" Oct 30 19:17:34.768380 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:34.768345 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-k8hhs_ea74abc8-3819-4e41-9705-9c95421816ce/manager/0.log" Oct 30 19:17:34.898296 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:34.898268 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-v4fq8_7695bc48-f4a1-4380-b25f-a32c35f6a500/manager/0.log" Oct 30 19:17:35.656820 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:35.656785 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-h2877_ec2b6bb1-c8d6-43ee-8234-0556646382a7/manager/0.log" Oct 30 19:17:35.797037 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:35.796991 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-nxg2g_dd0fb59a-f6bb-43fd-95e4-6f740ace1c6e/manager/0.log" Oct 30 19:17:35.835736 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:35.835700 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-nxg2g_dd0fb59a-f6bb-43fd-95e4-6f740ace1c6e/manager/1.log" Oct 30 19:17:35.872055 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:35.872030 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-h2877_ec2b6bb1-c8d6-43ee-8234-0556646382a7/manager/1.log" Oct 30 19:17:36.152752 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:36.152725 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-dv6zs_24752fc1-eb92-452c-8f20-91cd12fc0ab0/manager/0.log" Oct 30 19:17:36.161647 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:36.161627 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-dv6zs_24752fc1-eb92-452c-8f20-91cd12fc0ab0/manager/1.log" Oct 30 19:17:36.691981 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:36.691947 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wj67d/perf-node-gather-daemonset-k65zr"] Oct 30 19:17:36.697688 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:36.697665 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-k65zr" Oct 30 19:17:36.721758 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:36.721725 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wj67d/perf-node-gather-daemonset-k65zr"] Oct 30 19:17:36.749143 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:36.749099 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/efcc2b62-267c-492f-ab3f-e63559bd5fef-proc\") pod \"perf-node-gather-daemonset-k65zr\" (UID: \"efcc2b62-267c-492f-ab3f-e63559bd5fef\") " pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-k65zr" Oct 30 19:17:36.749143 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:36.749148 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9b8p\" (UniqueName: \"kubernetes.io/projected/efcc2b62-267c-492f-ab3f-e63559bd5fef-kube-api-access-p9b8p\") pod \"perf-node-gather-daemonset-k65zr\" (UID: \"efcc2b62-267c-492f-ab3f-e63559bd5fef\") " pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-k65zr" Oct 30 19:17:36.749359 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:36.749215 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/efcc2b62-267c-492f-ab3f-e63559bd5fef-lib-modules\") pod \"perf-node-gather-daemonset-k65zr\" (UID: \"efcc2b62-267c-492f-ab3f-e63559bd5fef\") " pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-k65zr" Oct 30 19:17:36.749359 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:36.749233 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/efcc2b62-267c-492f-ab3f-e63559bd5fef-podres\") pod \"perf-node-gather-daemonset-k65zr\" (UID: \"efcc2b62-267c-492f-ab3f-e63559bd5fef\") " pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-k65zr" Oct 30 19:17:36.749359 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:36.749288 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/efcc2b62-267c-492f-ab3f-e63559bd5fef-sys\") pod \"perf-node-gather-daemonset-k65zr\" (UID: \"efcc2b62-267c-492f-ab3f-e63559bd5fef\") " pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-k65zr" Oct 30 19:17:36.806301 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:36.806271 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wj67d\"/\"kube-root-ca.crt\"" Oct 30 19:17:36.806301 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:36.806287 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wj67d\"/\"openshift-service-ca.crt\"" Oct 30 19:17:36.806727 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:36.806388 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wj67d\"/\"default-dockercfg-g4gj2\"" Oct 30 19:17:36.852310 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:36.852281 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/efcc2b62-267c-492f-ab3f-e63559bd5fef-sys\") pod \"perf-node-gather-daemonset-k65zr\" (UID: \"efcc2b62-267c-492f-ab3f-e63559bd5fef\") " pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-k65zr" Oct 30 19:17:36.852624 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:36.852395 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/efcc2b62-267c-492f-ab3f-e63559bd5fef-sys\") pod \"perf-node-gather-daemonset-k65zr\" (UID: \"efcc2b62-267c-492f-ab3f-e63559bd5fef\") " pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-k65zr" Oct 30 19:17:36.852752 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:36.852599 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/efcc2b62-267c-492f-ab3f-e63559bd5fef-proc\") pod \"perf-node-gather-daemonset-k65zr\" (UID: \"efcc2b62-267c-492f-ab3f-e63559bd5fef\") " pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-k65zr" Oct 30 19:17:36.852752 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:36.852659 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/efcc2b62-267c-492f-ab3f-e63559bd5fef-proc\") pod \"perf-node-gather-daemonset-k65zr\" (UID: \"efcc2b62-267c-492f-ab3f-e63559bd5fef\") " pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-k65zr" Oct 30 19:17:36.852752 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:36.852665 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9b8p\" (UniqueName: \"kubernetes.io/projected/efcc2b62-267c-492f-ab3f-e63559bd5fef-kube-api-access-p9b8p\") pod \"perf-node-gather-daemonset-k65zr\" (UID: \"efcc2b62-267c-492f-ab3f-e63559bd5fef\") " pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-k65zr" Oct 30 19:17:36.852752 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:36.852738 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/efcc2b62-267c-492f-ab3f-e63559bd5fef-lib-modules\") pod \"perf-node-gather-daemonset-k65zr\" (UID: \"efcc2b62-267c-492f-ab3f-e63559bd5fef\") " pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-k65zr" Oct 30 19:17:36.852994 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:36.852771 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/efcc2b62-267c-492f-ab3f-e63559bd5fef-podres\") pod \"perf-node-gather-daemonset-k65zr\" (UID: \"efcc2b62-267c-492f-ab3f-e63559bd5fef\") " pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-k65zr" Oct 30 19:17:36.852994 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:36.852845 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/efcc2b62-267c-492f-ab3f-e63559bd5fef-lib-modules\") pod \"perf-node-gather-daemonset-k65zr\" (UID: \"efcc2b62-267c-492f-ab3f-e63559bd5fef\") " pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-k65zr" Oct 30 19:17:36.852994 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:36.852946 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/efcc2b62-267c-492f-ab3f-e63559bd5fef-podres\") pod \"perf-node-gather-daemonset-k65zr\" (UID: \"efcc2b62-267c-492f-ab3f-e63559bd5fef\") " pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-k65zr" Oct 30 19:17:36.880021 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:36.879938 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9b8p\" (UniqueName: \"kubernetes.io/projected/efcc2b62-267c-492f-ab3f-e63559bd5fef-kube-api-access-p9b8p\") pod \"perf-node-gather-daemonset-k65zr\" (UID: \"efcc2b62-267c-492f-ab3f-e63559bd5fef\") " pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-k65zr" Oct 30 19:17:36.944220 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:36.944071 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-67cc46c64-66wvf_024b2852-2e03-4f0b-a7dd-858dc9ed69ed/manager/0.log" Oct 30 19:17:37.010888 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:37.010859 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-k65zr" Oct 30 19:17:37.098570 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:37.098543 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-nxg2g_dd0fb59a-f6bb-43fd-95e4-6f740ace1c6e/manager/0.log" Oct 30 19:17:37.146530 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:37.146503 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-nxg2g_dd0fb59a-f6bb-43fd-95e4-6f740ace1c6e/manager/1.log" Oct 30 19:17:37.151575 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:37.151551 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-67cc46c64-66wvf_024b2852-2e03-4f0b-a7dd-858dc9ed69ed/manager/1.log" Oct 30 19:17:37.321268 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:37.321242 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wj67d/perf-node-gather-daemonset-k65zr"] Oct 30 19:17:37.322926 ip-10-0-138-176 kubenswrapper[4442]: W1030 19:17:37.322899 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podefcc2b62_267c_492f_ab3f_e63559bd5fef.slice/crio-a8469e62fa838f56582cebc2273ce3bce65ecd98907aa755b4bba691406447e1 WatchSource:0}: Error finding container a8469e62fa838f56582cebc2273ce3bce65ecd98907aa755b4bba691406447e1: Status 404 returned error can't find the container with id a8469e62fa838f56582cebc2273ce3bce65ecd98907aa755b4bba691406447e1 Oct 30 19:17:37.355717 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:37.355678 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-k65zr" event={"ID":"efcc2b62-267c-492f-ab3f-e63559bd5fef","Type":"ContainerStarted","Data":"a8469e62fa838f56582cebc2273ce3bce65ecd98907aa755b4bba691406447e1"} Oct 30 19:17:38.187231 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:38.187196 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-67cc46c64-66wvf_024b2852-2e03-4f0b-a7dd-858dc9ed69ed/manager/0.log" Oct 30 19:17:38.365003 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:38.364967 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-k65zr" event={"ID":"efcc2b62-267c-492f-ab3f-e63559bd5fef","Type":"ContainerStarted","Data":"3d4a1f6e5d9e2f737751c1800ec563f7409e37cbd4f1bef5ebd595c24c3acc28"} Oct 30 19:17:38.365220 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:38.365088 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-k65zr" Oct 30 19:17:38.385666 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:38.385638 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-67cc46c64-66wvf_024b2852-2e03-4f0b-a7dd-858dc9ed69ed/manager/1.log" Oct 30 19:17:38.404749 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:38.404696 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-k65zr" podStartSLOduration=2.40468082 podStartE2EDuration="2.40468082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:17:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:38.403741357 +0000 UTC m=+4755.493803697" watchObservedRunningTime="2026-10-30 19:17:38.40468082 +0000 UTC m=+4755.494743158" Oct 30 19:17:39.015583 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:39.015558 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-cw2zd_b6361df3-6ef7-42bc-8c50-c6f317d9d46d/kube-state-metrics/0.log" Oct 30 19:17:39.160774 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:39.160745 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-cw2zd_b6361df3-6ef7-42bc-8c50-c6f317d9d46d/kube-rbac-proxy-main/0.log" Oct 30 19:17:39.276074 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:39.274791 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-nxg2g_dd0fb59a-f6bb-43fd-95e4-6f740ace1c6e/manager/0.log" Oct 30 19:17:39.298120 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:39.298096 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-nxg2g_dd0fb59a-f6bb-43fd-95e4-6f740ace1c6e/manager/1.log" Oct 30 19:17:39.307517 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:39.307488 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-cw2zd_b6361df3-6ef7-42bc-8c50-c6f317d9d46d/kube-rbac-proxy-self/0.log" Oct 30 19:17:39.453834 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:39.453810 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-78c74bf4c6-prjr2_9101c57f-0762-487c-9f61-c1efa4900965/metrics-server/0.log" Oct 30 19:17:39.777110 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:39.777076 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6zxkn_29e7a0a9-5a42-47b2-bf89-eccde79a39fd/node-exporter/0.log" Oct 30 19:17:39.969811 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:39.969783 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6zxkn_29e7a0a9-5a42-47b2-bf89-eccde79a39fd/kube-rbac-proxy/0.log" Oct 30 19:17:40.137647 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:40.137622 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6zxkn_29e7a0a9-5a42-47b2-bf89-eccde79a39fd/init-textfile/0.log" Oct 30 19:17:40.188530 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:40.188493 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-nxg2g_dd0fb59a-f6bb-43fd-95e4-6f740ace1c6e/manager/0.log" Oct 30 19:17:40.238407 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:40.238377 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-nxg2g_dd0fb59a-f6bb-43fd-95e4-6f740ace1c6e/manager/1.log" Oct 30 19:17:40.247941 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:40.247913 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-67cc46c64-66wvf_024b2852-2e03-4f0b-a7dd-858dc9ed69ed/manager/0.log" Oct 30 19:17:40.417901 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:40.417821 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-67cc46c64-66wvf_024b2852-2e03-4f0b-a7dd-858dc9ed69ed/manager/1.log" Oct 30 19:17:41.264727 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:41.264694 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-67cc46c64-66wvf_024b2852-2e03-4f0b-a7dd-858dc9ed69ed/manager/0.log" Oct 30 19:17:41.446985 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:41.446955 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-67cc46c64-66wvf_024b2852-2e03-4f0b-a7dd-858dc9ed69ed/manager/1.log" Oct 30 19:17:42.405867 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:42.405843 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6545b48489hbqng_c165fa36-f6e3-481f-a56e-05f046c197cd/istio-proxy/0.log" Oct 30 19:17:42.450241 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:42.450206 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-nvgn2_3ae3331e-2881-427e-8685-2aa0473590bd/prometheus-operator/0.log" Oct 30 19:17:42.577753 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:42.577699 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-nvgn2_3ae3331e-2881-427e-8685-2aa0473590bd/kube-rbac-proxy/0.log" Oct 30 19:17:43.302997 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:43.302963 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-nxg2g_dd0fb59a-f6bb-43fd-95e4-6f740ace1c6e/manager/0.log" Oct 30 19:17:43.339004 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:43.338965 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-nxg2g_dd0fb59a-f6bb-43fd-95e4-6f740ace1c6e/manager/1.log" Oct 30 19:17:44.386528 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:44.386491 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-k65zr" Oct 30 19:17:44.449081 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:44.449041 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-67cc46c64-66wvf_024b2852-2e03-4f0b-a7dd-858dc9ed69ed/manager/0.log" Oct 30 19:17:44.706456 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:44.706346 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-67cc46c64-66wvf_024b2852-2e03-4f0b-a7dd-858dc9ed69ed/manager/1.log" Oct 30 19:17:45.595191 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:45.595156 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-j44sc_8e1a75a1-61d7-4285-bf95-ad9b23b57ad2/discovery/0.log" Oct 30 19:17:46.037101 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:46.037069 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c_aa4194a4-34ec-4500-a28f-e7a55df72764/istio-proxy/0.log" Oct 30 19:17:46.492195 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:46.492168 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6545b48489hbqng_c165fa36-f6e3-481f-a56e-05f046c197cd/istio-proxy/0.log" Oct 30 19:17:47.565564 ip-10-0-138-176 kubenswrapper[4442]: E1030 19:17:47.565524 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vlnv2" containerName="extractor" Oct 30 19:17:48.736406 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:48.736367 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5fb9f6977f-bl8xj_05978eb8-6b34-4cfa-ab02-c57212d5d8d8/console/0.log" Oct 30 19:17:49.496172 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:49.496115 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-j44sc_8e1a75a1-61d7-4285-bf95-ad9b23b57ad2/discovery/0.log" Oct 30 19:17:49.880569 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:49.880538 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c_aa4194a4-34ec-4500-a28f-e7a55df72764/istio-proxy/0.log" Oct 30 19:17:52.134304 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:52.134275 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-k8hhs_ea74abc8-3819-4e41-9705-9c95421816ce/manager/0.log" Oct 30 19:17:52.197417 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:52.197387 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-v4fq8_7695bc48-f4a1-4380-b25f-a32c35f6a500/manager/0.log" Oct 30 19:17:52.321885 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:52.321851 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6545b48489hbqng_c165fa36-f6e3-481f-a56e-05f046c197cd/istio-proxy/0.log" Oct 30 19:17:52.629806 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:52.629775 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-h2877_ec2b6bb1-c8d6-43ee-8234-0556646382a7/manager/0.log" Oct 30 19:17:52.818257 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:52.818216 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-h2877_ec2b6bb1-c8d6-43ee-8234-0556646382a7/manager/1.log" Oct 30 19:17:52.830912 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:52.830873 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4qz5q_8215ee49-dab0-40f1-af73-3d4493e3bc04/dns/0.log" Oct 30 19:17:52.952105 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:52.952025 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4qz5q_8215ee49-dab0-40f1-af73-3d4493e3bc04/kube-rbac-proxy/0.log" Oct 30 19:17:52.955754 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:52.955718 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-dv6zs_24752fc1-eb92-452c-8f20-91cd12fc0ab0/manager/0.log" Oct 30 19:17:52.970406 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:52.970373 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-dv6zs_24752fc1-eb92-452c-8f20-91cd12fc0ab0/manager/1.log" Oct 30 19:17:53.933470 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:53.933434 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vp6gj_0772096e-be2c-449d-9bf6-c56cbd6c203a/dns-node-resolver/0.log" Oct 30 19:17:55.085399 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:55.085365 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-k8hhs_ea74abc8-3819-4e41-9705-9c95421816ce/manager/0.log" Oct 30 19:17:55.152290 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:55.152261 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-v4fq8_7695bc48-f4a1-4380-b25f-a32c35f6a500/manager/0.log" Oct 30 19:17:55.561954 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:55.561918 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-59dcddb9c7-k7vz2_5922c57f-e33c-4af5-a015-3c010e6a661f/registry/0.log" Oct 30 19:17:55.639685 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:55.639654 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-h2877_ec2b6bb1-c8d6-43ee-8234-0556646382a7/manager/0.log" Oct 30 19:17:55.863119 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:55.863033 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-h2877_ec2b6bb1-c8d6-43ee-8234-0556646382a7/manager/1.log" Oct 30 19:17:55.900327 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:55.900293 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-j44sc_8e1a75a1-61d7-4285-bf95-ad9b23b57ad2/discovery/0.log" Oct 30 19:17:55.954120 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:55.954090 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tsfch_e0a64047-334d-4df8-83de-3a211200ea62/node-ca/0.log" Oct 30 19:17:55.993414 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:55.993386 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-dv6zs_24752fc1-eb92-452c-8f20-91cd12fc0ab0/manager/0.log" Oct 30 19:17:56.005585 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:56.005565 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-dv6zs_24752fc1-eb92-452c-8f20-91cd12fc0ab0/manager/1.log" Oct 30 19:17:56.265777 ip-10-0-138-176 kubenswrapper[4442]: I1030 19:17:56.265746 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-77d8c79567-5b95c_aa4194a4-34ec-4500-a28f-e7a55df72764/istio-proxy/0.log"